KR102559789B1 - System for providing visual impairment support service using realtime streaming - Google Patents

Abstract

A system for providing a guide service for the visually impaired using real-time streaming is provided, and a wearable device that shoots a preset direction, a user terminal that receives a guide request and uploads the video captured from the wearable device in real time, and is connected to the user terminal's guide request and a guide terminal that overlays and outputs the direction and distance for each object on the screen when the video streamed from the wearable device is output on the screen, a registration unit that maps and stores the map so that the user terminal and the wearable device are interlocked, and a request for guidance from the user terminal In case of transmitting, a real-time relay unit that connects the guidance terminal and the user terminal and streams the video captured by the wearable device to the guidance terminal, a display unit that displays the direction and distance of each object in the screen of the guidance terminal in an overlay manner, and a guide service providing server including a guide unit for guiding and transmitting a direction and distance of an object corresponding to a request of a user terminal in a guide terminal.

Description

System for providing guidance service for the blind using real-time streaming {SYSTEM FOR PROVIDING VISUAL IMPAIRMENT SUPPORT SERVICE USING REALTIME STREAMING}

The present invention relates to a system for providing a guide service for the visually impaired using real-time streaming, and provides a system that allows a user terminal to provide voice guidance by receiving real-time streaming from a guide terminal of an image captured by a wearable device.

In general, walking is accomplished through the direction of walking down the street and the movement process, which is a form of movement. Direction refers to knowing one's position and spatial concept in a walking environment and using feeling, visual, and auditory senses. In this process, it refers to the process of moving through safer and real-time judgment. Visually impaired people are restricted in both direction and movement, which are elements of walking, due to their inability to see. Of course, it cannot be said that all visually impaired people are the same according to the degree of blindness, but many restrictions follow because they cannot analyze the surrounding environment and move safely by themselves. The gait of the blind person shows each gait difference according to the degree of blindness. The blind person from birth has no concept of recollection, whereas the severely blind person knows the existing spatial concept, so they think in their head. It has information you can understand. There is a difference between a congenital blind person who has a disability from the beginning and an acquired blind person caused by an accident or disease when walking and a difference in recognizing information. The non-blind person has a sense of space and judges it with their eyes, but the blind person can only perceive the extent to which they extend and close their hand.

At this time, in order to guide the way of the visually impaired, a method of guiding the way by connecting a terminal for the visually impaired and a counselor terminal has been researched and developed. Publication) and Korean Registered Patent No. 10-0952244 (published on April 9, 2010), the blind terminal and the counselor terminal are connected, and the shooting information and location information taken from the blind terminal are transmitted to the counselor terminal , By transmitting from the counselor's terminal to the terminal of the blind so that directions can be given, the surroundings of the visually impaired are filmed and relayed to the counselor's terminal via the server, and the visually impaired can receive detailed consultation on surrounding information from the counselor. Configuration, connecting the mobile communication terminal of the visually impaired to the route guidance call center, applying for route guidance service to the route guidance call center, providing the user's destination and current location information, and providing the user's destination and current location from the route guidance call center By guiding the user to the next movement location based on the information, configurations are disclosed that provide necessary information in real time, such as presence or absence of stairs, presence or absence of overpasses and crossings, and status of traffic lights.

However, in the former case, even if the video captured by the counselor terminal is viewed, the exact distance or direction in the video is not displayed, so it is impossible to accurately explain how many more steps to take to the desired location and which direction to go. Even in the latter case, the counselor cannot provide detailed information, such as how many more steps to go, even though it can inform the presence or absence of obstacles such as stairs and overpasses. Therefore, it is required to research and develop a system that allows a counselor to guide a user with an accurate distance and direction by combining and displaying spatial geographic information on a screen of a guidance terminal.

According to an embodiment of the present invention, when a user terminal and a wearable device are interlocked and a request for route guidance is detected by the user terminal, the user terminal and the guidance terminal are connected so that an image captured from the wearable device of the user terminal is provided to the guidance terminal in real time. It is streamed to be transmitted to, and spatial geographic information is displayed and provided based on GIS (Geographic Information System) so that it is overlaid on the image captured on the screen of the guide terminal, and the distance and direction to each object in the captured image is vector ( Vector) format, the guide terminal can help users not only walk, but also use kiosks, ticket offices, and public transportation, and help them find various things or carry out daily life not only outside but also inside the house. It is possible to provide a guide service providing system for the visually impaired using real-time streaming. However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, an embodiment of the present invention is a wearable device for photographing a preset direction, a user terminal for receiving a guide request and uploading an image captured from the wearable device in real time, and a user A guide terminal that is connected to the terminal's guidance request and outputs the direction and distance of each object on the screen as an overlay when outputting an image streamed from the wearable device on the screen, and a registration unit that maps and stores the user terminal and the wearable device so that they work together. , When a user terminal sends a guide request, a real-time relay unit that connects the guide terminal and the user terminal and streams the video captured by the wearable device to the guide terminal, and displays the direction and distance for each object on the screen of the guide terminal and a guide service providing server including a display unit for overlaying display, and a guide unit for guiding and transmitting the direction and distance of an object corresponding to a request of a user terminal in a guide terminal.

According to any one of the above-described problem solving means of the present invention, when the user terminal and the wearable device are interlocked and the user terminal detects a request for route guidance, the user terminal and the guidance terminal are connected to detect a photographed image from the wearable device of the user terminal. The video is streamed so that it is transmitted to the guide terminal in real time, and spatial geographic information is displayed and provided based on the Geographic Information System (GIS) so that it is overlaid on the video captured on the screen of the guide terminal. By displaying the distance and direction in a vector format, the guidance terminal enables the user to receive assistance not only for walking but also for using kiosks, ticket offices, and public transportation. We can help you out of trouble.

1 is a diagram for explaining a system for providing a guide service for the blind using real-time streaming according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating a guide service providing server included in the system of FIG. 1 .
3 and 4 are diagrams for explaining an embodiment in which a guide service for the visually impaired using real-time streaming according to an embodiment of the present invention is implemented.
5 is an operational flowchart illustrating a method of providing a guide service for the visually impaired using real-time streaming according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, this means that it may further include other components, not excluding other components, unless otherwise stated, and one or more other characteristics. However, it should be understood that it does not preclude the possibility of existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

As used throughout the specification, the terms "about", "substantially", etc., are used at or approximating that value when manufacturing and material tolerances inherent in the stated meaning are given, and do not convey an understanding of the present invention. Accurate or absolute figures are used to help prevent exploitation by unscrupulous infringers of the disclosed disclosure. The term "step of (doing)" or "step of" as used throughout the specification of the present invention does not mean "step for".

In this specification, a "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware. On the other hand, '~ unit' is not limited to software or hardware, and '~ unit' may be configured to be in an addressable storage medium or configured to reproduce one or more processors. Thus, as an example, '~unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Functions provided within components and '~units' may be combined into smaller numbers of components and '~units' or further separated into additional components and '~units'. In addition, components and '~units' may be implemented to play one or more CPUs in a device or a secure multimedia card.

In this specification, some of the operations or functions described as being performed by a terminal, device, or device may be performed instead by a server connected to the terminal, device, or device. Likewise, some of the operations or functions described as being performed by the server may also be performed by a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal mean mapping or matching the terminal's unique number or personal identification information, which is the terminal's identifying data. can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a system for providing a guide service for the blind using real-time streaming according to an embodiment of the present invention. Referring to FIG. 1 , a system 1 for providing a guide service for the visually impaired using real-time streaming includes at least one user terminal 100, a guide service providing server 300, at least one guide terminal 400, and at least one A wearable device 500 may be included. However, since the system 1 for providing a guide service for the visually impaired using real-time streaming of FIG. 1 is only one embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1 .

At this time, each component of FIG. 1 is generally connected through a network (Network, 200). For example, as shown in FIG. 1 , at least one user terminal 100 may be connected to a guidance service providing server 300 through a network 200 . Also, the guidance service providing server 300 may be connected to at least one user terminal 100 , at least one guidance terminal 400 , and at least one wearable device 500 through the network 200 . In addition, at least one guide terminal 400 may be connected to the guide service providing server 300 through the network 200 . Also, at least one wearable device 500 may be connected to at least one user terminal 100 , a guidance service providing server 300 , and at least one guidance terminal 400 through the network 200 .

Here, the network refers to a connection structure capable of exchanging information between nodes such as a plurality of terminals and servers, and examples of such networks include a local area network (LAN) and a wide area network (WAN: Wide Area Network), the Internet (WWW: World Wide Web), wired and wireless data communications networks, telephone networks, and wired and wireless television communications networks. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), 5th Generation Partnership Project (5GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi , Internet (Internet), LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth (Bluetooth) network, NFC ( A Near-Field Communication (Near-Field Communication) network, a satellite broadcasting network, an analog broadcasting network, a Digital Multimedia Broadcasting (DMB) network, etc. are included, but not limited thereto.

In the following, the term at least one is defined as a term including singular and plural, and even if at least one term does not exist, each component may exist in singular or plural, and may mean singular or plural. It will be self-evident. In addition, the singular or plural number of each component may be changed according to embodiments.

At least one user terminal 100 may be a terminal of a visually impaired user requesting guidance using a web page, app page, program, or application related to a guide service for the visually impaired using real-time streaming. To this end, the user terminal 100 is a user terminal that is linked with the wearable device 500 and transmits an image captured by the wearable device 500 to the guide terminal 400 via the guide service providing server 300 can be Also, the user terminal 100 may be a terminal that outputs the voice of the guidance terminal 400 through a speaker or an earphone.

Here, at least one user terminal 100 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser. In this case, at least one user terminal 100 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one user terminal 100 is, for example, a wireless communication device that ensures portability and mobility, navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet ) may include all types of handheld-based wireless communication devices such as terminals, smartphones, smart pads, tablet PCs, and the like.

The guide service providing server 300 may be a server that provides a guide service web page for the visually impaired using real-time streaming, an app page, a program, or an application. Also, the guidance service providing server 300 may be a server that maps and registers the user terminal 100 and the wearable device 500 . In addition, the guide service providing server 300 creates a channel in consideration of network resources and computing resources when creating a channel so that there is no delay in connection between the user terminal 100 and the guide terminal 400, and screen transmission is performed. It may be a server that prevents delay from occurring. In addition, the guidance service providing server 300 builds a GIS, stores each spatial information, vector information, and attribute information, sets at least one POI (Point of Interest) required for route guidance, and then sends the guidance terminal 400 It may be a server that guides POI information and object information to Text To Speech (TTS) along with a deep learning-based object detection model when the connection is delayed. In addition, the guide service providing server 300 may be a server that overlays and outputs GIS-based spatial information when the screen captured by the wearable device 500 is streamed in real time on the guide terminal 400. At this time, the guide service providing server 300 may be a server that overlays the distance and direction of each object in a vector format to prevent a situation in which spatial perception is lowered due to the guide not being on site.

Here, the guidance service providing server 300 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser.

At least one guide terminal 400 receives a guide request from the user terminal 100 using a web page, app page, program, or application related to a guide service for the visually impaired using real-time streaming, and controls the wearable device 500 Obtaining, while displaying the image transmitted from the wearable device 500 in real time, GIS-based spatial information if outdoors, and deep learning-based object detection model if indoors, Guidance can be provided by outputting information on each object on the screen It may be a terminal outputting a guide.

Here, at least one guide terminal 400 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser. At this time, at least one guide terminal 400 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one guide terminal 400 is, for example, a wireless communication device that ensures portability and mobility, and includes navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet ) may include all types of handheld-based wireless communication devices such as terminals, smartphones, smart pads, tablet PCs, and the like.

At least one wearable device 500 captures front, back, left and right or 360 degrees using a web page, app page, program or application related to a guide service for the visually impaired using real-time streaming and transmits it to the user terminal 100 can be a device. In this case, when the control right of the wearable device 500 is handed over to the guidance terminal 400, the wearable device 500 may be a device that adjusts a focus, viewing angle, etc. under the control of the guidance terminal 400.

Here, at least one wearable device 500 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser. In this case, at least one wearable device 500 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one wearable device 500 is, for example, a wireless communication device that ensures portability and mobility, and includes navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet ) may include all types of handheld-based wireless communication devices such as terminals, smartphones, smart pads, tablet PCs, and the like.

2 is a block diagram illustrating a guide service providing server included in the system of FIG. 1, and FIGS. 3 and 4 are implementations of a guide service for the visually impaired using real-time streaming according to an embodiment of the present invention. It is a drawing for explaining an example.

Referring to FIG. 2 , the guide service providing server 300 includes a registration unit 310, a real-time relay unit 320, a display unit 330, a guidance unit 340, a spatial information providing unit 350, and an object information providing unit. 360, a kiosk support unit 370, a depth estimation unit 380, a machine guide unit 390, and a view angle adjusting unit 391 may be included.

The guidance service providing server 300 according to an embodiment of the present invention or another server (not shown) operating in conjunction with at least one user terminal 100, at least one guidance terminal 400, and at least one wearable device In the case of transmitting a guide service application for the blind using real-time streaming to 500, a program, an app page, a web page, etc., at least one user terminal 100, at least one guidance terminal 400, and at least one wearable device (500), it is possible to install or open a guide service application for the visually impaired using real-time streaming, a program, an app page, a web page, and the like. Also, the service program may be driven in at least one user terminal 100 , at least one guidance terminal 400 , and at least one wearable device 500 by using a script executed in a web browser. Here, the web browser is a program that allows users to use the web (WWW: World Wide Web) service, and means a program that receives and displays hypertext described in HTML (Hyper Text Mark-up Language). For example, Netscape , Explorer, Chrome, and the like. In addition, an application means an application on a terminal, and includes, for example, an app running on a mobile terminal (smart phone).

Referring to FIG. 2 , the registration unit 310 may map and store the mapping so that the user terminal 100 and the wearable device 500 are linked. In this case, the wearable device 500 may be glasses, a body cam, a necklace or a hat, and may be worn on various parts of the body. However, it is not limited to those listed, and is not excluded for reasons not listed.

When a guide request is transmitted from the user terminal 100, the real-time relay unit 320 connects the guide terminal 400 and the user terminal 100 and transmits information captured by the wearable device 500 to the guide terminal 400. The video can be streamed. The wearable device 500 may photograph a preset direction. The user terminal 100 may receive a guidance request and upload an image captured from the wearable device 500 in real time. The guide terminal 400 may be connected to the guide request of the user terminal 100 . At this time, the wearable device 500 is a smart glass, and may be transmitted to the guidance terminal 400 in conjunction with a CCTV, etc. A detailed description thereof is provided in Korean Patent Registration No. 10-1929875 (2018 Notice on December 18th), so duplicate contents will not be described.

When uploading in real time, the real-time relay unit 320 collects network information such as the upload performance of the user terminal 100 in real time, since buffering or delay of video or audio may occur, and the encoding rate based on the network information. can be set. A typical ABR (Adaptive Bitrate Rate) streaming system provides real-time streaming by dividing video into files of small time units, encoding them in several resolutions, and transmitting the video according to the change in network available transmission rate experienced by each guide terminal 400. , In order to prepare the video in ABR format, the media server must transcode the video frame generated in real time, so a delay time of several seconds or more inevitably occurs, and the video is transmitted to the user terminal 100 and the guidance terminal 400 It is encoded at a predefined encoding rate without considering the network conditions connected to each other, so that the quality must be rapidly lowered frequently to guarantee low latency.

Accordingly, in one embodiment of the present invention, a structure in which a transmitter encodes real-time video at multiple encoding rates and transmits encoded frames to a media server in units of segments (Group of Pictures, GoP) may be used. In this case, in order to be compatible with the ABR streaming of the receiving end, the first frame of each segment is encoded as a key frame, and the media server can store the segment in a segment buffer and transmit it when requested by the receiving end. If a segment is stored beyond the target delay time, the media server drops the previous segment for low-latency transmission. Through this structure, low-latency transmission without transcoding can be realized. In addition to this, the media server on the system may calculate an optimized encoding rate for maximizing video quality by performing encoding. First, the media server estimates the available transmission amount collected from the receiving end, and clusters the estimated available transmission amount by the number of encoding rates through K-Means clustering. Finally, by determining the minimum value of the available transmission amount for each group as an encoding rate, it is possible to set an encoding rate that guarantees optimized video quality while guaranteeing low-latency performance.

Accordingly, a transmitter-based ABR encoding technique can be used to reduce a delay time caused by transcoding, and optimized video quality can be provided through an encoding rate calculation reflecting a network condition of a receiver. As a result, it is possible to optimize the video quality in real time according to the network conditions of multiple receivers while guaranteeing the end-to-end latency that occurred at the level of several seconds to less than 500 ms, which is essential for low-latency applications. Therefore, even if the video of the visually impaired is delayed by only a few seconds, it is possible to prevent accidents in which the blind person may fall from the railing of the subway or the guide cannot see and guide the passing car.

The display unit 330 may display the direction and distance of each object on the screen of the guidance terminal 400 in an overlay manner. When displaying an image streamed from the wearable device 500 on the screen, the guidance terminal 400 may overlay and output directions and distances for each object on the screen. At this time, after recognizing and identifying an object in the image, a basic concept of geometry is required to obtain the distance and direction from the camera to the object. What the user wants to know—that is, what the guide needs to be guided by—how far away each object actually is from the camera (the user), and what size, width, or height it is. Therefore, the problem to be solved is the problem of obtaining the corresponding 2D ground coordinates for the pixel coordinates, which are the coordinates of the object in the image. Once the 2D ground coordinates, which are the location of the object, are obtained, the size of the object can be easily calculated through a simple proportional formula. At this time, the assumption of this problem is that the height and direction of the camera are fixed, that is, if the user wears them as glasses, hangs them as a necklace, or attaches a body cam, the object is always attached to the ground. It is assumed that the parameters of the camera, for example, the distortion coefficient, height and angle of the lens, are known in advance, and the problem to be solved is to obtain 2D ground coordinates corresponding to the image coordinates of the object. At this time, three approaches are possible. In fact, the method of manually solving geometrically using a hand and a pencil is skipped because it needs to be calculated manually, and the method using HomoGraphy and the method using 3D transformation can be used. there is. Alternatively, in order to augment the recent AR content, the position and direction of each object is estimated and the AR content is overlaid on the object. This is applied to the guidance terminal 400 to determine how many meters the user is currently away from each object and in which direction. Whether it is located is displayed in a vector format, but it may be augmented with AR content for each object.

The guidance unit 340 may guide and deliver the direction and distance of the object corresponding to the request of the user terminal 100 in the guidance terminal 400 . For example, the handrail of a stairway can be explained based on the data indicated by the above-described image geometry so that it can tell in detail how many steps to go in which direction, whether it is right or left, diagonal direction, up, down, left, right, etc. . In addition, if you want to order a Big Mac set at McDonald's, how many centimeters should you move your finger on the current screen, where is the button to go to the next page, how many spans or how many fingers should you move, etc. can do it

When the location of the user terminal 100 is outdoors, the geospatial information providing unit 350 may overlay geographic information system (GIS)-based geospatial information on the screen of the guidance terminal 400 and output the result. . Since the start of the National Geographic Information System (NGIS) construction project, the government has been producing paper maps in the form of digital maps, which were produced in the form of digital maps 1.0 and 2.0. According to the Ordinance of the Ministry of Land, Infrastructure and Transport No. 209 「Digital Map Preparation Work Rules」, digital map 1.0 is a method for correcting or supplementing drawing data or map input data using data obtained from geographic surveys and field surveys. Refers to the map where the location editing work has been completed. In addition, digital map 2.0 refers to a digital map that has been structured and edited to compose geographic features that have been edited in place in order to grasp the geographical correlation between data. In particular, digital map 2.0 expresses topography and features based on layer items composed of 8 major categories (transportation, vegetation, topography, boundary, cycle, water system/ocean, culture/facility, and building/facility) and 104 sub-categories. On the other hand, necessary items of spatial information required to assist the safe movement and convenient walking of the walking weak, such as the visually impaired, can be derived by reviewing and analyzing related laws and integrating the survey results for the visually impaired. In addition, national geographic information in the form of a digital map is currently being produced in Korea, and through this analysis, it is possible to analyze and build items of spatial information necessary for movement for the visually impaired. The National Geographic Information Institute is producing digital maps by scale (ie, 1:1,000, 1:5,000, 1:25,000), and provides a legend table of standard codes for expressed topography and features. For safe and convenient walking of the walking weak, the spatial information items related to the purpose can be extracted from the digital topographic map topographical and feature standard codes provided by the National Geographic Information Institute notification digital topographic map creation work regulations.

In addition, in Korea, various databases created by performing national tasks are integrated and processed for the purpose of location-based services. The result is constructed in the form of information on points of interest (Point of Interest) and is currently serviced as a national Internet map of the National Geographic Information Institute. In the national internet map, it is possible to check not only the location of individual buildings but also their names. This is the result of building location and linkage names expressed in planar shapes as a database, so when using the data, it is possible to build spatial information for the walking handicapped without spending a lot of time and manpower in building the database. In addition, in Korea, public transportation information is linked and integrated in a standardized manner in accordance with Article 90, Paragraph 4 of the 「National Integrated Transport System Efficiency Act」 and Article 82, Paragraph 6 of the Enforcement Decree of the same Act for the purpose of providing national public transportation services. The Information Center (TAGO) is in operation. The center provides traffic information through information linkage with central government departments or local governments that produce information related to bus, rail, air, and shipping. In the case of city buses, the traffic information system operated by each local government is provided by linking to the center's homepage. Accordingly, route information of buses stopping at each bus stop is provided, and traffic information-related data managed by each local government can be used as attribute information on city bus stops and route numbers.

<nodes and links>

GIS-based geospatial information can layer various types of information on a map in layers, and when defining nodes that are POIs and link information connecting them, in the guidance terminal 400 When guiding, various information is displayed together, so the guiding is easy, and even if the guide is not connected right away, it explains where the user is currently passing because it explains each node, link, and what kind of POIs are on the way to the destination. Also good to understand. In addition, since the map produced by the country is used, there is almost no GPS error, so it can be an excellent guide in terms of accuracy. At this time, including the location and attribute information of the digital map 2.0 layer, the newly added layer may be summarized as shown in Table 1 below. Of course, GIS-based geospatial information can be modified, added, and transformed, so it is not limited thereto.

building Public facilities, government offices, social welfare facilities, medical facilities, schools, etc. underpass Underpass entrance, facilities for the disabled (elevator, etc.) city bus stop Route number, whether voice information is provided, whether braille information boards are installed Overpass Presence of walkable ramps and elevators subway station Elevator Wheelchair Lift Presence and location (linked to the entrance number) beacon Beacon type (manual or remote control), normal operation Pedestrian access road main entrance Adequacy of width and slope, elimination of height differences vehicle speed reduction Plateau intersection, crosswalk, speed bump installation location Disabled parking area Number of parking surfaces, elimination of height difference, appropriateness of slope (slope) Pedestrian Traffic Island Adequacy of installation location, width, presence or absence of bollard installation sidewalk Appropriateness of width and inclination, lowering of the chin, separation of roads, existence of pedestrian safety zones, existence of construction zones at vehicle entrances and exits traffic signal Appropriateness of signal button installation height, normal operation crosswalk Crosswalk installation protective fence Installation location and material braille block Presence and appropriateness of point-type and linear blocks lighting facility Artificial lighting installation facilities video information device Installation of CCTV and emergency button, availability of blind people vlad Whether or not to absorb shock, whether or not to install point-type blocks and their appropriateness

When objects existing on the surface are expressed on paper or digital maps, they represent each item, and are implemented within the spatial range of the relevant layer through a method of producing signs and symbols. In the case of digital maps, symbols related to layers are produced and used for each topography and feature. In addition, in the geographic information system environment, each topography and feature according to the scale can be defined in geometric form such as a point, line, and plane and expressed on a map for effective utilization of the spatial information layer. Pedestrian roads, which are places for pedestrian traffic, can be divided into pedestrian roads for mixed sidewalks and pedestrians, pedestrian roads exclusively for walking, road crossing facilities, and pedestrian roads connecting entrances and exits. Among the nodes and links shown in the detailed road network map manual for pedestrians produced by the National Geographic Information Institute, the detailed road network map for pedestrians was pilot-produced using the road name address base map, digital topographic map 2.0, and serial digital topographic map. Another type uses the national base map to construct and utilize a road-based network, which is maintained and updated through the annual national base map project. However, node properties include node information that intersects with roads, but items of tracks or railroads that intersect with roads are missing. So, a new layer for this layer can be created and added to GIS-based spatial information. Details for pedestrians Tables related to road network links are also composed of distribution and management. Attribute items of distribution information include pedestrian road type (WROAD_CATE), link facility information (LINK_FACIL), link type (LINK_CATE), walking restrictions (LIMIT), etc. Attributes of management information include construction method (W_TYPE) and update operation method (U_TYPE). At this time, the type of pedestrian path may be further subdivided, and pedestrian paths such as elevators and wheelchair lifts, which are not considered in the type of pedestrian path, may be coded and added. In addition, the link facility information code indicates information on facilities connected and installed for the purpose of allowing pedestrians to move from a specific place to another place. separated by However, it is necessary to add a separate code even in the case of temporary closure in which walking is temporarily prohibited due to reasons such as repairing or construction of a walking path to improve the walking environment. there is. In addition, the node properties can be added or modified so that route guidance for the visually impaired is possible by adding the line intersection, short road section, curved section, curb, sound signal code details, and rain gutter cover code of the node properties.

number code history Node setting standard One Pedestrian road intersection Point where pedestrian roads intersect 2 Attribute change point The point at which the properties of a pedestrian path change, such as the width of a sidewalk. 3 Pedestrian road facilities Starting and ending points of road crossing facilities such as crosswalks, overpasses, and underpasses, or road facilities (tunnels, bridges, overpasses, underpasses, etc.) 4 walking limit Points where pedestrian movement is physically controlled, such as highways or motorways 5 Administrative boundary crossing A point that is divided into city, county, and district boundaries 6 Pseudo Intersection Underground public sidewalk facilities consisting of subway or underpass entrances
Points that are not actually crossed by pedestrian roads, crosswalks, overpasses or underpasses, but appear to intersect when turning in two dimensions 7 Doyeop intersection A point that is divided by the map leaf boundary of the 2.0 map leaf on a digital topographic map at a scale of 1/5000 announced by the National Geographic Information Institute 8 railroad crossing The point where the pedestrian road and the track intersect 9 section of the curve Pedestrian path points with curves 10 rain gutter cover The point where the rain gutter cover is located on the pedestrian path 11 short road section The starting point and ending point of the short road section 12 curb Point where there is a curb on the pedestrian path 13 beacon The point where there is an acoustic signal on the pedestrian path

Even in the case of links, they are classified into sidewalks, pedestrian paths for mixed use of pedestrians, pedestrian paths for road crossing facilities, pedestrian paths connecting entrances and exits, pedestrian paths exclusively for walking, pedestrian paths for underground public sidewalks, and pedestrian paths for mixed bicycles. It is divided into general concrete, special concrete, concrete sidewalk block, soil concrete, rubber chip and resin mixture, wood chip and polyurethane, floor tile, unpaved and others so that the visually impaired do not slip because they cannot predict the material of the floor. build a database In the data model for walking route guidance for the visually impaired, since there is no need to consider wheelchair-related parts, it can be excluded and constructed by adding information related to risk factors in terms of walking route guidance.

<POI for the visually impaired>

The information on points of national interest constructed by the National Geographic Information Institute of the Ministry of Land, Infrastructure and Transport is divided into administration, education/health, public/environment, facilities, lodging/food, leisure/tourism/art, residence, industry, and linear topography/landmarks. In an embodiment of the present invention, a scheme for providing a customized POI only for the visually impaired can be used by analyzing the pre-established national POI information classification items and POI types. To this end, the layer of the offline braille map currently provided by the National Geographic Information Institute can be referred to, and POIs necessary for the visually impaired can be added to it. That is, it is possible to add additional subcategory items necessary for each individual, such as department stores, grocery stores, banks, ticket offices, jjimjilbangs, rest rooms, restaurants, and hospitals. In the data model for road guidance, pedestrian accessibility can be divided into walking space and walking additional information. The walking space is composed of pedestrian roads and landmarks, and the walking additional information may be composed of guidance and danger.

When the location of the user terminal 100 is indoors, the object information providing unit 360 recognizes and identifies each object in the image captured by the wearable device 500 with a deep learning-based object detection model, and then detects the user terminal ( 100), the distance and direction between the user and the object may be measured with a deep learning-based object detection model, and may be overlaid and output on the screen of the guidance terminal 400 in a vector format. Even if the above-described GIS-based spatial information is established, the GIS-based spatial information cannot be used when entering the room. Of course, national institutions such as various public institutions may have built GIS-based spatial information, but GPS is not well captured indoors, so the location may not be accurate. Even if the building ledger is downloaded, structures other than reinforced concrete, such as interior temporary walls, are unknown and are not included in GIS spatial information. Of course, the deep learning-based object detection model can also be used outdoors.

<object recognition>

Object Detection, which aims to reproduce human visual abilities through artificial intelligence, analyzes the given image to determine the location and class of a specific object when a specific image is input. is to do The object detection algorithm first needs to find the localization of an object in an image, and then classifies what kind of object the object at that location is. Various methods have been tried to perform this, but it is narrowed down to a single-stage (1-Stage) detection method and a two-stage (2-Stage) detection method. The single-step detection method simultaneously performs position detection and classification for all regions. The two-step detection method performs coarse position detection and then performs classification on selected candidates. Since single-step detection simultaneously detects and classifies objects in all areas, it is faster but has a disadvantage that its accuracy is slightly lower than that of the two-step detection method. On the contrary, the two-stage detector has relatively high accuracy but low speed because object position detection and classification are performed sequentially. In one embodiment of the present invention, both methods are used, but a single-step detection method can be used in an urgent case, and a two-step detection method can be used when detailed instructions are required even if it takes time.

When the image transmitted from the user terminal 100 is an image of the kiosk, the kiosk support unit 370 requests the guide terminal 400 to describe the screen within the kiosk, and selects an item requested by the user terminal 100. The direction and distance between the user's finger and the requested item may be displayed on the screen of the guidance terminal 400 to provide voice guidance to the user terminal 100 .

The depth estimator 380 may estimate a depth from an image transmitted from the user terminal 100 to at least one preset point of interest (POI) and provide a distance to the at least one POI. . At this time, in order to estimate the depth with a single lens rather than a stereo camera, the image geometry as described above can be used, but in the case of a stereo camera, that is, in the case of two or more cameras, a 3D camera module technology can be used, which is Computational Camera It is a technology that can calculate the depth value of each pixel of an image, which was not possible in the existing 2D. Using this, the depth can be estimated and the distance from the user's location to each POI can be estimated. If the guide is not on-site and there is no sense of distance while only watching the video, if the depth is estimated, the guide will take a few steps to the user to the ticket office. Detailed guidance such as whether you can go is possible.

When an immediate connection between the user terminal 100 and the guidance terminal 400 is not possible, the machine guidance unit 390 receives and captures images captured by the wearable device 500 while being connected to the guidance terminal 400 by streaming. A description of at least one object included in the image may be automatically generated and output as a voice in the user terminal 100 . Using the above-described deep learning or GIS-based spatial information, guidance may be provided until the machine is connected to the guide when the guide is not immediately connected, or during call waiting or connection.

When the user terminal 100 and the guidance terminal 400 are interlocked, the view angle adjusting unit 391 controls the guidance terminal 400 to increase or decrease the field of view of the camera in the wearable device 500. access rights can be provided. The wearable device 500 may be a device including a camera that performs at least one of front, rear, left, right, and 360 degree shooting. The camera can adjust the angle of view or focus. If the focus is not good, the guide at a remote location may see a blurry image, and if the angle of view is narrowed, it must be widened, but this is can't expand it any further. Accordingly, by handing over control of the camera to the guide, the guide may widen the viewing angle of the camera to see a wider area, or change a blurred image into a clear image by focusing.

Hereinafter, an operation process according to the configuration of the information service providing server of FIG. 2 described above will be described in detail with reference to FIGS. 3 and 4 as examples. However, it will be apparent that the embodiment is only any one of various embodiments of the present invention, and is not limited thereto.

Referring to FIG. 3, (a) the information service providing server 300 maps and stores the two devices through linkage between the user terminal 100 and the wearable device 500, and, as shown in (b), GIS-based spatial information When there is a guide request from the user terminal 100 as shown in (c), the guide terminal 400 is connected and the image transmitted from the wearable device 500 is transmitted to the guide terminal 400. At this time, as shown in (d), the GIS-based spatial information and the deep learning-based object detection model can be used together indoors and outdoors, and the object detected by the deep-learning-based object detection model indoors is displayed on the video streaming screen of the guidance terminal 400. By displaying, layers can be provided so that guidance on which object is in front, how far or near it is, and where the direction is, etc. can be made in detail.

In addition, by providing spatial information as shown in FIG. 4 (a), matters that cannot be seen in the guide terminal 400, for example, whether the guide braille on the road is broken, whether there is a chin, what the material of the floor is, etc. can be guided by the guide terminal 400, and by allowing the guide terminal 400 to control the angle of view or focus of the wearable device 500 as shown in (b), it is out of focus or the angle of view is widened or allow it to be narrowed down. In addition, in the case of a kiosk, etc., as in (c), it is possible to guide the user how much and where to move his or her finger to press a desired button, etc., and estimate the distance between each object and the user through POI depth estimation as in (d) After that, information can be provided so that guidance can be provided to the guidance terminal 400 .

Matters not described for the method of providing a guide service for the visually impaired using real-time streaming of FIGS. 2 to 4 are the same as or described for the method of providing a guide service for the visually impaired using real-time streaming through FIG. 1 above. Since it can be easily inferred from the contents, the following description will be omitted.

5 is a diagram illustrating a process of transmitting and receiving data between components included in the system for providing a guide service for the visually impaired using real-time streaming of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of a process of transmitting and receiving data between each component will be described through FIG. 5, but the present application is not limited to such an embodiment, and according to various embodiments described above, It is obvious to those skilled in the art that a process of transmitting and receiving data may be changed.

Referring to FIG. 5, the guidance service providing server maps and stores the information so that the user terminal and the wearable device are interlocked (S5100), and when the user terminal transmits a guidance request, connecting the guidance terminal and the user terminal to the guidance terminal. An image captured by the wearable device is streamed (S5200).

In addition, the guide service providing server displays the direction and distance of each object on the screen of the guide terminal in an overlay manner (S5300), and guides and delivers the direction and distance of the object corresponding to the request of the user terminal in the guide terminal (S5300). S5400).

The order between the above-described steps (S5100 to S5400) is only an example, and is not limited thereto. That is, the order of the above-described steps (S5100 to S5400) may be mutually changed, and some of the steps may be simultaneously executed or deleted.

Matters not described in the method for providing a guide service for the visually impaired using real-time streaming in FIG. 5 are the same as or described for the method for providing a guide service for the visually impaired using real-time streaming through FIGS. 1 to 4 above. Since it can be easily inferred from the contents, the following description will be omitted.

The method of providing a guide service for the visually impaired using real-time streaming according to an embodiment described with reference to FIG. 5 is implemented in the form of a recording medium including instructions executable by a computer, such as an application or program module executed by a computer. It can be. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The above-described method for providing a guide service for the visually impaired using real-time streaming according to an embodiment of the present invention is based on an application (this may include a program included in a platform or operating system, etc., which is basically installed in the terminal) It can also be executed by an application (i.e., a program) that a user directly installs in the master terminal through an application providing server such as an application store server, an application or a web server related to the corresponding service. In this sense, the above-described method for providing a guide service for the visually impaired using real-time streaming according to an embodiment of the present invention is implemented as an application (ie, a program) that is basically installed in a terminal or directly installed by a user, and is installed on a computer such as a terminal. can be recorded on a readable recording medium.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

Claims (7)

a wearable device that captures a preset direction;
a user terminal receiving a guide request and uploading an image taken from the wearable device in real time;
a guide terminal that is connected to the guide request of the user terminal and outputs an overlay of directions and distances for each object in the screen when outputting an image streamed from the wearable device to the screen; and
A registration unit that maps and stores the mapping so that the user terminal and the wearable device are interlocked, and when the user terminal transmits a guide request, the guide terminal and the user terminal are connected, and the video taken by the wearable device is streamed to the guide terminal a real-time relay unit for providing information, a display unit for overlaying display of the direction and distance of each object on the screen of the guidance terminal, and a guide unit for guiding and transmitting the direction and distance of the object corresponding to the request of the user terminal in the guidance terminal. And when the location of the user terminal is indoors, after recognizing and identifying each object in the image captured by the wearable device with a deep learning-based object detection model, the distance and direction between the user of the user terminal and the object are determined by the deep learning. a guide service providing server including an object information providing unit that measures measurements using a running-based object detection model and outputs the overlay on the screen of the guide terminal in a vector format;
A guide service providing system for the visually impaired using real-time streaming including a.
According to claim 1,
The guidance service providing server,
a geospatial information provider configured to overlay geographic information system (GIS)-based geospatial information on the screen of the guide terminal and output the result when the location of the user terminal is outdoors;
A system for providing a guide service for the visually impaired using real-time streaming, characterized in that it further comprises.
delete According to claim 1,
The guidance service providing server,
When the image transmitted from the user terminal is an image of a kiosk, the guide terminal is requested to describe the screen in the kiosk, and the direction and distance between the user's finger and the requested item are determined so that the requested item is selected from the user terminal. a kiosk support unit that displays on the screen of the guide terminal and supports voice guidance to the user terminal;
A system for providing a guide service for the visually impaired using real-time streaming, characterized in that it further comprises.
According to claim 1,
The guidance service providing server,
a depth estimation unit estimating a depth from the image transmitted from the user terminal to at least one predetermined POI (Point of Interest) and providing a distance to the at least one POI;
A system for providing a guide service for the visually impaired using real-time streaming, characterized in that it further comprises.
According to claim 1,
The guidance service providing server,
When an immediate connection between the user terminal and the guide terminal is not possible, while being connected to the guide terminal, an image captured by the wearable device is streamed and a description of at least one object included in the captured image is automatically provided. a machine guidance unit that generates and outputs a voice in the user terminal;
A system for providing a guide service for the visually impaired using real-time streaming, characterized in that it further comprises.
According to claim 1,
The wearable device,
A device including a camera that performs at least one of front, rear, left, right, and 360-degree shooting;
The guidance service providing server,
an angle-of-view adjusting unit for providing access to a control right to increase or decrease a field of view of a camera in the wearable device in the guidance terminal when interworking between the user terminal and the guide terminal is established;
A system for providing a guide service for the visually impaired using real-time streaming, characterized in that it further comprises.

Images