WO2014017719A1

WO2014017719A1 - Method and system for guiding visually impaired person using multi-view camera having parallel image filtering module, and computer-readable recording medium

Info

Publication number: WO2014017719A1
Application number: PCT/KR2013/001040
Authority: WO
Inventors: 박영성; 한동숭
Original assignee: 전주대학교 산학협력단
Priority date: 2012-07-25
Filing date: 2013-02-08
Publication date: 2014-01-30
Also published as: KR101353860B1

Abstract

A method for guiding a visually impaired person using a multi-view camera having a parallel image filtering module, according to one aspect of the present invention, comprises the steps of: (a) extracting a common feature point from images respectively inputted from a plurality of cameras;(b) calculating a relative distance between the feature point and an observer; and (c) generating an alarm having stimulation corresponding to the relative distance between the feature point and the observer.

Description

Visually impaired guide method, system and computer readable recording medium using multi-view camera with parallel image filtering module

The present invention relates to a visually impaired guide method, a system and a computer-readable recording medium using a multi-view camera having a parallel image filtering module, and more particularly, to a distance from an obstacle located in front of a plurality of cameras and The present invention relates to a visually impaired guide method, a system and a computer-readable recording medium using a multi-view camera equipped with a parallel image filtering module that detects a direction and notifies the observer, so that even a blind person can walk independently.

Recently, high-performance digital cameras with a large number of pixels have been developed, small CCD cameras have not only become widespread, but also high-performance computing devices and increasingly smaller and faster memories can process large amounts of image data more quickly and accurately. Image processing technology is rapidly evolving.

In accordance with this trend, an analysis algorithm for multi-view images taken at the same time has also been actively researched and developed in various fields. As a representative example, technologies related to 3D movies, 3D cameras, 3D TVs, etc., using two or more cameras to restore 3D data of a target object from a video, or to synthesize a virtual object in an image, have been actively studied.

As described above, a technology for providing a multi-view image by mounting a plurality of cameras in a single device has been disclosed through the August 8, 2008 publication, No. 10-2008-0073073, titled 'Dual Camera Module'. Referring to the above technique, an apparatus and method for efficiently communicating with two image sensors implemented in a dual camera module used for capturing near and far images are disclosed. However, the above prior art does not disclose a specific application of the image processing algorithm using a dual camera.

As such, in recent years, the necessity of developing a system for application research of an image processing algorithm as well as an image processing algorithm itself for a multi-view image photographed at the same time is increasing.

The present invention aims to solve all the above-mentioned problems.

In addition, the present invention obtains a multi-view image using a plurality of cameras, by using the same to obtain the position and direction of the obstacle located in front of the observer to provide to the observer so that the observer can walk independently when the visually impaired It is an object of the present invention to provide a visually impaired guide method, a system and a computer-readable recording medium.

Another object of the present invention is to provide an image processing algorithm capable of processing a large amount of image data more quickly and accurately by connecting a plurality of image filtering modules in parallel.

Representative configuration of the present invention for achieving the above object is as follows.

According to an aspect of the present invention, a method for guiding a visually impaired person using a multi-view camera having a parallel image filtering module, comprising: (a) extracting a common feature point from each input image from a plurality of cameras; Calculating a relative distance between the feature point and the observer, and (c) generating an alarm having a stimulus corresponding to the relative distance between the feature point and the observer.

According to another aspect of the present invention, there is provided a visually impaired guide system using a multi-view camera having a parallel image filtering module, wherein a plurality of cameras and a common feature point are extracted from each input image from the camera, A system is provided that includes a central processing unit for calculating a relative distance of and an alarm device for generating an alarm having a stimulus corresponding to the relative distance between the feature point and the observer.

In addition, there is further provided a computer readable recording medium for recording another method, system for implementing the present invention and a computer program for executing the method.

According to the present invention, by using a plurality of cameras to detect the distance and the direction of the obstacle located in front and to notify the observer can be able to walk independently even in the case of the visually impaired.

According to the present invention, a plurality of image filtering modules may be connected in parallel to process a large amount of image data more quickly and accurately.

1 is a block diagram illustrating a visually impaired guide system using a multi-view camera having a parallel image filtering module according to an embodiment of the present invention.

2 is a flowchart illustrating an object searching method using a multiview camera having a parallel image filtering module according to an embodiment of the present invention.

FIG. 3 is a diagram for describing a method of calculating a relative distance between a feature point and an observer common in an object searching system using a multiview camera having a parallel image filtering module according to an embodiment of the present invention.

100: camera

110: image memory

200: spacer

300: image filtering module

310: graphics memory

320: graphics computing device

400: central processing unit

410: central operation memory

420: central processing unit

430: persistent storage

440: network communication device

450: alarm device

DETAILED DESCRIPTION The following detailed description of the invention refers to the accompanying drawings that show, by way of illustration, specific embodiments in which the invention may be practiced. These embodiments are described in sufficient detail to enable those skilled in the art to practice the invention. It should be understood that the various embodiments of the present invention are different but need not be mutually exclusive. For example, certain shapes, structures, and characteristics described herein may be embodied in other embodiments without departing from the spirit and scope of the invention with respect to one embodiment. In addition, it is to be understood that the location or arrangement of individual components within each disclosed embodiment may be changed without departing from the spirit and scope of the invention. The following detailed description, therefore, is not to be taken in a limiting sense, and the scope of the present invention, if properly described, is defined only by the appended claims, along with the full range of equivalents to which such claims are entitled. Like reference numerals in the drawings refer to the same or similar functions throughout the several aspects.

DETAILED DESCRIPTION Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those skilled in the art can easily implement the present invention.

[Preferred Embodiments of the Invention]

Configuration of the entire system

Referring to FIG. 1, the visually impaired guide system includes a plurality of cameras 100, a gap adjusting device 200 for adjusting a gap between the cameras 100, and a plurality of cameras connected in parallel with each other. The image filtering module 300 may include a central processing unit 400 interoperating with the image filtering module 300.

In more detail, the camera 100 may include an image memory 110 for temporarily storing an image captured therein, and the image filtering module 300 may include at least one graphic memory 310 and a graphic. And a central processing unit (400), wherein the central processing unit (400) includes a central processing unit (410), a central processing unit (420), a persistent storage unit (430), and a network communication unit (440). ), And an alarm device 450.

Here, the visually impaired guide system according to an embodiment of the present invention is a digital device, and includes a microprocessor and a memory means such as a personal computer (for example, a tablet computer, a laptop computer, etc.), a PDA, a web pad, a mobile phone, and the like. Any portable digital device equipped with a computing power may be adopted as a visually impaired guide system according to the present invention. More preferably, the portable digital device may be manufactured in the form of accessories such as glasses or necklaces so that the visually impaired person can easily equip and move.

The camera 100 according to an embodiment of the present invention may be a digital camera 100 configured as a conventional general imaging device, for example, using a CCD and generating a digital image signal. Image information input through the camera 100 may be temporarily stored in the image memory 110 of the camera 100, and may be stored in the center through a plurality of image filtering modules 300 connected in parallel with the camera 100. May be sent to the processing device 400.

Next, the interval adjusting device 200 according to an embodiment of the present invention adjusts the interval between the cameras 100 to obtain a common feature point from each image input from the camera 100. Performs a function that enables extraction. That is, in the multi-view camera system, if the distance between the cameras 100 is sufficiently small, the distance between the respective cameras 100 is increased by using a possibility that a common feature point is extracted from an image input from each camera 100. By adjusting the common feature point can be extracted. The feature point may be a portion of an obstacle located in front of the observer.

Next, the image filtering module 300 according to an embodiment of the present invention may include at least one graphic memory 310 and a graphic computing device 320 therein, and the image filtering module 300 may include It is connected in parallel with the camera 100 and performs a function of receiving and filtering image information photographed using the camera 100 from the image memory 110. In more detail, the graphic memory 310 may receive and load a filtering program to be used for image filtering from the CPU 400, and the graphic computing device 320 may use the filtering program to load the camera 100. Performs a filtering operation on the image information input from the. This is similar to the shader program of the 3D graphics card used in the current computer system, but the computation process in the graphics card shows a one-way characteristic of unilaterally transmitting information from the central processing unit to the graphics card, The computing device 320 may transmit and receive filtered image information by communicating with the central processing unit 400 in a bidirectional manner.

Meanwhile, the image filtering module 300 is connected to the camera 100 in parallel and performs a function of processing two or more image information in real time. The filtering operation through the image filtering module 300 requires various algorithms or various threshold processing according to the information to be extracted from the input image, and the time required for individual image processing is large so that two or more pieces of information can be simultaneously extracted. There is a limit. In particular, in order to process an image in real time, a faster image processing speed is required. Therefore, according to an exemplary embodiment of the present invention, the plurality of image filtering modules 300 may be connected to the camera 100 in parallel to satisfy a speed required for real-time image processing.

According to an embodiment of the present disclosure, the graphic memory 310 and the graphic computing device 320 may be program modules in which at least some of them are in communication with the camera 100 or the central processing unit 400. Such program modules may be included in the image filtering module 300 in the form of an operating system, an application module, and other program modules, and may be physically stored on various known storage devices. In addition, such program modules may be stored in a remote storage device that can communicate with the image filtering module 300. On the other hand, such program modules include, but are not limited to, routines, subroutines, programs, objects, components, data structures, etc. that perform particular tasks or execute particular abstract data types, described below, in accordance with the present invention.

Finally, the central processing unit 400 according to the embodiment of the present invention includes a central processing unit 410, a central processing unit 420, a persistent storage unit 430, a network communication unit 440, and an alarm unit. 450, and performs a function of controlling the flow of data between the camera 100 and the image filtering module 300. That is, the central processing unit 400 controls the flow of data to and from the outside or between the components of the visually impaired guide system, so that the camera 100 and the image filtering module 300 each have unique functions. Control to perform.

In more detail, the central processing memory 410 is connected in parallel with the image filtering module 300 to store image information transmitted from the plurality of image filtering modules 300, and each processed and stored image. Loads a program that can aggregate information and extract other information.

The central processing unit 420 aggregates a plurality of image information input through a plurality of cameras 100 and filtered through the image filtering module 300, and is located in front of an observer. Calculates the position and direction of the camera, and configures the visually impaired guide system 100, the image filtering module 300, the central processing unit 410, the persistent storage device 430, the network in the central processing unit 400 It performs a function of controlling the communication device 440, the alarm device 450 and the like.

The persistent storage device 430 stores the data processed by the central processing unit 420.

The network communication device 440 performs a function of receiving data from the outside or transmitting data to the outside through a communication network. In one embodiment, the network communication device 440 may receive an externally created program through the communication network. According to an embodiment of the present invention, the communication network may be configured as an aspect of wireless communication and may be configured with various communication networks such as a mobile communication network and a satellite communication network. More specifically, the communication network according to the present invention includes a concept including all known wireless LAN (WLAN), code division multiple access (CDMA), wideband code division multiple access (WCDMA), global system for mobile communications (GSM), and the like. It should be understood that

The alarm device 450 functions to generate an alarm having a stimulus corresponding to the relative distance according to the relative distance between the obstacle and the observer calculated by the central processing unit 420. In addition, the alarm device 450 performs a function of notifying the observer of the direction in which the obstacle is located.

According to an embodiment of the present invention, the central operation memory 410, the central processing unit 420, the persistent storage device 430, the network communication device 440, and the alarm device 450 is at least a portion thereof The image filtering module 300 may be program modules communicating with an external system. Such program modules may be included in the central processing unit 400 in the form of operating systems, application modules, and other program modules, and may be physically stored on a variety of known storage devices. Such program modules may also be stored in a remote storage device that can communicate with the central processing unit 400. On the other hand, such program modules include, but are not limited to, routines, subroutines, programs, objects, components, data structures, etc. that perform particular tasks or execute particular abstract data types, described below, in accordance with the present invention.

Operation of the entire system

Referring to FIG. 2, the guide method of the visually impaired person of the present invention photographs an image using a plurality of cameras 100 (S100), and extracts feature points from the photographed images (S110). The feature point refers to a predetermined portion of a subject photographed using the camera 100 and may be set to, for example, a portion protruding most toward the observer of the subject or a point closest to the observer. In each of the images photographed by the plurality of cameras 100, it is determined whether a common feature point exists (S120), and when there is no common feature point, the camera (eg, using the interval adjusting device 200) is used. Adjust the interval between 100) (S130). For example, if it is determined that the distance between the cameras 100 is too far to search for a common feature point, the distance between the two cameras 100 is narrowed, and then the process returns to step S100 to again use the plurality of cameras 100. The video will be taken. In extracting the common feature points, a SIFT or SURF algorithm may be used.

The Scale Invariant Feature Transform (SIFT) algorithm was proposed by David G. Lowe, which extracts feature points generated from edges or vertices of objects as vectors, and provides excellent matching for deformations due to image size changes, rotations, and light changes. Has performance. The SIFT Descriptor maintains constant feature matching even with image transformation, scale, rotation, and noise effects. The feature extraction process using SIFT can be divided into two stages: finding a feature point and generating a descriptor of the selected feature point. First, in the step of extracting the feature points, the candidate feature points in the scale space are extracted and the stability of the candidate feature points is examined to correct the positions of the stable feature points to the detailed positions. In the generating of the descriptor, the direction component is obtained through the gradient of the peripheral area around the selected points, and the descriptor is generated by resetting the region of interest based on the obtained direction component.

The Speeded Up Robust Features (SURF) algorithm is an algorithm for extracting feature points that are robust to changes in size and rotation, like the SIFT algorithm, and is used for object recognition, panorama image generation, and 3D image reconstruction. The SURF algorithm generates image pyramids and extracts feature points from the generated image pyramids to represent a scale space for extracting feature points robust to a change in size.

If a common feature point is extracted from the images captured by the plurality of cameras 100, the relative distance between the feature point and the observer is calculated using the common feature point (S140). A method of calculating the relative distance between the feature point and the observer will be described in more detail with reference to FIG. 3. Next, the alarm device 450 of the central processing unit 400 generates an alarm having a stimulus corresponding to the relative distance between the feature point and the observer based on the calculated distance (S150). Examples of the alarm may include a loudness or an intensity of vibration, and the closer the feature point is to the observer, the louder the sound or the stronger the vibration so as to inform the observer of the distance to the obstacle. More preferably, the observer may be notified of the direction of the feature point using the x and y coordinate values of the feature point obtained as described with reference to FIG. 3 below (S160).

Referring to FIG. 3, assuming that the first camera and the second camera are located at a distance d from the origin, the coordinate F of the first camera is (u, v, w) and the coordinate F of the second camera. If '(u', v ', w') and the coordinate of the feature point P is (x, y, z), the feature point P is located within the shooting radius of the fan shape of the first and second cameras. May be a common feature point in images captured by the first and second cameras.

In the image photographed using the first camera, the feature point P forms an image at (px, py) spaced apart by a focal length f of the first camera, and similarly the feature point P in the image photographed using the second camera. The image is formed at (p'x, p'y) separated by the focal length f 'of the second camera.

Using px, py, p'x, and p'y, the coordinates (x, y, z) of the feature point P can be obtained. In more detail,

By using the above equation it is possible to calculate the relative coordinates of the feature point.

If the origin is the position of the observer, the relative coordinates of the feature point and the minimum distance of the origin can be calculated to calculate the relative position between the observer and the obstacle.

Embodiments according to the present invention described above may be implemented in the form of program instructions that may be executed by various computer components, and may be recorded in a computer-readable recording medium. The computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the computer-readable recording medium may be those specially designed and configured for the present invention, or may be known and available to those skilled in the computer software arts. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tape, optical recording media such as CD-ROMs, DVDs, and magneto-optical media such as floptical disks. media), and hardware devices specifically configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include not only machine code generated by a compiler, but also high-level language code that can be executed by a computer using an interpreter or the like. The hardware device may be configured to operate as one or more software modules to perform the process according to the invention, and vice versa.

Although the present invention has been described by specific embodiments such as specific components and the like, but the embodiments and the drawings are provided to assist in a more general understanding of the present invention, the present invention is not limited to the above embodiments. For those skilled in the art, various modifications and variations can be made from these descriptions.

Accordingly, the spirit of the present invention should not be limited to the above-described embodiments, and all of the equivalents or equivalents of the claims, as well as the appended claims, fall within the scope of the spirit of the present invention. I will say.

Claims

A visually impaired guide method using a multi-view camera with a parallel image filtering module,

(a) extracting a common feature point from each input image from a plurality of cameras;

(b) calculating a relative distance between the feature point and the observer; And

(c) generating an alert having a stimulus corresponding to the relative distance between the feature point and the observer.
The method of claim 1,

Step (a) is characterized in that to extract a common feature point using a SIFT or SURF algorithm.
The method of claim 1,

In step (b),

And calculating a relative coordinate of the feature point using the equation.
The method of claim 3,

and (d) informing the observer of the direction of the feature point obtained using the x and y coordinate values.
The method of claim 1,

And a plurality of image filtering modules connected in parallel to each other including a graphic memory and a graphic computing device to filter images input from the camera in cooperation with the camera.
The method of claim 1,

The camera is characterized in that to extract a common feature point from each of the images input from the camera by adjusting the interval between each camera using a spacing device for adjusting the interval between each camera.
A visually impaired guide system using a multi-view camera with a parallel image filtering module,

A plurality of cameras;

A central processing unit extracting a common feature point from each input image from the camera and calculating a relative distance between the feature point and an observer; And

An alarm device for generating an alarm having a stimulus corresponding to the feature point and the relative distance to the observer.
The method of claim 7, wherein

And the central processing unit extracts common feature points using SIFT or SURF algorithm.
The method of claim 7, wherein

The central processing unit,

And calculate a relative coordinate of the feature point using the equation.
The method of claim 7, wherein

And the alerting device notifies the observer of the direction of the feature point obtained using the x and y coordinate values.
The method of claim 7, wherein

Further comprising a plurality of image filtering modules connected in parallel to each other including a graphics memory and a graphics computing device,

The image filtering module is configured to filter an image input from the camera in cooperation with the camera.
The method of claim 7, wherein

Further comprising a spacing device for adjusting the spacing between the camera,

And adjusting a gap between the cameras using the gap adjusting device to extract a common feature point from each of the images input from the cameras.
A computer-readable recording medium having recorded thereon a computer program for executing the method according to any one of claims 1 to 6.