JP2005266014A - Image recognition device for visually handicapped person, method and program thereof - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To enhance accuracy of automatic recognition by restricting candidates for a recognized object and to provide a visually handicapped person with support information for stable walk by an acoustic signal, etc. <P>SOLUTION: This image recognition device 10 for visually handicapped person is provided with a multi-spectrum image sensor 20, a posture sensor 30, an image recognition processing means 40, a processing mode control (compulsive switching) means 50, an acoustic signal generation means 60 and an acoustic signal reproduction means 70. The image recognition processing means 40 is provided with a processing mode setting part 41 which selects the optimal processing mode, an image recognition part 42, a radon conversion means 43, a staircase recognition means 44, a neural network 45 and a support information output means 46. The image recognition part 42 performs recognition processing of an image in the selected processing mode. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to an image recognition apparatus for visually impaired persons, and more particularly to a portable image recognition apparatus for assisting visually impaired persons in walking on the street in the field of welfare engineering.

In the conventional visually impaired guidance device, the IC chip is embedded in the blind plate, the electromagnetic wave source is embedded in the walking path, or a guide wire of a different color is installed for each destination. For example, a device that provides walking position information to a recognition device carried by a pedestrian has been mainly used (see Non-Patent Document 1). Since the area where these walking support environments can be established is limited to a very narrow area, when a visually impaired person walks on a general street, assistance by a caregiver or assistance by a guide dog with a limited number of people remains. Was necessary.
On the other hand, in the automatic image recognition technology on the street, most of the objects are recognition of vehicle numbers (see non-patent document 2) and road signs whose shapes and colors are unified (see non-patent document 3). ) And various obstacles that visually impaired people encounter when walking on general streets (for example, bicycles placed on the plate for blind people, barricades under construction, street steps, stairs and homes) It has been considered that the automatic image recognition technology is hardly applicable to steps, traffic lights at intersections, and the like.
Y. Kaneko, T. Harada, Y. Hirahara et al: Development of the Navigation System for the Visually Impaired, Proc.IEEE APBME # 5.2.3, (Kyoto 2003) Hiroyoshi Fujiyoshi, Taizo Umezaki, Tomohiko Imamura, Takeo Kanade: License plate position detection by neural network, IEICE Transactions, Vol.J80-D-II, No.6, pp 1627-1634 (1997) ) M. Nakamura, S. Kodama, N. Kitamura and M. Umeno: Vision Supporting System for Car Driver using Neural Network, Proc. ICONIP98, pp.1082-1085 (1998)

That is, in the conventional apparatus and technique, when identifying an object that obstructs the walking of visually impaired persons and an object that helps walking, from the image information obtained from the image acquisition apparatus carried by the pedestrian, the recognition target object However, it was difficult to construct a stable recognition mechanism.
On the other hand, pedestrians with normal visual functions, when looking at their feet, looking horizontally or sideways, aim their eyes at the target and at the same time use the image recognition mechanism in the brain for the object. Switching to a processing system suitable for recognition ensures high recognition ability. For example, in the mode of looking at the feet, the process with the intention to determine the color of the traffic light is not performed. On the other hand, when the field of view is required far away, the process of recognizing the hole at the foot is suppressed.

  In the present invention, the above-described human image recognition system is used as a handheld shape that allows a visually impaired person to freely change the object direction of the image acquisition system, and a posture sensor (three-dimensional) added to the image acquisition system. A priori information relating to the shooting situation, such as from which direction the target image was acquired, and in a stable still state, is provided to the recognition system, and the recognition system is obtained based on this a priori information. The image processing mode is changed, a more accurate recognition result is generated, and the result is provided to the visually impaired by an acoustic signal or the like.

An image recognition apparatus for a visually impaired person according to the present invention includes:
A multispectral image sensor that simultaneously acquires a plurality of images each having a different wavelength band; and
An attitude sensor (for example, a three-dimensional acceleration sensor, an inclination angle sensor, etc.) for acquiring attitude information of the device (that is, a shooting direction of the image sensor);
Based on the posture information acquired by the posture sensor (that is, the angle formed between the optical axis of the image sensor and the horizontal plane in the apparatus), an image having a wavelength band that includes the wavelength of light emitted by the traffic signal sign A traffic signal sign recognition processing mode for performing image recognition processing, an obstacle recognition processing mode for performing normal image recognition processing, and a road surface recognition processing mode for performing image recognition processing limited to the center of each of the plurality of images Image recognition processing means for selecting one of a plurality of processing modes including, and performing recognition processing of at least one image of the plurality of images in the selected processing mode;
Acoustic signal generating means for generating an acoustic signal according to the result of the image recognition processing by the image recognition processing means;
Sound reproduction means (speakers, earphones, etc.) for reproducing the generated acoustic signal;
It is characterized by comprising.
According to the present invention, what is conventionally executed in a single image processing mode can be set to an optimum processing mode according to posture information, so that the accuracy of image recognition can be increased to a practical level. It becomes possible. In other words, since the recognition process is executed only from the various data perceived to the data required in the processing mode, the calculation cost can be further reduced while improving the accuracy. That is, the present invention adds an acceleration sensor to a portable image automatic recognition system that recognizes walking obstacles and targets on the road from multispectral images, and increases the accuracy of automatic recognition by limiting the candidates for recognition objects, Provide stable walking support information for visually impaired people with acoustic signals. Note that the posture of the apparatus can be consciously changed by the user according to a desired processing mode.

In addition, the image recognition apparatus for the visually impaired according to the present invention is as follows.
An input means for inputting a processing mode desired by the user among the plurality of processing modes;
Processing mode control means for controlling the image recognition processing means to operate by forcibly setting the input processing mode;
It is characterized by comprising.
According to the present invention, when a user who is visually impaired wants to switch to a specific processing mode, it is possible to switch immediately.

Furthermore, the image recognition apparatus for the visually impaired according to the present invention is as follows.
The image recognition means divides at least one image of the plurality of images into a plurality of regions, performs Fourier transform for each region, and represents a power (that is, a complex amount of the region) indicating a change amount of luminance value for each region. Power calculation means for obtaining a factor)
It is characterized by that.
According to the present invention, it is possible to accurately detect how complicated each small region is, and an obstacle recognition processing mode for detecting an obstacle by this complexity, or a monotonous road surface. It functions very effectively in a road surface recognition processing mode that discriminates between complexity and complexity of objects other than the road surface.

Furthermore, the image recognition apparatus for the visually impaired according to the present invention is as follows.
The image recognition means includes
Radon conversion means for performing radon conversion on the entire area of the image;
A step identification unit that divides the Radon transformed image into a plurality of regions, obtains a maximum luminance value in each region, and identifies a step based on an arrangement pattern of regions exceeding a predetermined luminance value;
It is characterized by that.
According to this configuration, the presence / absence of a staircase, and whether it is an ascending staircase or a descending staircase can be recognized with high accuracy based on the arrangement pattern of regions. In particular, since the staircase is included in the most dangerous structure for walking for visually impaired people, providing such a highly accurate staircase identification function can significantly reduce the risk of injury to persons with disabilities. become able to.

Furthermore, the image recognition apparatus for the visually impaired according to the present invention is as follows.
The multispectral image sensor is
Photographing means for photographing an image;
An optical bandpass filter that is provided between the object to be imaged and the image capturing means and transmits light of a plurality of wavelength bands;
It is characterized by that.
According to this configuration, it is possible to easily configure a multispectral image sensor by combining an inexpensive filter and an inexpensive photographing unit such as a CCD camera or a CMOS camera. In particular, a filter that selectively passes only the red, yellow, or blue wavelength bands of signal signs can function very effectively when identifying the most dangerous and important traffic signs. To do.

Furthermore, the image recognition apparatus for the visually impaired according to the present invention is as follows.
The posture sensor includes a three-dimensional acceleration sensor,
The device is
Image sensor control means for controlling the multispectral image sensor to acquire the plurality of images only when the three-dimensional acceleration sensor recognizes that the device is substantially stationary.
It is characterized by that.
According to the present invention, the probability of erroneous recognition of an object due to image blurring can be significantly reduced.

Furthermore, the image recognition apparatus for the visually impaired according to the present invention is as follows.
The image recognition processing means includes
A neural network that has been pre-trained with characteristic data including the spectral characteristics for learning obtained from the image, frequency characteristics for learning, and linearity of the edge for learning, and teacher data describing the actual situation in the image;
Based on spectral characteristics, frequency characteristics and edge linearity acquired from the plurality of images using the neural network, walking support information (out of course, degree of danger during walking, notification of obstacle direction, etc.) Supporting information output means for outputting
The acoustic signal generating means generates the acoustic signal based on the walking support information.
It is characterized by that.
ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to give comprehensive walk assistance information to a user, and it is possible to further improve the safety at the time of a user's walk.

As described above, the solution of the present invention has been described as an apparatus. However, the present invention can be realized as a method, a program, and a storage medium storing the program substantially corresponding to these, and the scope of the present invention. It should be understood that these are also included.
For example, when the image recognition apparatus for visually impaired persons according to the present invention is realized as a method, the image recognition method for visually impaired persons is:
A multispectral image sensing step of simultaneously acquiring a plurality of images having different wavelength bands using a multispectral image sensor;
Attitude sensing step for obtaining attitude information of the multispectral image sensor using an attitude sensor;
Based on the posture information acquired in the posture sensing step, a traffic signal sign recognition processing mode for performing image recognition processing on an image having a wavelength band including a wavelength of light emitted by a traffic signal sign, normal image recognition Select one of a plurality of processing modes including an obstacle recognition processing mode for performing processing and a road surface recognition processing mode for performing image recognition processing limited to the center of each of the plurality of images. An image recognition processing step of performing recognition processing of at least one image of the plurality of images using a calculation means in the processed mode,
According to the result of the image recognition processing by the image recognition processing step, an acoustic signal generation step of generating an acoustic signal using a calculation means and a signal generation circuit;
It is characterized by including.
Alternatively, the present invention can be realized as a program that causes a computer to execute the method including the above steps.
A multi-spectral image sensing step for simultaneously obtaining a plurality of images having different wavelength bands using a multi-spectral image sensor for causing a computer to execute an image recognition method for the visually impaired ,
Attitude sensing step for obtaining attitude information of the multispectral image sensor using an attitude sensor;
Based on the posture information acquired in the posture sensing step, a traffic signal sign recognition processing mode for performing image recognition processing on an image having a wavelength band including a wavelength of light emitted by a traffic signal sign, normal image recognition Select one of a plurality of processing modes including an obstacle recognition processing mode for performing processing and a road surface recognition processing mode for performing image recognition processing limited to the center of each of the plurality of images. An image recognition processing step for recognizing at least one of the plurality of images in the processed mode,
An acoustic signal generation step of generating an acoustic signal using a signal generation circuit according to the result of the image recognition processing by the image recognition processing step;
It is an image recognition program for visually impaired persons characterized by including.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a block diagram showing a basic configuration of an image recognition apparatus for a visually impaired person according to the present invention. As shown in the figure, the image recognition apparatus 10 for the visually impaired includes a multispectral image sensor 20, a posture sensor 30, an image recognition processing means 40, a processing mode control (forced switching) means 50, an acoustic signal generation means 60, and an acoustic signal. A signal reproducing means 70 is provided. The image recognition processing unit 40 includes a processing mode setting unit 41 that selects an optimal processing mode, an image recognition unit 42, a radon conversion unit 43, a staircase recognition unit 44, a neural network 45, and a support information output unit 46. The processing mode setting unit 41 is a traffic signal sign recognition processing mode for performing image recognition processing on an image having a wavelength band including the wavelength of light emitted by the traffic signal sign based on the posture information acquired by the posture sensor. One of a plurality of processing modes including an obstacle recognition processing mode for performing normal image recognition processing and a road surface recognition processing mode for performing image recognition processing limited to the center of each of the plurality of images. Then, the image recognition unit 42 performs image recognition processing in the selected processing mode. The functions of the other components are as described above.

FIG. 2 is a block diagram showing a specific apparatus configuration example of the image recognition apparatus for the visually impaired according to the present invention.
This apparatus includes a multispectral image sensor 80 that can simultaneously acquire images of a plurality of wavelengths, a processing mode switch 82 that switches the processing mode of the image input, and posture information for providing the processing mode switch. According to the three-dimensional acceleration sensor 84, a sensor posture calculator 86 for calculating a sensor posture from the acceleration, an image recognition processing unit 90 including a plurality of recognition processing mechanisms for performing recognition processing in a processing mode suitable for each sensor posture, and the recognition result From an acoustic signal generator 92 that changes the frequency, tone, etc., or generates an acoustic signal, a forcible processing switch 88 that forcibly changes / fixes the processing mode, and a speaker 94 that reproduces the changed or generated acoustic signal Composed.

Hereinafter, the function of each component will be described.
Image analysis mode switching mechanism (automatic / manual)
FIG. 3 is a configuration diagram of a sensor unit mounted on the apparatus according to the present invention. As shown in the figure, the 3D acceleration sensor 100 is installed on the optical axis (OO ′) of the CCD camera 102 fixed to the grip portion 106, and the output is sent to a sensor attitude calculator (CPU). A filter lens 104 is attached to the CCD camera 102.
Now, assuming that the angle between the horizontal plane and the optical axis is φ, φ varies from 0 degree (horizontal state of the optical axis) to 90 degrees (the optical axis is perpendicular to the road surface) depending on the gripping angle of the grip 106. . First, in order to confirm that the sensor system is in a stationary state, an image is acquired when the vector sum of accelerations in three orthogonal directions of the 3D acceleration sensor substantially matches the gravitational acceleration 9.8 m / s ² . As a result, the probability of misrecognition due to image blur can be reduced.

Next, when the sensor system is almost stationary, the angle φ formed between the horizontal plane and the optical axis is calculated from the angle φ ′ of the gravitational acceleration vector with respect to the optical axis by the formula “φ = 90 degrees−φ ′”. To do. The processing mode switching unit switches the following three modes according to the magnitude of φ, and drives an image processing system suitable for each angular range.
Mode 1: −δ1 <φ <δ1: Traffic signal sign recognition processing mode Mode 2: δ1 ≦ φ <(90 degrees−δ2): Obstacle recognition processing mode mode 3: (90 degrees−δ2) ≦ φ <(90 degrees + Δ2): Road surface recognition processing mode The allowable angles δ1 and δ2 are set in advance according to the environment to be used. In addition, a forced mode selection switch is installed at the upper end of the grip, and each mode can be selected in preference to the above switching mode.

ここでP_maxはフーリエ変換画像の最大輝度値（グローバルな最大値）、θは閾値、Z₁（ｉ，ｊ）はフーリエ変換後画像の各ピクセルの輝度値、Z₂（ｉ，ｊ）は2値化後の各ピクセルの輝度値、Powerは各小領域のパワーである。
画面内の複数領域についてパワーがあらかじめ設定した閾値を上回る場合には、何らかの障害物が発見されたものと解釈して警告音を発生する。このとき、最大のパワーが得られた小正方領域の画面内の位置に応じて音響信号発生器の発生信号の音色および周波数を時間的に変化させることで、画像内のどの領域に障害物が見られるかを視覚障害者に知らしめる。 Image complexity detection mechanism Obstacles on the road (for example, abandoned bicycles, fences, construction barricades, etc.) generally have a higher ratio of components having a higher spatial frequency than flat road surfaces. Therefore, as shown in FIG. 4, the input image 110 is divided into a large number of square small regions 112, subjected to two-dimensional fast Fourier transform for each small region 112, and binarized after Fourier transform by Equation (1). A threshold for determining the threshold is determined.
Next, the converted image is binarized by Expression (2), and the power of each small region is calculated according to Expression (3).

Here, P _max is the maximum brightness value (global maximum value) of the Fourier transform image, θ is a threshold value, Z ₁ (i, j) is the brightness value of each pixel of the image after Fourier transform, and Z ₂ (i, j) is The luminance value of each pixel after binarization, Power, is the power of each small area.
When the power exceeds a preset threshold value for a plurality of areas in the screen, it is interpreted that some kind of obstacle has been found and a warning sound is generated. At this time, by changing temporally the timbre and frequency of the generated signal of the acoustic signal generator according to the position in the screen of the small square area where the maximum power was obtained, an obstacle in any area in the image Let the visually impaired know if you can see it.

c) Staircase image recognition mechanism by radon transformation Radon transformation is performed on the entire acquired image area. The presence or absence of a linear component can be determined from the image after the Radon transform. The image is divided into regions, and the maximum luminance value in the region is acquired. The obtained luminance value of each region is classified into about five levels, and the connectivity and the pattern of the existing position of each level are compared with the pattern in which the staircase is present for determination.

d) Configuration of a simple multispectral image sensor system using a dual pass filter In recent years, LED traffic signals have become widespread in various countries, but LED traffic signals have sharp emission spectrum intensity characteristics as shown in FIG. That is, each of the red, blue, and yellow signals has a unique wavelength band.
Therefore, by inserting an optical bandpass filter having a sharp passband characteristic, it is possible to perform signal recognition processing with a good S / N. However, in a normal color CCD image sensor, it is difficult to extract a sharp spectrum because a mosaic filter having passband characteristics approximating the RGB color system is attached to the front surface of the semiconductor. Therefore, in the present invention, an optical filter having dual-band pass characteristics (pass wavelength bands 480 to 520 nm and 620 to 660 nm) is attached to the front surface of the existing RGB color CCD, which corresponds to traffic light and red. Extracts wavelength components effectively.

e) Road surface recognition processing mode When the sensor system of FIG. 3 is held at the position of mode 3, or when the “road surface recognition processing mode” is forcibly selected by the selection switch, the sensitivity receptive field of the sensor is set. The object is recognized based on the spectral information of the portion limited to the center of the image. In this case, in addition to the two wavelength bands used in d) above, a 4-wavelength multispectral image sensor in which the G band of 480 to 520 nm and the near infrared band (880 to 960 nm) are added is used in the sensor section of FIG. . By inputting these spectrum intensities to a multilayer perceptron of a certain neural network that has been learned in advance, it is possible to discriminate with high accuracy whether or not there are blind plates, asphalt roads, and puddles on the road surface during walking.

  Up to now, the application of image processing technology to the real world has been focused only on certain specific things such as white lines and stairs. In the present invention, for a plurality of objects, the range of the object and the processing system suitable for the recognition are selected according to the mode in which the angle at which the sensor system is held is a priori information or is forcibly selected. Thus, a more realistic recognition result can be obtained. In the “obstacle recognition processing mode” in particular, it is very useful from the standpoint of the user who uses this device by acquiring extremely vague information such as “determination of whether walking is possible” from the viewpoint of frequency characteristics. Output can be given. In addition, by analyzing the obstacles only when there is a possibility of being unable to walk, unnecessary analysis in an unsuspecting place is omitted to improve the responsiveness of the system. A feeling of fatigue can be reduced.

Equipment shapes Figure 6 is a plan view of a sensor unit for use in embodiments of the visually impaired image recognition apparatus according to the present invention. A four-channel multispectral observation system is configured by optical filters Fl to F3 (F3 not shown) installed in front of three CCD cameras CCD1-3 (CCD3 not shown) with a structure using a half mirror. Yes. Fl is a dual-pass filter and takes charge of two channels in combination with a mosaic filter built in CCD1 having RGB output. It forms a single-axis image acquisition machine that can simultaneously acquire 4-channel spectra for the same part of the object. The acquired information is sent to the notebook PC. General-purpose numerical analysis software installed on a PC (in this embodiment, software called MATLAB) receives such information and performs the above-described various image recognition processes. Appropriate flags such as an obstacle flag and various traffic signal flags are set based on the image recognition processing result, and the flag is sent to the audio output module.

Image Processing Example As an example of the image recognition processing, a discrimination model and result by a neural network using 4ch spectrum information, a walkable right and wrong discrimination model and result, a staircase recognition model and result are shown.
1) Identification model and results using 4ch spectrum information In the image recognition apparatus for visually impaired persons according to the present invention, using 4ch spectrum information, a certain scene on the road (road, white line, puddle, blue light, red light) The image including the signal is analyzed and the identification results (road, white line, puddle, blue light, red light) are shown in the confusion matrix in Table 1.

The discrimination results calculated from the diagonal components in Table 1 average about 90% and can be expected to be practical. The results can be used for blind tile plates, tracking of white lines, signal recognition, etc. In particular, the high recognition rate of blue and red for traffic signals that are highly important in terms of safety indicates the usefulness of the technique according to the present invention.
7 and 8 show the images of the four channels actually measured by the apparatus according to the present invention. FIG. 7 is a picture of a pedestrian traffic light (LED, blue). It can be seen that there is a clear difference between the channels. That is, in the blue and green spectral images, as shown in the figure, the “human-shaped mark” of the blue signal can be captured very sharply. Similarly, FIG. 8 is a photograph of a pedestrian traffic light (LED, red), and in the red spectrum image, as shown in the figure, the “mark” of the red signal can be captured very sharply.

2) Walkable right and wrong discrimination model and result FIG. 9 shows an image of a corridor, and FIG. 10 shows an image obtained by subjecting this original image to a two-dimensional Fourier transform for each part and binarizing the above equation. It can be seen that a characteristic white image can be obtained in the wall portion compared to the floor surface. FIG. 11 shows the accumulated luminance values for each local area and displays them as power. Walkable floors have low power and can be clearly distinguished from walls and edges.
FIG. 12 shows another example of the corridor image. It can be seen from the figure that the corridor that can be walked is recognized as a safety zone, and the green part (edge) that should be noted is recognized as a danger zone. When the dangerous area enters the designated area in the image, a warning signal is delivered to the user by voice to notify the danger.

3) Staircase recognition model and results FIG. 13 shows an example of an acquired image of an upstairs (left image in the figure), an example of an edge extraction image (right image in the figure), and FIG. 14 shows a Radon transform image of the image in FIG. Indicates the luminance value.
From the figure, it can be seen that there are multiple linear components around 90 ° when entering the upstairs. In order to detect this feature amount, the luminance value from 60 ° to 120 ° in the central part is scanned and discriminated to 5 levels from 0 to 4 (level 1 was forcibly set to level 0 for clarity of discrimination) Is shown in FIG. From the figure, it can be seen that there are two (or more) continuous or independent entities of level 2 or higher when entering the upstairs.

Next, FIG. 16 shows an example of a descending staircase acquired image (left image in the figure), an edge extracted image example (right image in the figure), and FIG. 17 shows the luminance values of the Radon converted image of the image in FIG.
From the figure, it can be seen that there is a continuous linear component around 90 ° when entering the downstairs. In order to detect this feature quantity, the luminance value from the central part 60 ° to 120 ° is scanned and discriminated into five levels (level 1 is forcibly set to level 0 for the sake of discrimination) in FIG. Show. From the figure, it can be identified that the repetition of the characteristic straight line seen on the center of the descending staircase image appears in the form of 3 consecutive levels above level 2. Do.

From these characteristics, the upstairs and downstairs are identified. The conditions are summarized as follows.
(1) Scan the luminance value from 60 ° to 120 ° in the center of the Radon conversion image, and discriminate the luminance value into 5 levels. (2) If there are 3 (or more) continuations of level 3 or higher, descending stair (3) If the above conditions are not met and there are two (or more) continuous or independent entities of level 2 or higher, notify the possibility of ascending stairs (4) The above two conditions If the condition is not satisfied, the possibility that there is no staircase is notified. This judgment system was applied to the corridor walking video, and the result was “no staircase”. And the result applied to the going down and up stairway approach animation is shown in FIG. 19 and FIG. The vertical axis in the figure indicates the determination result, where 0 indicates no staircase, 1 indicates an upstairs, and 2 indicates a downstairs. It can be seen that the presence of stairs and the up / down discrimination can be made when entering the stairs.

4) Integrated failure judgment For evaluation of the acquired images, characteristic values from the viewpoints of color characteristics, frequency characteristics, and linearity of the contour line were acquired and the respective results were obtained. In order to judge these in an integrated manner, a metaclass processing system is installed. FIG. 21 shows a conceptual diagram thereof.
As shown in the figure, the feature evaluations seen from each viewpoint exist in parallel (the feature evaluation viewpoints can be expanded). A learned neural network is placed as an integrated judgment of the situation in these upper classes. This learning may be performed by learning a general staircase, corridor, or road, and if the area where the user walks is limited, the learning may be performed based on the actual data. This neural network will give a final decision (out of course, walking warning, safety signal).
As described above, according to the method of the present invention, the image on the street is automatically analyzed online, and the information on the target object and the walking support information are high when the visually handicapped person walks on the street. It can be automatically generated with accuracy and provided to visually impaired persons by voice or the like.

It is a block diagram which shows the basic composition of the image recognition apparatus for visually impaired persons by this invention. It is a block diagram which shows the specific apparatus structural example of the image recognition apparatus for visually impaired persons by this invention. It is a block diagram of the sensor part mounted in the apparatus by this invention. It is a figure which shows the example of the division | segmentation of an input image. It is a figure which shows the emission spectrum intensity | strength characteristic of an LED type traffic signal apparatus. It is a top view of the sensor part used with the embodiment of the image recognition apparatus for visually impaired persons by this invention. It is an image (blue signal) of each of four channels actually measured by the apparatus according to the present invention. It is an image (red signal) for each of four channels actually measured by the apparatus according to the present invention. It is the image which imaged the corridor. FIG. 10 is a diagram in which the image of FIG. 9 is subjected to two-dimensional Fourier transform for each part and binarized by the above formula. It is the figure which accumulated the luminance value for every local in the image of FIG. 9, and displayed it as power. It is an image which shows another example of the image of a hallway. It is a figure which shows the acquisition image example (left image of a figure) of an upstairs, and the edge extraction image example (right image of a figure). It is a graph which shows the luminance value of the radon conversion image of the image of FIG. It is a figure which shows the result of having discriminate | determined the image of FIG. It is a figure which shows the acquisition image example (left image of a figure) of a downstairs, and the edge extraction image example (right image of a figure). It is a graph which shows the luminance value of the Radon conversion image of the image of FIG. It is a figure which shows the result of having discriminate | determined the image of FIG. It is a figure which shows the result applied to the descent | stairs approaching animation. It is a figure which shows the result applied to the upstairs approach moving image. It is a conceptual diagram of the processing system of the metaclass of integrated judgment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 10 Image recognition apparatus for visually impaired people 20 Multispectral image sensor 30 Posture sensor 40 Image recognition processing means 41 Processing mode setting part 42 Image recognition part 43 Radon conversion means 44 Staircase recognition means 45 Neural network 46 Support information output means 50 Processing mode control (Forced switching) means 60 Acoustic signal generating means 70 Acoustic signal reproducing means 80 Multispectral image sensor 82 Processing mode switch 84 Three-dimensional acceleration sensor 86 Sensor attitude calculator 88 Forced processing switch 90 Image recognition processing unit 92 Acoustic signal generator 94 Speaker 100 3D acceleration sensor 102 CCD camera 104 Filter lens 106 Grip part 110 Input image 112 Small area

Claims (9)

An image recognition device for a visually impaired person,
A multispectral image sensor that simultaneously acquires a plurality of images each having a different wavelength band; and
An attitude sensor for acquiring attitude information of the device;
Traffic signal sign recognition processing mode for performing image recognition processing on an image having a wavelength band including the wavelength of light emitted by the traffic signal sign based on the posture information acquired by the posture sensor, normal image recognition processing One of a plurality of processing modes including an obstacle recognition processing mode for performing image recognition and a road surface recognition processing mode for performing image recognition processing limited to the center of each of the plurality of images. Image recognition processing means for performing recognition processing of at least one of the plurality of images in the processing mode,
Acoustic signal generating means for generating an acoustic signal according to the result of the image recognition processing by the image recognition processing means;
An image recognition apparatus for the visually impaired, comprising: The image recognition apparatus for a visually impaired person according to claim 1,
A processing mode control means for controlling the image recognition processing means to forcibly set and operate in a processing mode desired by a user among the plurality of processing modes;
An image recognition apparatus for the visually impaired, comprising: The image recognition apparatus for visually impaired persons according to claim 1 or 2,
The image recognition means includes power calculation means for dividing at least one image of the plurality of images into a plurality of areas, performing Fourier transform for each area, and obtaining power that is a feature quantity indicating a change amount of a luminance value for each area. Including,
An image recognition apparatus for visually impaired persons. The image recognition apparatus for a visually impaired person according to any one of claims 1 to 3,
The image recognition means includes
Radon conversion means for performing Radon conversion on the entire area of at least one of the plurality of images;
A step identification unit that divides the Radon transformed image into a plurality of regions, obtains a maximum luminance value in each region, and identifies a step based on an arrangement pattern of regions exceeding a predetermined luminance value;
An image recognition apparatus for visually impaired persons. In the image recognition apparatus for visually impaired persons according to any one of claims 1 to 4,
The multispectral image sensor is
Photographing means for photographing an image;
An optical bandpass filter that passes light of a plurality of wavelength bands,
An image recognition apparatus for visually impaired persons. In the image recognition apparatus for visually impaired persons according to any one of claims 1 to 5,
The posture sensor includes a three-dimensional acceleration sensor,
The device is
Image sensor control means for controlling the multispectral image sensor to acquire the plurality of images only when the three-dimensional acceleration sensor recognizes that the device is substantially stationary.
An image recognition apparatus for visually impaired persons. In the image recognition apparatus for visually impaired persons according to any one of claims 1 to 6,
The image recognition processing means includes
A neural network that has been pre-trained with characteristic data including the spectral characteristics for learning obtained from the image, frequency characteristics for learning, and linearity of the edge for learning, and teacher data describing the actual situation in the image;
Support information output means for outputting walking support information based on spectral characteristics, frequency characteristics and edge linearity acquired from the plurality of images using the neural network,
The acoustic signal generating means generates the acoustic signal based on the walking support information.
An image recognition apparatus for visually impaired persons. An image recognition method for the visually impaired,
A multispectral image sensing step of simultaneously acquiring a plurality of images having different wavelength bands using a multispectral image sensor;
Attitude sensing step for obtaining attitude information of the multispectral image sensor using an attitude sensor;
Based on the posture information acquired in the posture sensing step, a traffic signal sign recognition processing mode for performing image recognition processing on an image having a wavelength band including a wavelength of light emitted by a traffic signal sign, normal image recognition Select one of a plurality of processing modes including an obstacle recognition processing mode for performing processing and a road surface recognition processing mode for performing image recognition processing limited to the center of each of the plurality of images. An image recognition processing step of performing recognition processing of at least one image of the plurality of images using a calculation means in the processed mode,
According to the result of the image recognition processing by the image recognition processing step, an acoustic signal generation step of generating an acoustic signal using a calculation means and a signal generation circuit;
An image recognition method for the visually impaired, characterized by comprising: A multi-spectral image sensing step for simultaneously obtaining a plurality of images having different wavelength bands using a multi-spectral image sensor for causing a computer to execute an image recognition method for the visually impaired ,
Attitude sensing step for obtaining attitude information of the multispectral image sensor using an attitude sensor;
Based on the posture information acquired in the posture sensing step, a traffic signal sign recognition processing mode for performing image recognition processing on an image having a wavelength band including a wavelength of light emitted by a traffic signal sign, normal image recognition Select one of a plurality of processing modes including an obstacle recognition processing mode for performing processing and a road surface recognition processing mode for performing image recognition processing limited to the center of each of the plurality of images. An image recognition processing step for recognizing at least one of the plurality of images in the processed mode,
An acoustic signal generation step of generating an acoustic signal using a signal generation circuit according to the result of the image recognition processing by the image recognition processing step;
An image recognition program for the visually impaired, characterized by comprising:

Images