KR100690562B1 - Reproduction of Visual Image using Auditory Stimulation and Method Controlling of the same - Google Patents

Abstract

The present invention relates to a visual reproducing apparatus and a control method using the auditory to convert the visual information into auditory information to obtain visual information by changing the sound that can be perceived by the hearing impaired. More specifically, an input unit including a plurality of distance sensors or one or more image sensors and a distance detected by a plurality of distance sensors included in the input unit, a position where the distance sensors are arranged, or an image detected by the image sensor Generate a plurality of three-dimensional information through the information contained in the plurality of pixels included in the data, generate a plurality of sound data based on the three-dimensional information, and divide the sound data at the left and right outputs at the same time It is characterized in that the recognition.

Visual information, hearing information, visually impaired person, forward recognition

Description

Reproduction of Visual Image using Auditory Stimulation and Method Controlling of the same}

1 is a block diagram of a device for converting visual information into audio information according to the prior art.

Figure 2 shows an example of the device configuration corresponding to the first embodiment of the present invention.

3 exemplarily shows an embodiment of the first embodiment of the present invention.

4 shows a method of obtaining three-dimensional information in the first embodiment of the present invention.

5 exemplarily shows an embodiment of the second embodiment of the present invention.

6 shows a method of obtaining three-dimensional information in the second embodiment of the present invention.

* Description of Major Symbols in Drawings *

10... Input

11... Distance sensor

12... Image sensor

20... Control

21... Processor

30... Output

31... Left output part

32... Right output

The present invention relates to a visual reproducing apparatus and a control method using the auditory to convert the visual information into auditory information to obtain visual information by changing the sound that can be perceived by the hearing impaired. More specifically, an input unit including a plurality of distance sensors or one or more image sensors and a distance detected by a plurality of distance sensors included in the input unit, a position where the distance sensors are arranged, or an image detected by the image sensor Generate a plurality of three-dimensional information through the information contained in the plurality of pixels included in the data, generate a plurality of sound data based on the three-dimensional information, and divide the sound data at the left and right outputs at the same time It is characterized in that the recognition.

For the blind, it was common to use the sense of touch and hearing, except vision, for obstacle passage, stairway movement, and pathfinding. In more detail, the visually impaired people were able to pass obstacles and check stairs using a wand for visually impaired persons, and could walk using a braille guide or a visually impaired visual aid block located in a pedestrian passage. Although the cane for the visually impaired is low in cost, it is widely used as a device for checking the presence or absence of obstacles on the foot when walking a road or checking the deformation of a walking road such as a stairway. There was a drawback that the cane could be more dangerous by the cane, such as being hungry or causing discomfort to others. In addition, the cane could not detect the obstacle located in the upper part that could be caught on the head, there was a risk of being injured by the obstacle located in the upper part.

Or you can use the method of receiving help from the guide dog who performed special training, but there is a space where the guide dog is not recognized as an essential helper to guide the visually impaired, and is treated as a general dog and is not allowed to enter. In addition, the visually impaired people had a disadvantage in that they could not be used conveniently, and also had a disadvantage in that they were not widely used due to high cost.

In order to replace the devices or methods for safe walking of the visually impaired people, a device that assists the visually impaired people who can not recognize visual information when there is an obstacle in front of them when walking or doing activities. Are being developed. As an example of the devices, a conventional device may include Korean Patent Laid-Open Publication No. 10-2003-0083031 (Method and apparatus for converting visual information into auditory information).

The Korean Patent Publication No. 10-2003-0083031 (method and apparatus for converting visual information into auditory information), as shown in FIG. A method of converting visual information into auditory information and an apparatus using the same are provided. The method and apparatus for converting visual information into auditory information allocates frequencies differently according to distances to objects, and converts and provides paths that are differently sensed by distance sensors on both sides, thereby providing distances and positions of objects. It can be known through repetitive learning using auditory instead of visual. To obtain distance and position information about the object in front, the beam is irradiated and the reflected wave is received from the device spaced left and right. By telling the distance information of the audible sound, the user can estimate the distance and position of the object by using the left and right separation distance of the receiver and the deviation of the tone perceived by the left and right ears. The method and apparatus for converting visual information into auditory information has an advantage of allowing a distance to an object to be recognized by displaying a distance to the object through a frequency, but only one point of the object in front of the user. Because of the perception, there was a drawback of not being able to fully recognize the state of the front.

Therefore, a device and a method for allowing the visually impaired to recognize the overall data on visual information through hearing are being developed.

SUMMARY OF THE INVENTION The present invention has been made to solve the above problems, and proposes an apparatus and method for converting forward visual information that can be accepted as visual information into auditory information so that the visual information can be recognized as a whole. There is a purpose.

In addition, the present invention includes a plurality of distance sensors or image sensors, in order to generate three-dimensional information about the visual information, a plurality of distance values recognized by the plurality of distance sensors and the position where the distance sensor is arranged An object of the present invention is to propose an apparatus and method for generating three-dimensional information through information generated by a plurality of pixels included in image data input through at least one or more image sensors.

In addition, the three-dimensional information is selected from the three parameters among the stereo sound method using the size, height, tone (kind), parallax, left and right speakers that are the parameters of the sound, and contrasts the selected parameter of the sound with the three-dimensional information, respectively By generating sound data representing the three-dimensional information, and simultaneously listening to each sound data for the three-dimensional information, the device and method for listening to a variety of simultaneous sounds to allow the user to recognize the overall appearance of the front It is for the purpose of suggestion.

In addition, the apparatus and method that can adjust the amount of visual information to be converted into the auditory information according to whether the user skillfully use the device, that is, the ability to distinguish the auditory information mixed with the various sounds The purpose is to propose.

In order to achieve the above object, in the present invention, a plurality of distance sensors, each consisting of a transmitter for transmitting an outgoing signal having a frequency to the outside and a receiver for receiving a received signal reflected by the outgoing signal in real time, are spaced apart at predetermined intervals. An input unit including at least one distance sensor array;
A timer for measuring a reception time which is a time from when the transmission signal is transmitted by the transmission unit to the reception unit detecting the reception signal generated by the reflection of the transmission signal, characteristics of the transmission signal or reception signal, and the reception signal; Each distance data of the plurality of distance sensors is generated through time, and the distance sensor is located in the distance distance sensor array, the horizontal array is x-axis data, the vertical array is y-axis data, and the generated distance data. Generate range profile image data including three-dimensional information using z-axis data, and select one of a loudness, a loudness, a kind of sound, a parallax of a sound, and a stereo sound. A plurality of pieces included in the distance profile image data such that the three selected parameters change according to the axis data, the y axis data, and the z axis data. A digital signal processor (DSP) for generating respective sound data for three-dimensional information and generating left and right auditory data output to the left and right output units through the plurality of sound data; A control unit including a processor configured in a combination including at least one of a micro-processing unit (MPU); And
And an output unit configured as one of an earphone, a headphone, or a bone conduction device in which left and right sides are distinguishable, and outputting the left auditory data and the right auditory data so that a user can listen.
The output unit outputs left and right auditory data generated through a plurality of sound data generated with respect to the distance profile image data, and simultaneously outputs a plurality of sound data included in the left and right auditory data. We propose a visual representation apparatus using hearing.

Preferably, the transmission signal having the frequency of the distance sensor included in the input unit of the visual reproducing apparatus using the auditory sound is any one of ultrasonic waves, infrared rays, electromagnetic waves, and lasers.

The control unit may convert an analog signal into a digital signal so that the received signal transmitted from the input unit may be processed by a digital signal processor (DSP) or a micro-processing unit (MPU). An additional A / D converter (Analog / Digital Converter), and when the receiving unit detects the received signal, in order to distinguish the outgoing signal that generated the received signal, the interval of the outgoing signal emitted from the transmitting unit to the outside It is preferable to further include a pulse generating unit for controlling the. In addition, in order to convert only a part of the distance profile image data into sound data, it is preferable to further include an adjusting unit for adjusting a part of the distance profile image data.

The distance data is generated by dividing the frequency or amplitude of the outgoing signal or the received signal by the reception time by two, or the distance data is generated by the sound speed data generated by the sound speed calculation method using temperature as a variable. Preferably generated by dividing by two. In addition, in order to generate the sound speed data, it is more preferable to further include a thermometer for measuring the temperature for measuring the temperature, which is one of the variables of the sound speed calculation method, and transmits the digital data to the control unit. More preferably, the distance data may be generated through the average sound speed by setting the temperature used in the sound speed calculation method to an average temperature.

In addition, the distance sensor array of the input unit is preferably spaced apart from the user at a predetermined distance on the left and right. The x-axis data located on the left side of the x-axis center is generated by the distance sensor included in the distance distance sensor array on the left side, and the x-axis data on the right side is generated by the distance distance sensor array on the right side. Can be.

As the y-axis data becomes larger, the sound data becomes louder (the frequency becomes smaller), and as the z-axis data becomes larger, the loudness (sound pressure or intensity, sound pressure level (dB number)) becomes larger. The sound data for the distance profile image data having the x-axis data located on the left side of the center of the x-axis is generated as left audio data, and the sound data for the distance profile image data having the x-axis data on the right side is generated. It is preferable to generate the right auditory data.

The present invention also includes an input unit including at least one image sensor;
A pulse generator configured to set an operation interval of the image sensor, a plurality of pixels included in the image data input through the image sensor, the horizontal arrangement of which is located in the image data as x-axis data, and the vertical arrangement as y-axis data And generating three-dimensional information using z-axis data as a distance between the pixel generated by the color information, the contrast information, the chroma information, the saturation information, and the characteristics of the plurality of image sensors. And selecting one of loudness, height, type, parallax, and stereo sound to obtain respective sound data for the plurality of pixels in which the selected parameter changes according to the x-axis data, y-axis data, and z-axis data. Generating left and right auditory data output to a left output unit and a right output unit through the plurality of sound data; The controller including a; digital signal processor (Digital Signal Processor:: DSP) or a compact processing unit (MPU Micro-processing unit) And
And an output unit which can be used as one of an earphone, a headphone, or a bone conduction device in which left and right sides are distinguishable, and outputs the left auditory data and the right auditory data so that a user can listen.
The output unit outputs left and right auditory data generated through a plurality of sound data generated with respect to the distance profile image data, and simultaneously outputs a plurality of sound data included in the left and right auditory data. We propose a visual representation apparatus using hearing.

delete

The control unit may control the digital signal from the analog signal so that the value recognized by the plurality of image sensors of the input unit may be processed by a digital signal processor (DSP) or a micro-processing unit (MPU). It is preferable to further include an A / D converter (Analog / Digital Converter) for converting into a signal.

The input unit may include image sensors positioned at left and right sides of the user, and the left image sensor and the right image sensor may be positioned to recognize the same point. In addition, the control unit controls the left image sensor and the right image sensor to operate at the same time through the pulse generating unit, the recognition by the left image sensor generates the left image data, the right image sensor is recognized by the right image It is preferable to generate data, and the color information, the contrast information, the chroma information, etc. of the pixels included in the left and right image data are examined, and the pixels included in the left image data and the right image data indicating the same point are included. Recognizing the pixel, and generating the z-axis data through the interval between the left and right image sensor that generated the left and right image data and the position where the pixels representing the same point is arranged in the left image data and the right image data desirable.

As the y-axis data becomes larger, the sound data becomes louder (the frequency becomes smaller), and as the z-axis data becomes larger, the loudness (sound pressure or intensity, sound pressure level (dB number)) becomes larger. The sound data for the distance profile image data having the x-axis data located on the left side of the center of the x-axis is generated as left audio data, and the sound data for the distance profile image data having the x-axis data on the right side is generated. It is preferable to generate the right auditory data, and the output unit outputs left and right auditory data generated through a plurality of sound data generated with respect to the distance profile image data, and includes a plurality of auditory data included in the left and right auditory data. It is preferable to output two sound data simultaneously.

In addition, it is preferable to additionally generate data on color or texture for each pixel through the image data, and the sound data includes a tone in contrast to data on the color or texture.

The controller may further include an adjusting unit for adjusting a portion of the distance profile image data so as to convert only a portion of the image data input from the at least one image sensor into sound data.

In addition, the visual reproducing apparatus using the hearing may have a goggle shape in which the input portion is located in the lens portion of the goggles, and the output portion is located close to the ear fixing portion of the goggles, and the position is freely moved. Located, the output unit may be in the shape of glasses located close to the ear of the goggles while being free to move.

In addition, the input portion may be located in the hinge portion of the glasses or goggles, the output portion may be in the shape of glasses or goggles located close to the ear fixing portion of the goggles while being free to move. In addition, the input unit and the output unit may be in the shape of a headphone provided in each of the headphones.

Hereinafter, the present invention will be described in more detail with reference to the accompanying drawings.

The visual representation apparatus using hearing proposed by the present invention includes an input unit, a controller, and an output unit. The input unit may be configured with at least one or more distance sensor arrays or at least one or more image sensors arranged with a plurality of distance sensors having a transmitter and a receiver.

The control unit includes a processor including a combination including any one or more of a digital signal processor (DSP) or a micro-processing unit (MPU). The control unit may include a timer for measuring until the outgoing signal emitted by the transmitter of the distance sensor is reflected and detected as a reception signal in the receiver. In addition, an A / D converter (Analog / Digital Converter) may be additionally included to change the received signal input by the input unit from an analog signal to a digital signal. The apparatus may further include a pulse generator configured to set an interval at which the input unit is operated. In order to measure the temperature, which is one of the variables necessary to obtain the sound velocity when measuring the distance using the sound velocity, the temperature may be measured. It may further include a temperature measuring unit for generating the obtained digital signal. In addition, according to the user's proficiency in using the device, that is, the proficiency of grasping the visual information in front of the user through auditory information recognized by the device, the control unit may further include a control unit for limiting or expanding the visual information converted into the auditory information. .

The pulse generator is a part for setting an interval at which an outgoing signal is emitted from an input unit and an interval at which a control unit receives a received signal. Preferably, the pulse generator sets different intervals of the outgoing signals to distinguish which outgoing signals are returned by reflecting the received signals received at the receiver. The visual reproducing apparatus using the hearing proposed by the present invention is characterized by receiving the information on the front in real time and converting it into auditory information so that it can be recognized by hearing, so that by the object near and far It is necessary to distinguish which transmission signal is reflected and reached when the time of reflection and arrival at the reception part is different. Therefore, by determining the interval of the outgoing signal, it is possible to determine which outgoing signal by the outgoing signal at the interval of the received signal.

The timer counts a number at regular intervals to measure the time from when the transmitting signal of the distance sensor, which is a variable necessary for obtaining the distance, to the reception of the received signal by the transmitting signal. Becomes a part. The timer counts a time when the outgoing signal is emitted to the outside and counts the number according to a predetermined interval, and measures until the reception signal is reached, thereby helping to determine the reception time. Since distance can be obtained by multiplying speed (or speed) by time, the reception time is an important variable for measuring distance.

The A / D converter (Analog / Digital Converter) is a part for converting the received signal received by the receiver of the sensor arranged in the distance sensor array from the analog signal form to the digital signal form. In order to perform analysis and calculation in the digital signal processor (DSP) or micro-processing unit (MPU), the data must be in the form of a digital signal to enable discrete operation. The D converter (Analog / Digital Converter) performs the function of sampling the analog signal and converting it into a digital signal form.

The temperature measuring unit preferably includes a temperature sensor that is converted into an analog or digital signal by being processed into an electrical signal by amplifying and linearizing material expansion, resistance change, and thermal power generation. As described above, when the transmission signal having a frequency in the input unit is an ultrasonic wave, the temperature measuring unit measures an air temperature required to more accurately calculate a sound velocity, which is a variable necessary for measuring a distance. The temperature measuring unit may be additionally included in the controller when using ultrasonic waves, that is, when using sound waves as the transmission signal.

The control unit is a part for adjusting the amount of visual information converted into auditory information. If the user receives the auditory information of the visual information in the front by using the visual reproduction apparatus using the hearing proposed by the present invention, the auditory information according to the proficiency to grasp the visual information through the recognized auditory information May not be properly recognized as visual information, and thus a part for adjusting a part for converting the visual information into auditory information. The processor of the controller recognizes the degree of adjustment of the controller, and converts only the degree of adjustment of the controller into auditory information among the visual information recognized by the input unit.

A processor composed of a combination including at least one of the digital signal processor (DSP) and the micro-processing unit (MPU) is a part that performs the most important calculation in the controller. In order to reduce the cost and simplify the operation of the device, the processor is more preferably configured as a DSP, and more preferably, the DSP and the MPU may be mixed. As described above, when the DSP and the MPU are configured to be mixed, the DSP capable of performing the operation faster to perform the real-time data operation may perform the operation, and the MPU may be used for the command and the control. .

The DSP performs arithmetic operations on digital data obtained by converting an analog signal through an A / D converter, so that the digital data can be quickly calculated mathematically for signal processing such as filtering or spectrum analysis. to be. Of course, the MPU may be performed in the MPU, but it is slower than a DSP that performs only mathematical operations. The DSP includes circuits capable of processing repetitive operations such as addition, subtraction and multiplication at high speed in order to process digital signals at high speed. Although initially used very limitedly, in recent years, its use has become very wide, and is used for voice coding for converting voice signals into digital signals, and is used in digital mobile communications, answering machines, video telephones, and multimedia. It also specializes in mathematical operations, so it is also used in the field of 3D image processing and animation, and can control small machines. It also acts as a digital filter for communication, and works with the MPU to reduce the throughput of the MPU.

The processor generates a combination of various pieces of information transmitted from the input unit, the timer, the temperature measuring unit, and the adjusting unit, and generates right audio data and left audio data to be transmitted to the output unit.

The output unit is a device for recognizing the left auditory data and the right auditory data generated by the processor of the controller, the left output unit for outputting the left audio data and the right output unit for outputting the right audio data. Preferably, the output unit is displayed to distinguish the left output unit and the right output unit. The output unit may be configured as a device such as an earphone or a headphone configured to ring the diaphragm by an electrical signal transmitted. In addition, it is configured as a bone conduction device using a conduction mode that causes sound sensation by transmitting sound vibration directly from the human skull to the inner ear lymph and stimulating the auditory nerve. It is also possible.

As shown in FIG. 3, the visual representation apparatus using hearing may be configured in a goggle form, and may be variously configured in a glasses form or a headphone form although not shown. If the input is configured to look forward of the user and the output is located close to the user's ear, it is obvious that any form is applicable.

More preferably, when the input unit is composed of at least one distance sensor array including a plurality of distance sensors including a transmitter and a receiver, as shown in FIG. The output unit may be located at an ear fixing portion of the goggles, located close to the ear, and configured to freely change the position. In addition, the output unit may be connected by wire as shown in FIG. 3, or may be connected to the controller wirelessly.

In addition, as shown in Figure 5, it is possible to configure in the form of glasses. Positioning the lens portion of the glasses wide to configure the distance sensor array, the output portion may be positioned so as to be close to the ear and the position change free, as in the form of the goggles.

In addition, if the distance sensor array of the input unit is configured to be located at least one each on the left and right, or if the input unit is composed of at least one or more image sensors located on the left and right around the user, the input unit hinges of glasses or goggles It can be configured to be located at. In addition, it is obvious that the present invention may be applied in any form as long as the device is configured so that it does not look awkward to others when the user is mounted.

First embodiment

Figure 2 shows an example of the device configuration corresponding to the first embodiment of the present invention. 3 exemplarily shows an embodiment of the first embodiment of the present invention.

A first embodiment of the present invention proposes a visual representation apparatus using hearing having an input unit including at least one distance sensor array having a plurality of distance sensors including a transmitter and a receiver arranged at predetermined intervals.

The input unit according to the first embodiment includes a transmitter that emits an outgoing signal having a frequency (pulse) to the outside so as to measure a distance, and a receiver that receives a received signal reflected by an object located in front of the outgoing signal. At least one distance sensor array comprising a plurality of distance sensors arranged at a predetermined interval.

The outgoing signal having the frequency includes an ultrasonic wave, an infrared ray, a laser, an electromagnetic wave, and the like, and in particular, the ultrasonic wave is an outgoing signal frequently used for the purpose of measuring a distance. In addition, the infrared ray may detect a distance from an object located at a close distance, and may detect whether the color of the object is light or dark based on the reflected signal.

The ultrasonic wave is a sound wave of about 16000 kHz or more, which is generally equal to or more than a frequency of a sound heard in a human ear, and the infrared ray has a wavelength range of about 0.75 following visible light. An electromagnetic wave having a thickness of about 1 to 1 mm, which corresponds to the outside of the red portion in the light spectrum. In addition, there are ultraviolet lasers, visible lasers, infrared lasers, and far infrared lasers in accordance with the above-mentioned wavelength range. There are various kinds of gases, solids, liquids, and semiconductors for the operating medium.

The ultrasonic wave, which is an originating signal of an ultrasonic sensor mainly used to measure distance, does not correlate to transparent, opaque or correlated objects, and does not care about metallic or non-metallic, liquid or solid state, powder phase, and environmental conditions. There is an advantage that the problem that the receiving distance of the ultrasound is narrow does not occur.

As the name indicates that the ultrasonic wave is a sound wave, the distance can be detected more easily by using the sound velocity generated by the sound velocity calculation method using temperature as a variable.

The sound velocity calculation method is as shown in Equation (1).

Assuming that the average temperature is 25 ° C, the sound velocity can be assumed to be 346 kHz by the above calculation method. The distance between the sound velocity obtained as described above and the time from when it is transmitted to when it is received may be measured and multiplied by half of the time.

Hereinafter, a first embodiment will be described based on the ultrasonic sensor.

The input unit may include at least one distance sensor array arranged at a predetermined interval apart from a plurality of distance sensors including a transmitter and a receiver. In the distance sensor array, the distance between the distance sensors is a characteristic of the sensor. In this case, it is desirable to determine in consideration of the sensing angle of the distance sensor. This is to prevent interference caused by the received signal reflected from the outgoing signal transmitted from another transmitter. Ultrasonic sensors have a sensing angle of about 45 ° and are less accurate than values outside of 45 °. In the case of an ultrasonic sensor, it is very likely that errors due to interference occur, so it is important to determine the interval.

In addition, it is preferable to determine the direction so that the distance sensor located on the right side is biased to the right side so that the distance sensor located on the right side is centered on the right side so as to recognize a wider range forward. The farther away the distance sensor is, the more biased it is to the right or left side. In addition, the distance sensor located on the upper side with respect to the center of the distance sensor array is preferably oriented so that the distance sensor located on the lower side is biased to the lower side. It is desirable to have a biased direction.

The sensors arranged in the distance sensor array may be 16 columns × 16 rows, as shown in FIG. 3, and the distance sensor array may be configured as more sensors than the above examples in order to recognize a wider range. Is self explanatory.

The sensor arranged in the distance sensor array is operated by a pulse generator which may be included in the control unit, and more specifically, the transmitter of the sensor emits the outgoing signal to have a certain rule, the outgoing signal as described above The reason for transmitting the signal is to make it easy to determine which transmission signal is a reception signal generated by reflecting the reception signal when the reception unit detects the reception signal.

The timer of the controller measures from the time when the transmission signal is emitted until the receiver detects the reception signal for the transmission signal, generates it as a reception time, and transmits the reception time to the processor of the controller.

The processor of the controller recognizes the reception time for each sensor, and generates distances between the sensor and the reflected object as distance data through the reception time. The distance between the sensor and the reflected object may be obtained through the reception time and the speed of the signal by the characteristics of the transmission signal and the reception signal of the sensor. When the sensor of the input unit is an ultrasonic sensor, distance data may be generated through sound speed and the reception time. The sound speed may be preset to a specific value, or as mentioned above, a more accurate speed of the ultrasonic wave may be obtained through a sound speed calculation method using temperature as a variable. The control unit may further include a temperature measuring unit for measuring the temperature in order to obtain the speed of the ultrasonic wave through the sound velocity calculation method.

Range profile image data including three-dimensional information in relation to the position of the sensor spaced apart at regular intervals in the distance sensor array and the distance data generated through the reception time generated by each sensor. data). The x-axis (horizontal) and y-axis (height) of the distance profile image data are determined based on the position of the sensor, and the z-axis (length) of the distance profile image data is determined through the distance data. Configure. Subsequently, three parameters are selected among the stereo sound, which is displayed by outputting the sound at different sizes, such as loudness, height, type (tone), parallax, and left and right output units, and the x-axis data, y-axis data, and z-axis. Generates sound data whose values change in accordance with the data. The loudness may be represented by any one of sound pressure, intensity, or sound pressure level (dB number), and the height of the sound may be represented as a frequency. In addition, the type of sound (tone) may be represented to be divided into sounds of musical instruments, that is, string instruments such as violins and cellos, pianos, keyboard instruments such as organs, and wind instruments such as flutes and trumpets. In addition, the parallax means an interval in which sound is generated like a morse code. Preferably, the change of the y-axis data is represented by the height of the sound, the change of the z-axis data is represented by the volume of the sound, and the change of the x-axis data may be represented by the stereo sound.

4 exemplarily shows the x-axis and the z-axis of the distance profile image data. As shown in FIG. 4, the outgoing signal transmitted from the sensors arranged in horizontal 16 columns A to Q and the height 16 rows 101 to 116 is straightly reflected and converted into a received signal and sensed by the receiving unit. The x-axis (A-Q) and the z-axis (101-116) of the distance profile image data are determined by the position of each sensor. In the sensors located at A101 to Q116, when the reception time is determined, the distance between each object and the sensor is determined according to the reception time, and the distance becomes the y axis.

As described above, sound data for 256 values of A101 to Q116 is generated from the distance profile image data.

An example of a process of generating the sound data is as follows.

The sound data has a lower size as it is positioned below the y axis. The size of the sound data refers to the pressure of the ear felt by the user, and may be represented by sound pressure, intensity, and sound pressure level (dB number). In addition, the sound data has a higher height (smaller frequency, frequency) as it has a smaller value on the z-axis. The size of the sound data refers to the scale of the sound, and may be represented by a frequency.

Generates sound data in preparation for A101 to Q116, that is, for each distance profile image data, and generates left audio data through sound data located on the left side of the distance profile image data based on the x-axis center. The right auditory data is generated through the sound data located on the right side.

For example, in the column A, the left audio data is generated through sound data compared to the column I in column A, and the right audio data is generated through the sound data compared to the column Q in column J. FIG. The method of generating left or right auditory data through the sound data may be described as the sum of the respective sound data, and means transmitting the sound data simultaneously to the output unit.

If the front of the user is as shown in FIG. 4, the user may hear sound data corresponding to A101 to I116 having a large loud sound and a small loud sound through the left output unit. You can also hear the sound data corresponding to J101 ~ Q116 through the right output. The sound data corresponding to J101 to Q116 are farther than the left side but relatively close to the user, and the lower side is slightly lower than the left side, and therefore is a large size not larger than the left side and includes a low tone not lower than the left side. It will be smaller in size than the left, and will contain high notes not higher than the left. In this case, the user may not know exactly what, but the left and right outputs will recognize that there is something closest to the user in the lower left.

In addition, it is possible to distinguish the respective sound data output by the left and right output unit according to the degree of training of the user can recognize the front more accurately. The amount of visual information converted into auditory information may be adjusted according to a user's proficiency of using a device capable of distinguishing the sound data. The adjustment of the visual information may include a method of converting the visual information in the near distance to the auditory information or converting the visual information to the auditory information in the center in focus. However, this is only an example, and it is obvious that a method of controlling the amount of visual information can be used in more various ways. The apparatus may adjust the amount of visual information to be converted into auditory information according to the user's proficiency of recognizing auditory information.

Second embodiment

A second embodiment of the present invention proposes a visual representation apparatus using hearing that includes an input unit including at least one image sensor. 5 exemplarily shows an embodiment of the second embodiment.

The image sensor may use CCD (Charge Coupled Device) or CMOS technology. The CCD (Charge Coupled Device) is a technology that has been used for a long time in designing or manufacturing an image sensor, that is, a camera, and is mainly applied to a high quality camera. The CCD (Charge Coupled Device) has an excellent illumination sensitivity capable of generating an image of excellent quality even under low lighting conditions, and has an advantage of obtaining excellent color and vivid images compared to CMOS technology. It also has the advantage of having low background noise. The CCD (Charge Coupled Device) is a method widely used in various devices in which an image sensor is used. In addition, the CMOS technology is a standard technology that a chip such as a memory or a microprocessor is manufactured, and can be used more easily than the CCD technology. The CMOS technology does not require special components and costs a lot more than the CCD technology due to the standard process. In addition, there is an advantage that is easy to manufacture a small size camera. It is apparent that the technique of the image sensor included in the input unit of the visual reproducing apparatus using the hearing proposed by the present invention is applicable to both methods.

The image sensor stores the image (image) information in a digital form, and the image information includes a plurality of pixels, and the pixel includes color information, contrast information, saturation information, etc. indicating the position of the pixel and the pixel. It may include.

If there are two or more image sensors constituting the input unit, the two or more image sensors preferably have the same characteristics in order to perform less computation.

Image data, which is data about an image of the front inputted by the image sensor, includes a plurality of pixels, and includes color information, contrast information, and chroma information included in each pixel, and focal characteristics and aperture characteristics of the image sensor. The distance between the image sensor and the portion shown in each pixel may be measured by using the same or the like.

More preferably, in order to obtain more accurate three-dimensional information, including the at least two or more image sensors, the distance between the two image sensors and the distance between the image data generated by the two image sensors may be measured have. As described above, the method of obtaining three-dimensional information through two or more image data does not require any special equipment other than the image sensor, while performing calculation for each pixel included in the image data generated by the image sensor. Since it should be, there is a disadvantage in that a large amount of calculation. However, as a three-dimensional recognition method similar to the human visual processing method, more accurate three-dimensional information can be obtained, it is a frequently used method for three-dimensional implementation. Hereinafter, a process of obtaining three-dimensional information through the two image data will be described.

The input unit including two image sensors spaced apart at predetermined intervals and positioned to look in the same direction generates two image data and transmits the two image data to the control unit. Preferably, the two image data are generated at the same time, and are preferably generated at predetermined time intervals in succession. More preferably, the two image sensors are located on the left and right sides of the user.

The controller receives two image data including information of each pixel, analyzes pixel information included in the two image data, and searches for a portion representing the same point.

In more detail, the feature points included in the information of each pixel included in the left image data generated by the left image sensor and the right image data generated by the right image sensor are analyzed, and intersecting points, boundary lines, The feature point is recognized through consecutive points of the pixel having information different from the periphery of edge valleys, areas, and the like. Examine whether the feature point of one of the image data corresponds to the point representing the same point as that of the other image data. The review of the corresponding point may be performed by using the analysis of similar or identical areas of brightness information, similarity of brightness information, and the like as variables.

After recognizing the corresponding point of the left image data and the feature point of the left image data as described above, after reviewing the two-dimensional displacement of the corresponding point of the two image data, the two-dimensional displacement and the two image data generated Using distances between two image sensors, distances between the two image sensors and the points representing the respective two-dimensional displacements are calculated and generated as distance data.

FIG. 6 is a block diagram briefly illustrating a method of generating distance data based on information included in pixels included in image data generated by the image sensor.

As shown in Fig. 6A, (x ₁ , y ₁ ) of image 1 (right) and (x ₂ , y ₂ ) of image 2 (left) represent the external space w. With respect to the focal point of the image sensor, x ₁ corresponds to the X ₁ coordinate of w and x ₂ corresponds to the X ₂ coordinate of w. Therefore, the distance between the image sensor and w can be obtained by the method according to Equation 2, and the distance is represented by Z. 'B' in Equation 2 means a distance between two image sensors, as shown in FIGS. 5A and 6B.

In addition to the above method, after analyzing two image data by overlapping, finding the inconsistency of the two image data, and generating a disparity image (Disparity Image) indicating the inconsistency of the two image data, After filtering, the image may be converted into an image representing a distance through a variable representing contrast or color, and then a distance data may be calculated using the distance image. In this case, the distance data may be generated as the ratio using the focal length of the image sensor, the maximum value of the horizontal and vertical values of the distance image, and the maximum value of the horizontal and vertical values of the image data.

FIG. 6C illustrates the disparity image and the disparity image as a distance image through filtering. The left image of FIG. 6C is an original mismatched image, and contrast is distinguished by color and brightness. In addition, the intermediate image of FIG. 6C more regularly shows pixel values that appear irregularly by performing post-processing on the left image of FIG. 6C. Shown on the right side of FIG. 6D is a numerical representation of the degree of discrepancy with respect to the left image of FIG. 6C. In addition, the right image of FIG. 6C is converted into a distance image by performing post-processing on the intermediate image of FIG. 6C. As shown in the image on the right of FIG. 6C, a place where the distance from the user is close is represented by a smaller contrast value, and a place where the distance from the user is far away is represented by a large contrast value. 6E, the distance data of each pixel included in the right image of FIG. 6D is expressed numerically. The larger the value, the greater the distance between the point indicated by the pixel and the user.

The processor of the controller analyzes at least one or more image data transmitted from an input unit including at least one image sensor in the same manner as described above, and thus three-dimensional information on a point represented by the pixel for each pixel included in the image data. Create

Thereafter, sound data for the plurality of 3D information is generated in the same manner as in the first embodiment. In more detail, the sound data is generated such that the selected three parameters are changed according to the three-dimensional information by selecting among loudness, height, type, parallax, and stereo sound.

More preferably, the sound data becomes smaller in height as the distance data (z coordinate in the description of the second embodiment) becomes larger (frequency is increased), and is located higher in the image data (y in the description of the second embodiment). The larger the coordinate value), the larger the size. In addition, the sum of sound data for pixels located on the right side (based on the x coordinate in the description of the second embodiment) is generated as right audio data, and the sum of sound data for pixels located on the left side is generated as left audio data.

Thereafter, the controller transmits the right auditory data and the left auditory data to an output unit.

As in the first embodiment, the user can adjust the amount of visual information converted into auditory information according to the user's skill in using the device that can distinguish the sound data. The adjustment of the visual information may include a method of converting the visual information in the near distance to the auditory information or converting the visual information to the auditory information in the center in focus. However, this is only an example, and it is obvious that a method of controlling the amount of visual information can be used in more various ways. The apparatus may adjust the amount of visual information to be converted into auditory information according to the user's proficiency of recognizing auditory information.

The present invention detects the visual information of the front through a plurality of distance sensors or at least one or more image sensors as a whole, and configures the visual information to three-dimensional information, by converting the three-dimensional information to auditory information, to the user, There is an advantage that the user can fully recognize the three-dimensional visual information represented by the auditory information.

In addition, the present invention divides a plurality of sound data recognized according to the user's proficiency, so that the overall visual information can be obtained, and the sound data for a point close to a distance makes a loud sound, thereby increasing the user's recognition level. There is one advantage.

In addition, the left and right positions of the object are separated into the left output unit and the right output unit so that the left and right positions of the object can be distinguished.The distance to the object and the height of the object are two sound parameters selected from the size, height, type, and parallax. By showing through, there is an advantage that can be more safely walking by recognizing the visual information located in front.

Claims (26)

At least one distance distance sensor array comprising a plurality of distance sensors composed of a transmitter for transmitting a transmission signal having a frequency to the outside and a receiver for receiving a reception signal reflected by the transmission signal in real time spaced at a predetermined interval An input unit to perform; A timer for measuring a reception time which is a time from when the transmission signal is transmitted by the transmission unit to the reception unit detecting the reception signal generated by the reflection of the transmission signal, characteristics of the transmission signal or reception signal, and the reception signal; Each distance data of the plurality of distance sensors is generated through time, and the distance sensor is located in the distance distance sensor array, the horizontal array is x-axis data, the vertical array is y-axis data, and the generated distance data. Generate range profile image data including three-dimensional information using z-axis data, and select from among loudness, height of sound, type of sound, parallax of sound and stereo sound. A plurality of distance profile image data included in the distance profile image data such that the selected three parameters change according to the axis data, the y axis data, and the z axis data. A digital signal processor (DSP) for generating respective sound data for three-dimensional information and generating left and right auditory data output to the left and right output units through the plurality of sound data; A control unit including a processor configured in a combination including at least one of a micro-processing unit (MPU); And And an output unit configured as one of an earphone, a headphone, or a bone conduction device in which left and right sides are distinguishable, and outputting the left auditory data and the right auditory data so that a user can listen. The output unit outputs left and right auditory data generated through a plurality of sound data generated with respect to the distance profile image data, and simultaneously outputs a plurality of sound data included in the left and right auditory data. Visual representation apparatus using hearing. The method of claim 1, The outgoing signal having a frequency of the distance sensor included in the input unit is a visual representation apparatus using the auditory, characterized in that any one of ultrasonic waves, infrared rays, electromagnetic waves, laser. The method of claim 1, The control unit converts the received signal transmitted from the input unit into an analog signal to a digital signal to be processed by a digital signal processor (DSP) or a micro-processing unit (MPU). / D converter (Analog / Digital Converter) characterized in that it further comprises a visual reproducing apparatus using the auditory. The method of claim 1, The control unit further includes a pulse generator for adjusting the interval of the outgoing signal emitted from the transmitter of the input unit to distinguish the outgoing signal generating the received signal when the receiver detects the received signal, characterized in that Visual representation apparatus using hearing. The method of claim 1, And the control unit further comprises an adjusting unit for adjusting a portion of the distance profile image data to convert only a portion of the distance profile image data into sound data. The method of claim 1, And the distance data is generated by dividing a frequency or amplitude of an outgoing signal or a received signal by the reception time by two. The method of claim 1, The distance data is a visual representation device using the auditory, characterized in that the sound speed data generated by the sound speed calculation method using the temperature as a variable divided by the reception time divided by two. The method of claim 7, wherein The control unit may further include a thermometer for measuring the temperature for measuring the temperature, which is one of the variables of the sound velocity calculation method, and transmitting the digital speed data to the control unit to generate the sound speed data. Visual representation apparatus using hearing. The method of claim 7, wherein The distance data is a visual representation apparatus using the auditory, characterized in that generated by the average sound speed by setting the temperature used in the sound speed calculation method. The method of claim 1, The distance sensor array of the input unit is a visual representation device using the auditory, characterized in that spaced apart at a predetermined distance on the left and right with respect to the user. The method of claim 10, The x-axis data located on the left side of the x-axis center is generated by the distance sensor included in the distance distance sensor array on the left side, and the x-axis data on the right side is generated by the distance distance sensor array on the right side. A visual representation apparatus using hearing. The method of claim 1, As the y-axis data increases, the sound data becomes louder (the frequency becomes smaller), and as the z-axis data increases, the loudness (sound pressure or intensity, sound pressure level (dB number)) becomes larger. The sound data for the distance profile image data having the x-axis data located on the left side of the center of the x-axis is generated as left audio data, and the sound data for the distance profile image data having the x-axis data on the right side is generated. A visual representation apparatus using hearing, which is generated by right auditory data. delete An input unit including at least one image sensor; A pulse generator configured to set an operation interval of the image sensor, a plurality of pixels included in the image data input through the image sensor, the horizontal arrangement of which is located in the image data as x-axis data, and the vertical arrangement as y-axis data And generating three-dimensional information using z-axis data as a distance between the pixel generated by the color information, the contrast information, the chroma information, the saturation information, and the characteristics of the plurality of image sensors. And selecting one of loudness, height, type, parallax, and stereo sound to obtain respective sound data for the plurality of pixels in which the selected parameter changes according to the x-axis data, y-axis data, and z-axis data. Generating left and right auditory data output to a left output unit and a right output unit through the plurality of sound data; The controller including a; digital signal processor (Digital Signal Processor:: DSP) or a compact processing unit (MPU Micro-processing unit) And And an output unit which can be used as one of an earphone, a headphone, or a bone conduction device in which left and right sides are distinguishable, and outputs the left auditory data and the right auditory data so that a user can listen. The output unit outputs left and right auditory data generated through a plurality of sound data generated with respect to the distance profile image data, and simultaneously outputs a plurality of sound data included in the left and right auditory data. Visual representation apparatus using hearing. The method of claim 14, The control unit may control the digital signal from the analog signal so that the value recognized by the plurality of image sensors of the input unit may be processed by a digital signal processor (DSP) or a micro-processing unit (MPU). An audio / visual reproducing apparatus, comprising: an A / D converter (Analog / Digital Converter) for converting into a signal. The method of claim 14, The input unit includes an image sensor positioned on the left and right sides of the user, and the left image sensor and the right image sensor is a visual representation apparatus using the auditory, characterized in that positioned to recognize the same point. The method of claim 16, The control unit controls the left image sensor and the right image sensor to operate at the same time through the pulse generating unit, and the recognition by the left image sensor generates the left image data, the right image sensor is recognized by the right image data Visual representation apparatus using the auditory, characterized in that the generation. The method of claim 17, The processor of the controller examines the color information, the contrast information, the chroma information of the pixel included in the left and right image data, and recognizes the pixel included in the left image data and the right image data indicating the same point. And generating the z-axis data through a gap between a position at which pixels representing the same point are arranged in left image data and right image data, and a left and right image sensor generating the left and right image data. A visual representation apparatus using hearing. The method of claim 14, As the y-axis data becomes larger, the sound data becomes louder (the frequency becomes smaller), and as the z-axis data becomes larger, the loudness (sound pressure or intensity, sound pressure level (dB number)) becomes larger. The sound data for the distance profile image data having the x-axis data located on the left side of the center of the x-axis is generated as left audio data, and the sound data for the distance profile image data having the x-axis data on the right side is generated. A visual representation apparatus using hearing, which is generated by right auditory data. delete The method of claim 14, The visual representation apparatus using the auditory sound, characterized in that additionally generating data on the color or texture for each pixel through the spatial image data, the sound data includes a tone contrasted with the data for the color or texture . The method of claim 14, The control unit further comprises a control unit for adjusting a portion of the distance profile image data to convert only a portion of the image data input from the at least one image sensor into sound data. The method according to claim 1 or 14, The input is located in the lens portion of the goggles, the output is a visual reproduction device using the auditory, characterized in that the goggle shape is positioned so as to move freely near the ear of the goggles. The method according to claim 1 or 14, And the input unit is located on the lens portion of the glasses, and the output unit is in the shape of glasses positioned so as to move freely while being close to the ear of the goggles. The method according to claim 1 or 14, And a sight or goggle shape positioned on a hinge portion of the input additional glasses or goggles and positioned so as to move freely while being close to an ear fixing portion of the output additional goggles. The method according to claim 1 or 14, The visual representation apparatus using hearing, characterized in that the input unit and the output unit is a headphone shape provided in each of the headphones.

Images