RU2649422C2 - Method of image conversion to sound image - Google Patents

Abstract

FIELD: acoustics; data processing.

SUBSTANCE: invention relates to acoustics, in particular to methods for converting visual information into a sound form for its perception by the blind. A method for converting a digital bitmap image into an audio image for an arbitrary freeze-frame image or a single frame of a video image represented digitally as a two-dimensional pixel matrix includes the input of digital data of the pixel array of an image into the RAM of the microprocessor system, processing by a microprocessor, forming a one-dimensional data array in digital form from image data, subsequent reproduction of the received signal in the form of sound vibrations by means of an audiosystem. In this, two-dimensional pixel array of a bitmap image is dissected in a one-dimensional pixel array along the trajectory of an unwinding spiral from the center of the image to the edge, on the basis of the resulting one-dimensional pixel array, an array of digital sound wave data is formed, in which the amplitude of the sound wave is proportional to the brightness of the pixels, the frequency of the sound wave is encoded by the color of the pixels.

EFFECT: reduction in the number of computing resources.

4 cl, 4 dwg

Description

The invention relates to methods for converting visual information into other types with a view to its perception by the visually impaired (blind) or people with impaired vision. The present invention provides the conversion of visual information into a sound form (perceived by hearing). After learning perception, a mobile system based on this invention should replace blind vision.

In addition to the main application described above, this invention should also help sighted people ease the task of visual monitoring in applications such as video surveillance, object protection or the perception of additional views (for example, the perception by the car driver of an image from a rear view camera without being distracted from the main front view).

There are various ways to present visual information in audio form. For example, a method of forming an acoustic image [patent of the Russian Federation 2119785, 1998.10.10] requires a complex four-component system of audio sources that cannot be used in a mobile device.

The method and system for obtaining an audio image from graphic data [European patent EP 1369839, 2003.12.10] allows you to create audio images for small-sized images, such as characters. However, this method is currently not relevant due to the presence of direct TTS (text-to-speech = text-to-speech) systems that are capable of voicing symbolic information after recognizing text using a computer program, and for presenting an arbitrary image, this method unsuitable due to limited resolution.

The invention [Japanese patent JP 2004020869, 2004.01.22] uses a panoramic camera (all azimuth imaging apparatus), which allows you to receive an audio signal that this place is familiar to you.

The method of presenting video information [RF patent 2223552, 2004.02.10] is intended to represent color images as a combination of an array of convex and flat points (tactile perception) and sound signals (perception by hearing). This method gives a low resolution and requires a fairly long time to "feel" the image.

The color identification device [patent WO2005085781, 2005.09.15] emits only one color in the image, and this color converts into an audio signal. By this sound signal, a person recognizes the color. Obviously, this device provides too little information to target the blind.

The closest analogue (prototype) of the present invention is a method and device for visualizing images using sound [US patent US 6963656, 2005.11.08]. This method involves encrypting spatial information in the form of a polyphonic musical sequence. Due to the insufficient resolution of this method, it is supposed to divide the image into parts with the allocation of some features that are encoded in parts.

The disadvantage of the prototype is that this method is not universal due to the infinite variety of possible splitting of an arbitrary image into separate parts.

For a further review of the present invention, we define an understanding of the following terms.

A digital raster image is a collection of numerical elements (pixels, see below), which with a certain accuracy and scale represent a certain image that is either direct or indirect (for example, through the imagination of the artist) a reflection of the material world. These numerical elements can be processed in microprocessor systems, such as a computer, mobile phone, digital camera or video camera. For ease of processing, this set of numerical elements is represented as a two-dimensional array, so that a single image element has two indexes that determine the location coordinate of this element in the rectangular region bounding the image.

Pixel (English Pixel = Picture Element) - “the smallest element of the visualization surface, which can be independently set the color, intensity and other characteristics of the image” (GOST 27459-87). The pixel size determines the scale of the image, which is not significant for the purpose of presentation in audio form. Therefore, in the description below, we will mean that each individual pixel as an image element is characterized only by color and intensity (brightness).

So, the information perceived by vision can be converted into the form of a digital raster image using a scanner, camera or video camera. The digital data obtained by processing can be converted in one way or another into the form of sound vibrations, which when reproduced are perceived as sound (acoustic) images.

The essence of the present invention is that a two-dimensional data array (brightness and color for each pixel) of a raster image using a spiral scan (starting from the center of the image to its periphery) is represented as a one-dimensional array, which serves as the basis for the synthesis of a digital sound wave data array . In this case, the pixel luminance data forms the current amplitude (envelope) of the sound wave, and the pixel color data form the magnitude of the current sound wave frequency. For a monochrome image, the audio signal will be solid (the signal will only vary in amplitude).

In order to make it clear which part of the spiral the sound signal is currently displaying, after each half-turn of the spiral, one of the two splitter signals is inserted into the sound sequence of signals, having frequencies below and above all possible frequencies encoding the color of the image. For example, if a spiral starts from the top pixel of the initial revolution of the spiral, then after each odd half-turn, a low-frequency separator is inserted (at the bottom of the spiral), and after every even half-revolution a high-frequency separator is inserted (at the top of the spiral).

In the case of large images (with a large number of pixels), in order to obtain an acceptable sound wavelength, the image at the periphery is “roughened”, pixel data is combined and averaged. As a result, one “virtual” pixel replaces several original ones. For example, four adjacent pixels (a 2x2 square) are replaced by one equivalent pixel, in which the color will correspond to a frequency equal to the arithmetic average of the color frequencies of the four source pixels, and the brightness of the equivalent pixel will be equal to the arithmetic average of the brightness of the four source pixels. As you move away from the starting point of the spiral, the number of pixels to be combined increases. For example, at first a certain number of spiral revolutions is encoded without combining pixels, then several spiral revolutions are combined and averaged over 4 (2x2) adjacent pixels, capturing a 2-fold wider band of pixels, then a certain number of spiral revolutions are combined and averaged over 9 (3x3) adjacent pixels, then several turns of the spiral with the union and averaging of 16 (4x4) neighboring pixels, etc. Thus, the time required to reproduce the part of the sound wave corresponding to one revolution of the spiral on the periphery of the image, covering a wide area of the image, is significantly reduced. At the same time, the resolution decreases - the information from the peripheral parts of the image is presented in a generalized form - just like the human eye perceives information from the peripheral part of the picture when the gaze is concentrated in its center.

The central point, from which the spiral-like scanning of the image begins, serves as an analogue of the “point of view", i.e. places where the person’s eyes are focused when he begins to examine the picture. A sound wave formed on the basis of a spiral scan, starting from a certain starting point of the image, different from the central one, will be an analogue of “moving eyes” when viewing the picture with eyes. Moving the starting point of the scan from the center of the image to another point in the image, receiving and reproducing the corresponding sound images for hearing is similar to sequential viewing of the image, when the eyes focus the eye sequentially on different parts of the image. At the same time, the area directly adjacent to the starting point, where there is still no union and averaging of these pixels, will have the best resolution. Therefore, this area will be the basis for the formation of the most informative part of the sound wave. The presence of a pointing device in the system (mouse, touchpad, touch screen) will allow the blind to move the virtual “look” and repeatedly generate sound images for the same bitmap, perceiving which it will be possible to recognize the image presented in the picture.

Figure 1 shows the principle of a spiral scan of raster image data into a one-dimensional array for encoding in order to obtain a sound wave signal and its implementation options.

Figure 2 shows the principle of encoding pixel data using a set of sound waves for monochrome and color images. For the color image option, insertion of separation signals is shown.

Figure 3 shows the combination of pixels with coarsening of the resolution as the sweep moves from the starting point of the spiral to the peripheral portions of the image to quickly capture the entire image.

Figure 4 shows the sequential formation of several audio signals for a single raster image by changing the starting point of the spiral scan for better recognition of the images present in the image.

The implementation of the invention does not present technical problems. As a source of a raster image, a digital camera or digital video camera, a mobile phone camera, a web camera of a laptop, tablet or desktop computer can be used.

The conversion of video data into audio data according to the present method is carried out in a universal way, independent of what is shown in the image. Such a transformation is not associated with pattern recognition, and, therefore, it will not require large computational resources comparable to the needs of artificial intelligence. Therefore, it can be carried out using a mobile computer or microprocessor system in real time. As such a system, for example, the microprocessor system of a digital camera, digital video camera, mobile phone / smartphone, pocket / tablet computer, laptop can be used. If there is no mobility requirement, you can use a desktop computer.

For almost all of the listed devices, except for a digital camera, the sound reproduction system and pointing device (mouse, trackball, touchpad - touch panel or touch screen) are standard elements that are included in the package. Therefore, if there is a program that implements the above method of converting an image into an audio image, these systems can become a device that receives a technical result according to the present invention.

Claims (20)

1. A method of converting a digital raster image into an audio image for an arbitrary static image or a single frame of a video image, presented in digital form as a two-dimensional matrix of pixels, including: - input of digital data of an array of image pixels into the RAM of the microprocessor system; - processing using a microprocessor, forming from the image data a one-dimensional array of data in digital form; - subsequent reproduction of the received signal in the form of sound vibrations using the audio system, characterized in that - a two-dimensional array of pixels of a raster image is expanded into a one-dimensional array of pixels along the path of a spinning spiral from the center of the image to the edge, - based on the resulting one-dimensional array of pixels, an array of digital sound wave data is formed, in which - the amplitude of the sound wave is proportional to the brightness of the pixels, - The frequency of the sound wave is encoded in the color of the pixels. 2. A method of converting a digital raster image into an audio image according to claim 1, characterized in that after every half-turn and each turn of the spiral, separator signals are inserted, the frequency of which is more and less than the frequency values of all signals that encode the colors of the pixels. 3. The method of converting a digital raster image into an audio image according to claim 1 or 2, characterized in that - on the peripheral part of the scan path, the luminance and color data are averaged for several adjacent pixels, so that one point of the audio data of the amplitude / frequency of the sound wave is formed on the basis of several pixels of the original image, and - the number of pixels combined for averaging increases as you move away from the start point of the scan. 4. The method of converting a digital raster image into an audio image according to claim 1 or 2, characterized in that - reproduction of the sound wave for the same image can be interrupted by the listener and repeated several times from the beginning, - the beginning of the sweep using the pointing device is shifted by the listener to any point in the image other than the previous starting point, and for repeated playback, a modified sound wave is generated with a new sweep starting from the specified point.

Images