KR20230047889A - Apparatus and method for sensory substitution and multi-path transmission of visual information - Google Patents

Abstract

The present invention relates to an apparatus and method for sensory substitution and multi-path transfer of visual information to efficiently transfer visual information to a user. According to the present invention, the apparatus comprises: a target information extraction module extracting information corresponding to a type requested by a user from externally received image information on the basis of user input information; a sensory path determination module determining at least one sensory path through which the extracted information is transferred on the basis of the user input information; a replacement signal generation module converting the extracted information to generate a replacement signal corresponding to each of the at least one sensory path; and a control and output module outputting the replacement signal through the user's sensory signal transfer device.

Description

Apparatus and method for sensory substitution and multi-path transmission of visual information

The present invention substitutes and transmits information about a specific sense based on brain plasticity by substituting information that can be sensed by other sensory organs, thereby allowing the brain to recognize information about the specific sense from a converted signal. It relates to an apparatus and method. The present invention includes not only a technology for simply replacing sensory information, but also a technology and method for transmitting visual information as various sensory signals so that a user can have enhanced senses.

In general, sensory substitution is when the sensory information of the body whose functions are damaged or deteriorated due to accidents or aging is converted into other sensory signals and transmitted, and the user who has gone through a period of learning process replaces the original sensory information with the sensory information. It means a way to make it perceptible through. It is known that such a sensory substitution technique is based on a characteristic in which the human brain structurally or functionally changes and reorganizes to adapt to a new environment, that is, brain plasticity.

It is known that about 80% of the information that people perceive from the outside in their lives is made through vision. Therefore, vision is the most important information collection organ among the sense organs of the human body, and when vision is lost, problems such as inconvenience in walking and difficulty in grasping situations occur. In order to cope with these problems, various studies have been conducted to convert visual information into other sensory signals and transmit them.

Existing methods for converting visual information into other sensory signals include an auditory replacement technology for converting visual information into an auditory signal and a tactile replacement technology for converting visual information into a tactile signal. Existing auditory or tactile substitution technologies mainly convert input images into black and white or gray images through a pre-processing process, and then convert them into sensory signals other than visual signals according to pre-defined rules and transmit them.

vOICe, the most representative technology in auditory replacement technology, converts the input image into a gray scale 64x64 resolution 2D image, divides the image frame into 64 columns based on the time axis, and outputs pixel position and brightness information in each column as sound. It is a technology that generates sound information by mapping it to the frequency and amplitude of the sound and then transmits it. In addition, among auditory replacement technologies, EyeMusic converts images or videos into gray scale and processes them to solve the problem of loss of color information, which is an important element in images, by clustering the same color area in the input image and different colors for each color. It is a technology that maps the sound of an instrument and transmits it as a sound signal.

In general, visual information recognized by humans includes a variety of information, including shape including outlines, position of objects, distance, color, texture, path or direction, presence or absence of light, atmosphere, or emotion. However, existing auditory substitution or tactile substitution technologies cannot transmit various visual information because information limited to shape or color among visual information is replaced with other sensory information and provided. Specifically, the existing sensory substitution technology that transmits visual information by replacing it with other sensory signals has a problem in that it cannot effectively convey various visual information to the user by converting images or images in a predetermined way and delivering them to a single sensory organ. there is.

In addition, since the existing sensory substitution technology follows a method of simply transmitting signals, there is a problem in that the visual information required by the user cannot be selectively transmitted.

On the other hand, in the virtual/augmented environment represented by the non-face-to-face society and metaverse recently introduced by the corona, the need for technology for efficient transmission of visual information and augmentation or improvement of users' senses is increasing. However, the sensory replacement technology described above has a limitation in that it can provide only a detour path for simply providing lost functions.

In addition, in the case of most sensory information, the amount of information that can be sensed by sensory receptors may be limited depending on the characteristics of the sensory organ, and since the existing sensory substitution technology utilizes a single alternative sensory information (transmitted through a single path), Therefore, there is a limit to faithfully conveying the visual information of a video or image.

In order to overcome the above problems and limitations, the present invention converts a plurality of visual information that a user wants to receive among various visual information included in a video or image into other sensory signals suitable for the characteristics of the information, and converts the plurality of visual information into a plurality of sense signals. An object of the present invention is to provide an apparatus and method for sensory substitution and multi-path transmission of visual information, which can efficiently deliver visual information to a user by transmitting it to a dog's sensory organ.

In particular, if the main purpose of the existing hearing or tactile replacement technologies was to help the visually impaired people walk or live, the present invention is not only for the visually impaired, but also for military operations that require confidential information transmission and in extreme working environments underground or late at night. To efficiently transmit visual information or to maximize the user's experience in virtual reality or augmented reality represented by the metaverse, to provide a device and method for sensory substitution and multipath transmission of visual information that can augment and transmit visual information. for that purpose

The object of the present invention is not limited to the object mentioned above, and other objects not mentioned will be clearly understood by those skilled in the art from the description below.

An apparatus for sensory substitution and multipath transmission of visual information according to an embodiment of the present invention includes a target information extraction module for extracting information corresponding to a type requested by a user from image information received from the outside based on user input information; a sensory path determining module configured to determine at least one sensory path through which the extracted information is transmitted based on the user input information; an alternative signal generation module for generating an alternative signal corresponding to each of the at least one sensory pathway by converting the extracted information; and a control and output module outputting the alternative signal through the user's sensory signal transmission device. The type is included in the user input information.

In an embodiment of the present invention, the at least one sensory pathway is a plurality of sensory pathways, and the alternative signal generating module may convert the extracted information to generate alternative signals corresponding to each of the plurality of sensory pathways. there is.

In one embodiment of the present invention, the plurality of sensory pathways may include sensory pathways corresponding to at least two or more senses among sight, hearing, touch, and thermal sensation.

In one embodiment of the present invention, the target information extraction module may generate total visual information including at least one of shape information, spatial information, global information, and composite information by analyzing the image information, and the user Based on the input information, information corresponding to the type may be extracted from the total visual information.

In one embodiment of the present invention, the control and output module synchronizes the alternative signals corresponding to each of the plurality of sensory pathways based on the frame of the image information, and transmits the synchronized alternative signals to the sensory signal transmission device. can be output through

In one embodiment of the present invention, the control and output module recognizes the user's behavior based on the image information, corrects characteristics of the alternative signal based on the user's behavior, and corrects the corrected alternative signal. Can be output through the sensory signal transmission device.

In one embodiment of the present invention, the control and output module corrects the characteristics of the substitute signal based on the position of the image information input device and the position of the sensory signal transmission device, and converts the corrected substitute signal into the It can be output through a sensory signal transmission device.

In one embodiment of the present invention, the user input information may include information about transmission strength. In this case, the control and output module corrects the strength of the substitute signal according to the transmission strength, and outputs the corrected substitute signal through the sensory signal transmission device.

In one embodiment of the present invention, the control and output module may calculate the time required to convert the extracted information into the alternative signal by the alternative signal generating module. In this case, the alternative signal generation module determines whether to simplify the process of converting the extracted information into the alternative signal based on the time.

Further, a method for sensory substitution and multipath transmission of visual information according to an embodiment of the present invention includes extracting information corresponding to a type requested by a user from image information received from the outside based on user input information; determining at least one sensory pathway through which the extracted information is transmitted based on the user input information; generating an alternative signal corresponding to each of the at least one sensory pathway by converting the extracted information; and outputting the alternative signal through the user's sensory signal transmission device.

In one embodiment of the present invention, the at least one sensory pathway may be a plurality of sensory pathways. In this case, in the generating of the alternative signal, the extracted information may be used to generate an alternative signal corresponding to each of the plurality of sensory pathways.

In one embodiment of the present invention, the plurality of sensory pathways may include sensory pathways corresponding to at least two or more senses among sight, hearing, touch, and thermal sensation.

The method for transmitting sensory substitution and multipath of visual information according to an embodiment of the present invention further includes an alternative signal synchronization step of synchronizing the alternative signal based on a frame of the image information after the generating of the alternative signal. can do.

In the method for sensory substitution and multi-path transmission of visual information according to an embodiment of the present invention, after the step of generating the substitution signal, the substitution is performed based on the location of the image information input device and the location of the sensory signal transmission device. An alternative signal correction step of correcting the characteristics of the signal may be further included. In this case, in the outputting step, the corrected replacement signal is outputted through the sensory signal transmission device.

In one embodiment of the present invention, in the step of extracting the information corresponding to the type requested by the user, the image information is analyzed to provide a total view including at least one of shape information, spatial information, global information, and complex information. Information may be generated, and information corresponding to the type may be extracted from the total visual information based on the user input information.

In one embodiment of the present invention, in the outputting, alternative signals corresponding to each of the plurality of sensory pathways are synchronized based on a frame of the image information, and the synchronized alternative signals are transmitted through the sensory signal transmission device. can be printed out.

In one embodiment of the present invention, the user input information may include information about transmission strength. In this case, in the step of generating the replacement signal, the strength of the replacement signal may be corrected by reflecting the transmission strength.

In one embodiment of the present invention, in the step of generating the alternative signal, a time required to convert the extracted information into the alternative signal is calculated, and the extracted information is converted into the alternative signal based on the time. It is possible to determine whether to simplify the conversion process, and to generate the replacement signal by converting the extracted information based on the determination result.

According to an embodiment of the present invention, user-customized information can be created by extracting at least some of the visual information that the user wants to receive from among various visual information such as shape information, spatial information, and complex information generated from an image.

According to an embodiment of the present invention, it is possible to provide an improved sense of immersion to the user by converting the extracted visual information into other sensory signals suitable for each characteristic of the visual information. According to an embodiment of the present invention, a sensory signal converted from the extracted visual information may be corrected according to a user's action to provide an improved sense of immersion to the user.

According to an embodiment of the present invention, by synchronizing and transmitting a plurality of converted sensory signals from image information, it is possible to induce a user to more accurately recognize necessary visual information through alternative senses.

In addition, since various visual information can be accurately transmitted through the present invention, the present invention supports the walking / life of the visually impaired, which was the scope of application of the existing sensory replacement technology, as well as military operations and underground or It has the effect of being applicable to the extreme working environment at night.

In addition, according to the present invention, visual information can be applied to various services in which users can experience visual information by delivering personalized visual information more efficiently and realistically in an immersive virtual environment, augmented reality, or non-face-to-face society. there is.

The effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood by those skilled in the art from the description below. will be.

1 is a block diagram showing the configuration of an apparatus for sensory substitution and multipath transmission of visual information according to an embodiment of the present invention.
2 is a block diagram showing the detailed configuration of an apparatus for sensory substitution and multi-path transmission of visual information according to an embodiment of the present invention.
3 is a diagram showing the concept of sensory substitution and multipath transmission of visual information according to the present invention by way of example.
4 is a block diagram showing the configuration of an apparatus for sensory substitution and multi-path transmission of visual information according to another embodiment of the present invention.
5 ( FIGS. 5A and 5B ) is a flowchart illustrating a method for sensory substitution and multipath transfer of visual information according to an embodiment of the present invention;

Visual information perceived by the human eye is shape information including the shape and color of an object existing in an image, presence or absence of light and texture, spatial information in geometric aspects such as location, distance and direction, and Global information such as color and shape extracted from the entire region of the image, including objects and backgrounds existing in the image, and multiple information such as shape information, spatial information, and global information, including path, emotion and atmosphere, etc. It is divided into complex information, etc. However, existing sensory replacement devices that convert visual information into information about other senses mainly convert and transmit two-dimensional color image information, so it is difficult to convey a sense of distance to the user, or it is difficult to convey a sense of distance to the user by converting the entire input image in a predetermined method. There is a disadvantage in that it is difficult for the user to efficiently receive various visual information by transmitting it to a single sensory organ. In addition, existing sensory substitution technologies are mainly developed for the purpose of conveying information for the visually impaired, so it is difficult to apply them to various fields.

Sensory replacement technology for various visual information included in video or image is used not only for visually impaired people, but also in military fields such as infiltration or reconnaissance, underground or accidental power outages, night work, and general industrial sites. It can be used to efficiently recognize the surrounding environment in extreme environments, such as safely conveying dangerous situations analyzed by visual information to workers. In addition, by synchronizing the information converted according to the transmitted visual information and delivering it through various routes, it can be utilized so that the user can feel the visual information more realistically in virtual reality or augmented reality.

According to the present invention, a plurality of visual information that a user wants to be received is received, and information requested by the user is extracted from various visual information such as shape information, spatial information, global information, and complex information included in the collected images. However, by converting the extracted information into other sensory signals suitable for its characteristics and delivering them to a plurality of sensory organs, information can be efficiently delivered to the user.

An apparatus and method for transmitting various visual information through a multi-path according to the present invention will be described through an embodiment in which visual information is converted into an auditory signal, a tactile signal, and a thermal sensation signal. However, the device and method of the present invention are not limited to the conversion method according to the embodiment and can be applied to other sensory information.

Advantages and features of the present invention, and methods of achieving them, will become clear with reference to the detailed description of the following embodiments taken in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but will be implemented in various different forms, only these embodiments make the disclosure of the present invention complete, and common knowledge in the art to which the present invention belongs. It is provided to fully inform the holder of the scope of the invention, and the present invention is only defined by the scope of the claims. Meanwhile, terms used in this specification are for describing the embodiments and are not intended to limit the present invention. In this specification, singular forms also include plural forms unless specifically stated otherwise in a phrase. As used herein, "comprises" and/or "comprising" means that a stated component, step, operation, and/or element is the presence of one or more other components, steps, operations, and/or elements. or do not rule out additions.

In this specification, 'request information' refers to information about the type of object that the user is interested in or information indicating the type of visual information that the user wants to receive from image information.

In this specification, 'requested sensory information' is information that a user requests for a sense to receive the requested information, and refers to information in which the user's intention is expressed with respect to a sense that is a medium for transmitting the requested information.

In this specification, 'transfer intensity information' refers to information expressing the transfer intensity of sensation.

In this specification, 'visual information' is used as a general term for information generated based on image information, and may include shape information, spatial information, global information, and composite information. Visual information referred to in this specification includes not only visually recognizable information such as colors and outlines, but also information obtained through analysis of image information, such as a distance between a user and an object.

In this specification, 'target information' refers to visual information that meets the user's requested information.

In this specification, 'sensory signal' refers to a signal transmitted to the user's sensory organ by the sensory substitution and multipath delivery device according to the present invention.

In this specification, 'alternative signal' is a sensory signal that replaces target information. The alternative signal is generated by the apparatus for sensory substitution and multi-path transmission of visual information according to the present invention to convert target information into another form and transmit it to the user's sensory organs. For example, if the target information is visual information such as 'color of the ball', the replacement signal may be a thermal sensation signal having a predetermined temperature determined according to the 'color of the ball'.

In the present specification, a 'sense organ' is a human body organ that feels and receives external stimuli, and refers to an organ composed of various receptors that accept senses such as touch, warmth, cooling, pain, and proprioception.

In the present specification, a 'sensory pathway' means a path along which nerve stimulation follows from a user's sensory organ to a sensory region of the user's brain. In this specification, 'determining a sensory path' means determining a user's sensory organ to which target information or an alternative signal is to be delivered.

In describing the present invention, if it is determined that a detailed description of related known technologies may unnecessarily obscure the subject matter of the present invention, the detailed description will be omitted.

Prior to describing the embodiments of the present invention, the core operation of the present invention will be described. The core operation of the present invention consists of four steps S1 to S4.

Step S1 is a step of generating target information based on user input information and image information. Specifically, the apparatus for sensory substitution and multipath transmission of visual information according to the present invention generates visual information (hereinafter, target information) meeting a user's request from image information received from the outside, based on user input information. . The target information may be one or may be a plurality of pieces of information. That is, the apparatus for sensory substitution and multipath delivery of visual information according to the present invention generates at least one piece of target information based on user input information and image information. More specifically, the apparatus for sensory substitution and multipath transfer of visual information according to the present invention extracts requested information from user input information and determines the type of visual information desired by the user based on the requested information. The device generates target information by extracting visual information corresponding to the determined type from image information.

Step S2 is a step of determining at least one sensory path through which target information is transmitted based on user input information. Specifically, the apparatus for sensory substitution and multipath delivery of visual information according to the present invention parses/extracts requested sensory information from user input information to determine a sensory pathway through which target information is delivered. The device determines one or more sensory pathways through which target information is conveyed.

Step S3 is a step of converting the target information into an alternative signal corresponding to the sensory path determined in step S2.

Step S4 is a step of outputting an alternative signal through the user's sensory signal transmission device.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. In order to facilitate overall understanding in describing the present invention, the same reference numerals will be used for the same means regardless of the drawing numbers.

1 is a block diagram showing the configuration of an apparatus 100 for sensory substitution and multi-path delivery of visual information according to an embodiment of the present invention.

An apparatus for sensory substitution and multipath delivery of visual information 100 according to an embodiment of the present invention includes an input module 110, a sensory path determination module 120, an image information division module 130, and a target information extraction module 140. ), an alternative signal generation module 150 and a control and output module 160. The device 100 for sensory substitution and multipath transmission of visual information shown in FIG. 1 is according to an embodiment, and components of the device 100 are not limited to the embodiment shown in FIG. may be incorporated, added, changed or deleted according to

The input module 110 receives user input information and image information from the outside. The user input information may include 'request information'. The 'request information' may include information about the type of object that the user is interested in or information indicating the type of visual information the user wants to receive from the image information. Examples of requested information about the type of object may include people, cars, and animals, and examples of requested information about the type of visual information may include shape, location, distance, direction of movement, color, and the like. there is. Examples of requested information in which the type of object and the type of visual information are mixed may include an animal's movement direction, a vehicle's location, and a distance to a person.

The user input information may further include at least one of information about a sense to receive the requested information (hereinafter referred to as 'requested sensory information') and information expressing a transmission intensity of a sense (hereinafter referred to as 'transfer intensity information'). . Examples of requested sensory information may include sight, hearing, pressure, vibration, and heat sensation. For example, transfer intensity information may include high, medium, and low.

The user input information may include at least one or a combination of request information, request sense information, and transfer strength information. The requested sensory information and the requested information may be used by the sensory path determining module 120 to determine a sensory path to deliver visual information (target information) meeting the user's request to the user. In other words, the requested sensory information and the requested information may be used by the sensory path determining module 120 to determine which sensory signal to transmit target information corresponding to the requested information to the user. The request sense information and transmission strength information included in the user input information are information for personalization of the user. This is because sensory signals that are easy to recognize may be different for each user, and sensitivity for each sensory signal may also be different.

The input module 110 may receive user input information in various ways. For example, the input module 110 may receive user input information through a button or text input. The input module 110 may separate and extract request information, request sense information, and transfer strength information from user input information. For example, the input module 110 may obtain request information, request sense information, or transfer strength information by parsing user input information.

The image information received by the input module 110 may include at least one of still images and continuous images (moving images). Also, the image information may be an image (ie, a color image, a depth image, or a thermal image) obtained through an RGB camera, a 3D depth camera, or a thermal image camera. The input module 110 transmits the image information to the image information division module 130 . The input module 110 transmits user input information to the sensory path determination module 120, the target information extraction module 140, and the alternative signal generation module 150. The input module 110 may transmit request information, requested sensory information, and transmission strength information extracted from user input information to the sensory path determination module 120, the target information extraction module 140, and the alternative signal generation module 150. For example, the input module 110 transmits the requested sensory information and requested information to the sensory path determining module 120, transmits the requested information to the target information extraction module 140, and transmits the requested information to the alternative signal generating module 150. Transmission strength information may be transmitted.

The sensory path determining module 120 determines a sensory path to deliver target information based on user input information. The sensory path determining module 120 may determine a sensory path for each requested information. For example, the sensory path determining module 120 determines the sensory path for transmitting visual information through the goggles worn by the user and the sensory path according to a preset table with respect to the requested information 'location and distance of the mobile phone' and A plurality of sensory pathways composed of sensory pathways that transmit auditory signals through earphones may be determined.

The sensory path determination module 120 may determine the sensory path according to the requested sensory information included in the user input information, but may also determine the sensory path according to the requested sensory information included in the user input information. Also, the sensory path determining module 120 may determine a sensory path according to target information.

The sensory path determining module 120 may determine a sensory path for each target information, and the sensory path determining module 120 may use requested information or target information to match the target information and the sensory path. A method for the sensory path determination module 120 to determine the sensory path will be described later with reference to FIG. 2 . The sensory path determining module 120 may transmit information on the sensory path determined for each target information to the alternative signal generating module 150 .

The image information division module 130 divides or analyzes the image information to generate various types of visual information such as shape information, spatial information, and global information, and transmits the generated visual information to the target information extraction module 140. The image information division module 130 may generate time information corresponding to each frame of image information, or may generate time information corresponding to a plurality of frames of image information. For example, the image information segmentation module 130 may generate animal location information for each frame of image information, and may generate animal movement path information corresponding to 100 frames. The visual information generated by the image information segmentation module 130 may include at least one or a combination of shape information, spatial information, and global information.

The target information extraction module 140 may generate composite information based on the visual information generated by the image information division module 130 . In addition, the target information extraction module 140 may extract target information that meets the user's request information from among the visual information and composite information generated by the image information segmentation module 130 .

Hereinafter, in this specification, 'visual information' is used as a generic term for shape information, spatial information, global information, and composite information generated based on image information.

The target information extraction module 140 extracts time information that meets the requested information, that is, target information, from among time information (shape information, spatial information, global information, and composite information). That is, the target information extraction module 140 extracts target information from time information based on the requested information.

The target information may be single information or may be a plurality of pieces of information.

The target information extraction module 140 transmits the target information to the alternative signal generating module 150 . Also, the target information extraction module 140 may transmit the target information to the control and output module 160 .

The substitute signal generation module 150 converts target information based on the sensory path to generate substitute signals. The alternative signal generation module 150 converts the visual information extracted by the target information extraction module 140 into a sensory signal corresponding to the visual information, that is, an alternative signal. The alternative signal generation module 150 uses the sensory path determined by the sensory path determination module 120 when converting visual information corresponding to target information into an alternative signal. The substitute signal is a sensory signal that replaces target information, and is a signal generated by the substitute signal generation module 150 to convert the target information into a different form and deliver it to the user's sensory organ, including visual, auditory, tactile and It may be at least one sensory signal or a combination thereof among heat sensation signals. For example, if the target information is visual information such as the color of a ball, the replacement signal may be a thermal sensation signal having a predetermined temperature. A method of converting visual information into an alternative signal corresponding thereto by the alternative signal generation module 150 may follow a method of numerical conversion or mapping, but is not limited thereto. The alternative signal generating module 150 converts the target information according to the sensory path determined for each target information and generates an alternative signal.

The alternative signal generation module 150 may reflect transmission strength information among user input information in the process of generating an alternative signal. That is, the alternative signal generation module 150 may generate an alternative signal by converting the target information based on the transmission strength information and the sensory path. For example, when the alternative signal generation module 150 converts the target information to generate an auditory signal, if the transfer intensity information is 'high', a sound wave having a high frequency or amplitude may be generated.

The replacement signal generation module 150 transmits the generated replacement signal to the control and output module 160 .

The control and output module 160 controls the conversion of visual information and the transmission of alternative signals to the user. Control and output module 160 may be implemented in the form of a plurality of hardware. For example, the control and output module 160 may be implemented as a control device and an output device. In addition, the control and output module 160 generates control signals required to simultaneously transmit individual sensory signals to a plurality of paths. And, the control and output module 160 outputs the replacement signal through a single or multiple transmission paths.

The control and output module 160 controls overall operations related to setting the output order of alternative signals and synchronizing delivery time points. In addition, the control and output module 160 delivers alternative signals to the user. For example, the control and output module 160 transmits an auditory signal, a heat sensation signal, and a vibration signal to an earphone, a heater, and a vibration device worn by the user so that sound, heat, and vibration are output from the devices. and transmits an alternative signal to the user. The replacement signal may be a single signal, but may also be a plurality of signals as illustrated.

The control and output module 160 may deliver the target information received from the target information extraction module 140 to the user as it is along with the replacement signal. For example, the target information extraction module 140 extracts the target information (animal outline, distance between the user and the animal) extracted by the target information extraction module 140 from visual information generated from an image captured by an infrared camera at night. It is transmitted to the alternative signal generation module 150 and the control and output module 160. The alternative signal generation module 150 converts the target information to generate an auditory alternative signal and transmits it to the control and output module 160 . The control and output module 160 may display target information on goggles worn by the user and simultaneously output an auditory alternative signal (sound wave) through earphones worn by the user.

2 is a block diagram showing a detailed configuration of an apparatus 100 for sensory substitution and multi-path delivery of visual information according to an embodiment of the present invention.

The input module 110 receives user input information and image information. The input module 110 includes a user input unit 111 , a user input information processing unit 112 and an image input unit 113 .

The user input unit 111 receives user input information from the outside. The user input information may include at least one or a combination of request information, request sense information, and transfer strength information. Specifically, the user input information includes information on the type of object the user is interested in or information indicating the type of visual information the user wants to receive from the image information (request information), information on the senses to receive the requested information ( Requested sensory information) and information expressing the transmission intensity of sensation (transfer intensity information) may be included. For example, the request information may be location information of a specific type of object included in image information. As a specific example of user input information when the present invention is applied to virtual reality, the requested information may be location and distance information of other users on the metaverse, and the requested sensory information may be vibration and temperature (feeling of heat). . The transfer strength information may be set to high among high/medium/low. The apparatus 100 for sensor substitution and multi-path delivery of visual information can maximize the user's experience by mobilizing other senses in addition to sight. For example, when the other user approaches on the metaverse, the user's senses may be amplified due to increased vibration and heat sensation signals.

The user input unit 111 may use voice or touch through a screen to receive requested information. For example, a user designates desired information among backgrounds, people, animals, or products displayed in an image through voice or screen touch. For example, when a user touches a background part of an image on the screen, the user input unit 111 may recognize that 'background', which is request information, among user input information is input. When a user touches a person part included in an image on the screen, the user input unit 111 may recognize that 'person', which is requested information, among user input information is input. The requested information may be a single piece of information or a plurality of pieces of information. On the other hand, the user input unit 111 may receive user input information in a manner in which a preset value is applied through a predetermined button or lever in a special working environment, such as a military operation requiring a confidential environment or a noisy environment. . For example, when a specific button ('person' button, 'position' button, 'distance' button, or 'tactile sense' button) is pressed, the user input unit 111 transmits location information and distance information of a person appearing in the image to the user. It can be regarded as requested information, and 'touch' can be regarded as requested sensory information input by the user. Also, the user input unit 111 may receive user input information in the form of text. For example, the user input unit 111 may receive user input information including request information and request sensory information in the form of text, such as 'Give me the distance to the animal through a tactile sense'.

The user input information processor 112 separates and extracts request information, request sense information, and transfer strength information from user input information. For example, the user input information processing unit 112 may obtain request information, request sense information, or transfer strength information by parsing user input information in text form. The user input information processing unit 112 may transmit request information, requested sensory information, and transmission strength information extracted from user input information to the sensory path determination module 120, the target information extraction module 140, and the alternative signal generation module 150. there is. For example, the user input information processing unit 112 transmits the requested sensory information and requested information to the sensory path determination module 120, transmits the requested information to the target information extraction module 140, and substitute signal generation module 150. ) can transmit forwarding strength information.

The image input unit 113 collects image information to be transmitted to the user in the present invention. The image information may include at least one of still images and continuous images (moving images). The image input unit 113 may include an RGB camera, a 3D depth camera, a time-of-flight (ToF) image sensor, or an infrared camera to collect image information. In general, image information includes various visual information. In this specification, examples of visual information include shape, border, position, distance, color, texture, path, direction, presence or absence of light, emotion, and overall atmosphere for detailed description. listen and explain.

The sensory path determining module 120 determines a sensory path to deliver target information based on user input information. The sensory path determining module 120 may determine a sensory path for each requested information. The sensory path determination module 120 may determine the sensory path according to the requested sensory information included in the user input information, but may also determine the sensory path according to the requested sensory information included in the user input information. Also, the sensory path determining module 120 may determine a sensory path according to target information. The sensory path determining module 120 may determine a sensory path for each target information, and the sensory path determining module 120 may use requested information or target information to match the target information and the sensory path. The sensory path determining module 120 determines a sensory path through which target information is transmitted to the user based on user input information. Specifically, the sensory path determination module 120 determines whether to transmit visual information corresponding to target information as a single signal or as a plurality of signals based on user input information. The sensory path determining module 120 may determine a sensory path that converts each of a plurality of target information into a corresponding sensory signal and transmits the result to the user, and converts the same target information into a plurality of sensory signals and transmits the sensory path to the user. can also decide For example, when the positional information of a specific object shown in the image information is target information, the sensory path determination module 120 may determine the sensory path so that the left and right or vertical position is transmitted through sound, and tactile, vibration, and thermal signals. It is also possible to determine a sensory path through which location information is transmitted. The sensory path determining module 120 determines a sensory path based on user input information, and may determine a sensory path based on requested sensory information included in the user input information. Also, the sensory path determination module 120 may determine the sensory path based on request information included in user input information. For example, when the requested sensory information is 'a sense of heat', the sensory path determination module 120 determines a corresponding sensory path (eg, a sensory path that transmits temperature through the left hand) according to settings or a user's selection. can As another example, when the requested information is 'vehicle distance', the sensory path determination module 120 selects a preset sensory path for 'distance' (eg, an auditory sensory path that transmits sound through the left ear and vibration to the left hand). tactile sensory pathways) can be determined. A user's choice or setting determines which information a sensory pathway is to be determined on the basis of.

The sensory path determining module 120 may transmit information on the sensory path determined for each target information to the alternative signal generating module 150 .

The image information division module 130 divides or analyzes the image information to generate various types of visual information such as shape information, spatial information, and global information, and transmits the generated visual information to the target information extraction module 140. The image information division module 130 may generate time information corresponding to each frame of image information, or may generate time information corresponding to a plurality of frames of image information. For example, the image information segmentation module 130 may generate animal location information for each frame of image information, and may generate animal movement path information corresponding to 100 frames. The image information division module 130 may extract various visual information included in a frame of an image or video by dividing the image information step by step. The visual information generated by the image information segmentation module 130 includes raw information such as coordinates of pixels constituting a frame and information acquired through image processing such as a channel of a frame. In addition, the visual information may also include meaningful analysis results obtained using inference techniques using algorithms such as deep learning-based object detection, region segmentation, and relationship information analysis. Therefore, the image information division module 130 first divides morphological information that can be extracted from an image or video frame with a simple operation, then divides spatial information that relatively requires user input or additional analysis work, and finally Segmentation and analysis can be performed in order to generate global or composite information. The image information division module 130 may include a shape information generator 131, a spatial information generator 132, and a global information generator 133. The shape information generator 131 generates image information Shape information is generated by division, the spatial information generation unit 132 divides the image information to generate spatial information, and the global information generation unit 133 analyzes the image information to generate global information or composite information. Table 1 shows examples of shape information, spatial information, global information, and composite information.

visual information example shape information Shape, outline (border), color, presence or absence of light, texture Spatial information location, distance, direction, route global information Color or shape extracted from the entire video area complex information Route, emotion, mood (generated based on location information)

The target information extraction module 140 may generate composite information based on the visual information generated by the image information division module 130 . In addition, the target information extraction module 140 may extract target information that meets the user's request information from among the visual information and composite information generated by the image information segmentation module 130 . For example, when the requested information is 'animal outline', the target information extraction module 140 may extract the visual information showing the outline of a dog or cat from among the visual information as the target information.

As described above, the target information extraction module 140 receives the visual information (eg, shape information, spatial information, and global information) generated by the image information segmentation module 130 based on the image information. It can be used as a reference to selectively receive necessary visual information.

The target information extraction module 140 transmits the target information to the alternative signal generating module 150 . The target information extraction module 140 extracts visual information, that is, target information, that meets the requested information from among visual information such as shape information, spatial information, global information, and complex information. That is, the target information extraction module 140 selects among the visual information (shape information, spatial information, and global information) generated by the image information segmentation module 130 and the visual information (composite information) generated by the target information extraction module 140. Extract target information that matches the requested information. The target information extraction module 140 includes a composite information generator 141 and a target information extractor 142 .

The composite information generation unit 141 generates composite information based on the visual information generated by the image information segmentation module 130 . For example, the composite information generating unit 141 may obtain mood information based on shape information such as the shape and frame (outline) of an object and global information such as color extracted from the entire image area. The composite information generation unit 141 may generate composite information based on a plurality of shape information or a plurality of spatial information, and may generate composite information such as emotion or mood information based on overall image information (global information). In the process of creating such complex information, artificial intelligence techniques can be used.

The target information extraction unit 142 extracts target information matching the requested information from visual information (shape information, spatial information, global information, and composite information).

The substitute signal generation module 150 converts target information based on the sensory path to generate substitute signals. The substitute signal generation module 150 converts the target information extracted by the target information extraction module 140 into a sensory signal that replaces the target information, that is, a substitute signal. The alternative signal generation module 150 converts the target information extracted by the target information extraction module 140 into an alternative signal corresponding to the target information. The alternative signal generation module 150 identifies an alternative signal (sound wave) that meets the target information based on the sensory path for each target information (eg, the distance between the vehicle and the user is transmitted to the auditory sensory path through the left ear). It is judged based on reference data such as a setting table, and converts target information to generate an alternative signal. That is, the alternative signal generation module 150 converts the target information according to the sensory path determined for each target information and generates an alternative signal corresponding thereto.

On the other hand, the alternative signal generation module 150 uses the time information (information conversion time (T _c ), transmission time (T _t ), delay time (T _d ), recognition time (T _p )) calculated by the control and output module 160 )) can be reflected in the process of generating alternative signals. For example, when it is determined that the time necessary for delivery is insufficient according to the time analysis result of the control and output module 160, the alternative signal generation module 150 simplifies the sensory signal conversion process or the conversion result to convert information. Time (T _c ) or transfer time (T _t ) can be shortened. If the information conversion time (T _c ) or transmission time (T _t ) cannot be shortened in the alternative signal generation module 150, the transmission signal synchronization unit 162 of the control and output module 160 determines the time required for transmission. Synchronization time (time point) is extended to secure the delivery time as much as possible.

In this embodiment, the alternative signal generating module 150 includes a first information conversion unit 151 that converts target information into an auditory signal (sound) and a second information conversion unit 152 that converts target information into a tactile signal (vibration). ) and a third information conversion unit 153 that converts target information into a thermal sensation signal (temperature). However, components of the alternative signal generation module 150 may be added according to the addition of a convertible sense (eg, olfaction) or segmentation of the senses (eg, segmentation of a tactile signal into vibration, touch, and pressure). .

The first information conversion unit 151 converts the target information into an auditory signal (sound wave). The first information conversion unit 151 mainly converts form information or space information such as shape, location, presence or absence of light, color and direction into sound waves. For example, the first information conversion unit 151 may generate sound waves so that a difference in the vertical position of an object is divided into frequencies by defining a frequency for each pixel or group of pixels at a vertical position of the image in advance. That is, the first information conversion unit 151 may map the vertical position of the image to the frequency of the sound wave. In addition, the first information conversion unit 151 may map the horizontal position of the image to reproduction time information of sound or a stereo signal, and generate sound waves by mapping the brightness value of a pixel corresponding to the shape or edge of an object to an amplitude. can In addition, when spatial information is expressed as sound, the first information conversion unit 151 can generate a sound wave whose reproduction frequency and volume increase as the object distance is closer by combining frequency and amplitude in a complex manner. , sound waves with reduced reproduction frequency and reduced size can be generated as the distance between the objects increases. If the visual information is color information (e.g., object color, color extracted from the entire image area, emotion or mood based on global information, etc.), the distribution of color can be analyzed using deep learning technology, so the first information conversion The unit 151 may convert the visual information in such a way that the analyzed information is expressed as light-pitched sound waves in the case of a bright feeling or atmosphere, and expressed as slow or low-pitched sound waves in the case of a dark or gloomy atmosphere or emotion.

The second information conversion unit 152 converts the target information into a tactile signal. The second information conversion unit 152 mainly converts target information such as distance, texture, location, path or direction into tactile signals. In the apparatus 100 for sensory substitution and multipath transmission of visual information, a device to which the tactile signal is finally transmitted may be a vibration device attached to a wrist or worn on the body. When the vibration generating elements are located in the top, bottom, left, and right positions of the vibration device attached to the wrist, the second information conversion unit 152 generates tactile signals for a plurality of channels so that only elements corresponding to a specific path or direction generate vibration can do. The control and output module 160 transmits a control signal to the vibrating device based on the tactile signal generated in this way so that elements corresponding to the path or direction generate vibration, thereby conveying the direction or path of the object to the user.

Meanwhile, when the target information is distance information, the second information conversion unit 152 may convert the distance information into a tactile signal by changing a period or intensity of vibration according to the distance.

The third information conversion unit 153 converts the target information into a sensation signal. The third information conversion unit 153 may convert target information mainly about color, emotion or atmosphere into a heat sensation signal. In the device 100 for sensory substitution and multipath transmission of visual information, a device to which the heat sensation signal is finally transmitted may be a heat transfer device (heater). As a specific example, the third information conversion unit 153 converts target information of blue or dark atmosphere into a thermal sensation signal having a low temperature based on the temperature range that can be transmitted by the heat transfer device, and target information of red or hot atmosphere is set It can be converted into a thermal sensation signal having a high temperature in the temperature range.

Each of the information conversion units 151 , 152 , and 153 of the alternative signal generating module 150 may generate an alternative signal by converting the target information based on the transmitted intensity information and the sensory path. For example, when the first information conversion unit 151 converts target information to generate an auditory signal, the first information conversion unit 151 determines that the transmission strength information included in the user input information is 'high'. It is possible to generate a sound wave with increased frequency or amplitude by reflecting the

The alternative signal generation module 150 may individually convert a plurality of target information into different alternative signals or convert the same visual information into a plurality of alternative signals in order to efficiently deliver the target information to the user. In this case, the alternative signal generating module 150 may derive a final conversion result by combining the conversion results of the plurality of information conversion units 151, 152, and 153. For example, even in the case of converting single location information, the alternative signal generation module 150 converts the location information into sound waves indicating left and right through the first information conversion unit 151, and the second information conversion unit 152 ), the location information may be converted into a vibration signal to be transmitted to a leftward element of the vibration device worn on the wrist of the left hand. In this case, the control and output module 160 simultaneously outputs sound waves and vibration signals related to the location information so that the user can recognize the location information more efficiently.

The replacement signal generation module 150 transmits the generated replacement signal to the control and output module 160 .

The control and output module 160 controls overall operations related to setting the output order of alternative signals and synchronizing delivery time points. The control and output module 160 delivers the replacement signal generated by the replacement signal generation module 150 to the user. The control and output module 160 controls and manages the flow of transmission signals necessary for overall operation for multipath transmission of target information. Also, the control and output module 160 may output the alternate signal to multipath.

Meanwhile, the control and output module 160 may be physically separated as a separate device. For example, the control and output module 160 may be implemented by including a control device and an output device.

The control and output module 160 includes a transfer path setting unit 164 and a multipath output unit 165. The control and output module 160 may further include a time analysis unit 161 and a transmission signal synchronization unit 162 to synchronize the alternative signal, and further include an alternative signal correction unit 163 to correct the alternative signal. can do. Synchronization and correction of alternative signals of the control and output module 160 are necessary to effectively deliver target information to the user.

The control and output module 160 synchronizes a plurality of alternative signals for each target information based on the sensory path for each target information determined in the sensory path determination module 120 and delivers them to the user. In addition, the control and output module 160 corrects the characteristics (eg, intensity) or type of the substitute signal according to the image information and user behavior (user reaction). The control and output module 160 may recognize user behavior (user reaction) through analysis of image information.

First, the functions of the time analyzer 161 and the transmission signal synchronizer 162 in charge of synchronizing alternative signals will be described. The time analysis unit 161 provides the transmission signal synchronization unit 162 with time information necessary for the transmission signal synchronization unit 162 to determine the synchronization time point. The time analysis unit 161 determines information conversion time (T _c ) for each sense, ② transmission time (T _t ) for each sense, ③ delay time (T _d ) of the transmission device, and ④ information conversion time (T c ) for each sense, based on the sensory path for each target information. The recognition time (T _p ) is calculated, and the calculated time information (T _c , T _t , T _d , T _p ) is transmitted to the transfer signal synchronization unit 162 . Table 2 is a description of each time calculated by the time analyzer 161.

division sign explanation information conversion time
(Time for conversion) T _c Time required to convert target information into alternative signals for each sense delivery time
(Time for transmission) T _t Time required for transmission or reproduction per sense delay time
(Time for delay) T _d Delay time of the device that transmits to each sensory organ perception time
(Time for perception) T _p Time to recognize the transmitted signal in the target sensory receptor

The time analysis unit 161 converts information for each sense based on the time required when each conversion unit 151, 152, 153 of the alternative signal generation module 150 converts target information into an alternative signal (time required to generate an alternative signal). Time (T _c ) can be calculated. And, the transmission time for each sense (T _t ) means the time required to deliver information for each target sensory organ. The transit time (T _t ) for each sense may be different for each sense. For example, different lengths of time may be required for each sense, such as hearing for 0.5 seconds, touch for 0.8 seconds, and heat for 1 second. And, the delay time (T _d ) of the delivery device can be regarded as the transmission time of the delivery device, and varies depending on the characteristics of the equipment. Accordingly, the time analyzer 161 may calculate the delay time T _d by reflecting the information disclosed by the device manufacturer. In addition, the time analyzer 161 may calculate the recognition time Tp for each sensory organ based on a generally known value (result of previous research). However, since there may be a difference in perception time for each user and some people do not feel a specific sense, the substitute signal correction unit 163 corrects the substitute signal according to user behavior or user reaction.

The transmission signal synchronization unit 162 synchronizes a plurality of alternative signals to be transmitted to the user based on the frame of each time information extracted by the target information extraction module 140 and the time information calculated by the time analysis unit 161. Determine the point in time and perform synchronization work. In order for the transmission signal synchronizer 162 to synchronize the plurality of alternative signals, the time required to convert target information into alternative signals for each target sense (T _c : Time for conversion) and the time required for transmission or reproduction for each sense (T _t : Time for transmission), delay time (T _d : Time for delay) of the device that transmits to each sensory organ, and time for perceiving the transmitted signal in the target sensory receptor (T _p : Time for perception). The reason why such temporal consideration is necessary for synchronization is that when various visual information of a video or image is transmitted as a plurality of alternative signals, there is a limit to the strength and amount of information that can be transmitted for each sensory organ, and even the same signal is recognized by the user because there is a difference in For example, a replacement signal in the form of a sound wave requires a reproduction time due to its characteristics. On the other hand, the replacement signal in the form of a vibration signal may be shorter or longer than the reproduction time of the sound wave. Therefore, when the alternative signals obtained by converting different target information derived from the same image frame are transmitted through a plurality of sensory pathways, the control and output module 160 synchronizes the alternative signals for the same frame and delivers them to the user at the same time. shall.

As another example, even when the same visual information is transmitted to a user through a plurality of sensory pathways, since the amount of information that can be transmitted is different for each sensory organ, the transmitted information must be synchronized. A synchronization point (Ts: Sync. Time) for transmitting the converted alternative signal through a single path or a plurality of paths may vary according to request information or target information. The transmission signal synchronizer 162 converts the information for the target information extracted by the target information extraction module 140 (T _c ), the sum of the transmission time (playback time) and the delay time (T _t +T _d ), and sense In consideration of the recognition time (T _p ) for each institution, the longest required time can be determined as the synchronization time point as shown in Equation 1.

Next, the replacement signal correction unit 163 for correcting the replacement signal will be described as follows. When the replacement signal is transmitted to the user without correction despite the difference between the attachment position of the camera inputting the image information and the position of the device to which the converted alternative signal is transmitted, it is difficult for the user to accurately recognize the location or distance of the object. Therefore, the substitute signal correction unit 163 performs a function of correcting the substitute signal by resolving the inconsistency. For example, while a camera for inputting image information is generally attached to glasses, a necklace, or a chest using a fixing device, location information or depth information is transmitted to a vibration device attached to a sound or hand. In this case, since the position/distance viewed from the head or chest is relatively different from the position of the ear or hand that receives the alternative signal, the position information or distance information is not transmitted to the sound or tactile device. Therefore, it is essential to correct the difference between the location where the image information is input and the location where the alternative signal is transmitted so that the user can efficiently recognize it, and the alternative signal correction unit 163 performs this function.

In addition, the substitute signal correction unit 163 corrects the characteristics (eg, intensity) or type of the substitute signal according to image information, user behavior, or user response. The alternative signal correction unit 163 may recognize a user's behavior (or user's reaction) by analyzing image information. The substitute signal correction unit 163 may determine the user's sensitivity based on the image information and the user's behavior, and corrects the characteristics or types of the substitute signal based on the user's sensitivity. Specifically, when visual information is converted into auditory, tactile, or thermal sensations and transmitted, the sensitivity of feeling a specific sense may vary depending on the user. Therefore, when the alternative signal correction unit 163 determines that the user's sensitivity to the alternative signal is at a low level, it increases the strength of the alternative signal or converts the sensory path for the target information to another sensory path (eg, hearing). into a sense of heat). An example of increasing the strength of an alternative signal may be increasing the loudness of an auditory signal.

The control and output module 160 sets a single path or a plurality of paths by referring to the synchronization result of the transfer signal synchronization unit 162, and outputs an alternative signal to a device attached to the user through the set transfer path. The transmission path setting unit 164 receives the alternative signal converted by the alternative signal generation module 150, and based on the sensory path for each target information, the information conversion time for each path (T _c ) and the transfer time for each sense (T _t ), the delay time (T _d ) of the transmission device for each sense, and the perception time (T _p ) for each sense are taken into account to set the transmission path of the alternative signal. The control and output module 160 has four times calculated by the time analysis unit 161 (① information conversion time for each sense (T _c ), ② transmission time for each sense (T _t ), ③ delay time of the transmission device (T _d ) and ④ individual alternative signals are delivered to the user according to the synchronization time point derived by the transmission signal synchronizing unit 162 based on the recognition time (T _p ) for each sensory organ, and the user's action or user response to the individual senses is determined. Taking into account the transmission path and intensity (strength) of the alternative signal can be corrected. Visual information has the largest amount of information among the five senses, so information is reduced when converted to other senses. Accordingly, there is a problem in that an identical substitute signal can be generated through conversion of non-identical visual information, but the above problem can be solved through personalization correction based on user behavior. The multipath output unit 165 refers to the synchronization signal generated by the transmission signal synchronization unit 162 for individual replacement signals included in a predetermined temporal range through a transmission path set by the transmission path setting unit 164. deliver at the same time

3 is a diagram exemplarily illustrating the concept of sensory substitution and multipath transmission of visual information according to the present invention. FIG. 3 illustrates a situation in which target information is converted into an alternative signal and transmitted to a user through a multipath when performing a military operation among various application examples of the present invention.

The input module 110 receives user input information and image information (still image or video). In this case, the requested information included in the user input information is information suitable for the user's purpose of performing reconnaissance or military operations. It may be the position of the person, the distance of the person, the position of the vehicle, the distance of the vehicle, and the like. The input module 110 parses user input information to extract request information, request sense information, and transfer strength information. For example, the requested information may be 'vehicle distance', the requested sensory information may be 'feeling of heat', and the transfer intensity information may be 'high'.

The sensory path determination module 120 determines a sensory path based on user input information. The sensory path determining module 120 may determine a sensory path based on the requested sensory information, or may determine a sensory path based on the requested sensory information. For example, when the requested sensory information is 'a sense of heat', the sensory path determining module 120 may determine a corresponding sensory path (eg, a sensory path that transmits temperature through the left hand). As another example, when the requested information is 'vehicle distance', the sensory path determination module 120 selects a preset sensory path for 'distance' (eg, an auditory sensory path that transmits sound through the left ear and vibration to the left hand). tactile sensory pathways) can be determined. It depends on what information the sensory pathway is determined on the basis of.

The image information division module 130 divides the image information to generate various visual information. For example, the image information segmentation module 130 divides the image information to form information such as the shape of the object, the border of the object, the outline of the background, the color of the object, the presence or absence of light, the texture of the object, and the object. It generates spatial information such as the location of the user, the distance between the user and the object, the distance between the object and other objects, the movement direction of the object, and the global information such as the dominant color or shape of the entire image.

The target information extraction module 140 extracts target information, which is time information corresponding to the requested information, based on the time information generated by the image information segmentation module 130. For example, the target information extraction module 140 extracts vehicle location information and distance information between the user and the vehicle as target information corresponding to the requested information 'vehicle location' and 'vehicle distance'.

The alternative signal generation module 150 converts the target information by referring to the transmission strength information included in the user input information to generate an alternative signal. For example, the alternative signal generation module 150 converts the distance information of the vehicle to generate an audible signal and a vibration signal with a 'high' transmission intensity.

The control and output module 160 transmits alternative signals through multiple paths so that the user can efficiently perceive necessary information. For example, the control and output module 160 simultaneously outputs an auditory signal and a vibration signal through a Suwa vibration device worn by the user so that the user can recognize distance information of the vehicle.

Unlike conventional sensory substitution techniques, the sensory substitution and multipath transmission technique according to the present invention enables synchronized delivery of complex visual information, thereby improving user perception. As an example, as shown in FIG. 3 , information about a red ball near the left of a soldier holding a gun is complex information that could not be conveyed by existing sensory substitution technologies or devices. According to the sensory substitution and multipath transmission technique according to the present invention, the shape information (the round shape of the ball) is converted into sound, and the position information (the left side of the soldier) is transmitted to the left vibration element of the vibration device attached to the wrist. At the same time, the distance information (the distance between the soldier and the object in front of the soldier) is transmitted as the vibration period or intensity of the vibration device, and the color information (red) transmits the heat signal (temperature information) set to a high level. It can be transferred simultaneously through a heat transfer device.

4 is a block diagram showing the configuration of an apparatus 100' for sensory substitution and multi-path transmission of visual information according to another embodiment of the present invention.

An apparatus 100' for sensory substitution and multipath delivery of visual information according to another embodiment of the present invention includes an input module 110, a sensory path determination module 120, a target information extraction module 140', and an alternative signal generation module. 150 and a control and output module 160.

The embodiment of FIG. 4 is different from the embodiment of FIG. 1 in device configuration. The embodiment of FIG. 4 integrates the functions of the image information division module 130 and the target information extraction module 140 of the embodiment of FIG. 1 and implemented as one target information extraction module 140'. The embodiment of FIG. 4 can be applied to a case in which parallel processing of dividing image information and extracting target information is not required because the work load is not large compared to the embodiment of FIG. 1 .

The target information extraction module 140' divides the image information received from the input module 110 to generate various visual information such as shape information, spatial information, and global information, and generates composite information based on the visual information. , Target information corresponding to the requested information transmitted from the sensory path determination module 120 is extracted from the generated shape information, spatial information, global information, and composite information, and the extracted target information is transmitted to the alternative signal generation module 150. do. Also, the target information extraction module 140' may transmit the target information to the control and output module 160. The target information extraction module 140' includes the shape information generator 131, spatial information generator 132, global information generator 133, complex information generator 141, and target information to perform the above functions. It is composed of an extraction unit 142, and the functions of each component of the target information extraction module 140' are the same as those described above with reference to FIG. 2.

Meanwhile, functions of the input module 110, the sensory path determination module 120, the alternative signal generation module 150, and the control and output module 160 included in the device 100' for sensory substitution and multipath transmission of visual information Since is the same as the above in FIGS. 1 to 3, the description thereof is omitted.

5 ( FIGS. 5A and 5B ) is a flowchart for explaining a method for sensory substitution and multipath transfer of visual information according to an embodiment of the present invention.

The sensory substitution and multipath transfer method of visual information according to the present invention includes steps S210 to S280.

Step S210 is a request information extraction step. The sensory permutation and multipath transmission of visual information (100, 100′) receives user input information, and the user input information includes request information (information indicating the type of visual information the user wants to receive), requested sensory information ( Information about senses that are mediators of request information delivery), transmission strength information, and the like may be included. The apparatuses 100 and 100' for sensory substitution and multipath delivery of visual information may extract requested information from user input information.

Step S220 is an image information division step (visual information generation step). Sensory substitution of visual information and multi-path delivery devices 100 and 100' collect image information from the outside. Sensory substitution of visual information and multi-path delivery devices (100, 100') may use an RGB camera, a 3D depth camera, a Time-of-Flight (ToF) image sensor, or an infrared camera to collect image information. there is. The image information may include at least one of still images and continuous images (moving images). Sensory substitution of visual information and multi-path delivery devices (100, 100') generate visual information such as shape information, spatial information, and global information by segmenting image information, and create composite information based on the generated visual information. . As described above, in the present invention, information generated based on image information, such as shape information, spatial information, global information, and composite information, is collectively referred to as 'visual information'.

Step S230 is a step of extracting target information. Sensory substitution of visual information and multi-path delivery device (100, 100') determines whether each visual information generated in step S220 is necessary through analysis of the requested information, and extracts visual information that meets the requested information, that is, target information do. Since necessary information may appear in a complex manner according to the requested information, the sensory substitution and multi-path transmission apparatus 100, 100' of visual information first extracts necessary form information according to the requested information, and then extracts the information corresponding to the requested information. Spatial information is extracted, and the correlation between requested information and global/composite information is analyzed to extract global or composite information if necessary. In addition, the apparatuses 100 and 100' for sensory substitution and multi-path transmission of visual information set the extracted visual information (shape information, spatial information, global information, and composite information) as target information.

Step S230 will be described in detail below with reference to FIG. 5B. Step S230 includes steps S231 to S237.

Step S231 is a request information analysis step. The apparatuses 100 and 100' for sensory substitution and multipath delivery of visual information analyze the requested information and materialize necessary information. For example, when the present invention is applied to a survival game in virtual reality, when the requested information is determined to be 'human' based on a user input (screen or button touch), sensory substitution of visual information and multi-path transmission device 100, 100') is visual information that is related to the requested information, and can derive a shape, an outline (border), a location, a distance, a route, and the like. Based on these results, based on Table 1, the sensory substitution of visual information and the multi-path delivery device 100, 100' provide shape information (shape, Outline), spatial information (location, distance), and complex information (route) may be determined to be necessary.

Step S232 is a step of determining whether shape information is required. The apparatuses 100 and 100' for sensory substitution and multipath transmission of visual information perform step S233 if shape information is required as a result of analyzing the requested information, and step S234 otherwise.

Step S233 is a step of extracting shape information. The apparatuses 100 and 100' for sensory substitution and multipath transmission of visual information extract form information corresponding to the requested information from among the visual information generated in step S220, and then proceed to step S234.

Step S234 is a step of determining whether spatial information is needed. The apparatuses 100 and 100' for sensory substitution and multipath delivery of visual information perform step S235 if spatial information is required as a result of analyzing the target information, and step S236 otherwise.

Step S235 is a spatial information extraction step. The apparatuses 100 and 100' for sensory substitution and multipath transmission of visual information extract spatial information corresponding to the requested information from among the visual information generated in step S220, and then proceed to step S236.

Step S236 is a step of determining whether global information/composite information is necessary. The apparatuses 100 and 100' for sensory substitution and multipath delivery of visual information proceed to step S237 if global information or complex information is required as a result of analyzing the target information, and otherwise proceed to step S240.

Step S237 is a global information/composite information extraction step. The apparatuses 100 and 100' for sensory substitution and multipath transmission of visual information extract global information or composite information corresponding to the requested information from among the visual information generated in step S220, and then proceed to step S240.

Returning to FIG. 5A again, steps below step S240 will be described as follows.

Step S240 is a step of determining a sensory pathway. The sensory permutation and multipath transmission of visual information (100, 100') may determine a sensory pathway according to requested sensory information included in user input information, but may also determine a sensory pathway according to requested information included in user input information. there is. In addition, the apparatus 100, 100' for sensory substitution and multipath transmission of visual information may determine a sensory pathway according to target information.

The sensory substitution and multipath transmission devices 100 and 100' of visual information may determine sensory pathways for each target information. Request information or target information may be used for matching.

Step S250 is a step of generating an alternative signal by converting target information. The apparatuses 100 and 100' for sensory substitution and multipath transmission of visual information convert the target information extracted in step S230 into an alternative signal corresponding thereto. The alternative signal is a signal for transmitting target information to the user's sensory organs, and may be a visual signal, an auditory signal, a tactile signal, or a heat sensation signal. The sensory substitution of visual information and the multi-path transmission apparatus 100, 100' may convert single target information into several alternative signals or convert a plurality of target information into individual alternative signals corresponding to each target information. . For example, one piece of target information may be converted into a plurality of alternative signals such as a sound wave, a tactile signal, and a heat sensation signal, and three pieces of target information may be converted into sound waves, a tactile signal, and a heat sensation signal, respectively.

Step S260 is an alternative signal synchronization step. The apparatuses 100 and 100' for sensory substitution and multipath delivery of visual information synchronize the plurality of substitute signals generated in step S250 based on the target information and the sensory pathways for each target information. Sensory substitution and multi-path delivery devices (100, 100') of visual information are based on sensory pathways for each target information, ① information conversion time for each sense (T _c ), ② transfer time for each sense (T _t ), ③ The delay time (T _d ) of the transmission device and the recognition time (T _p ) for each sensory organ are calculated, and synchronization is performed for a plurality of alternative signals based on the calculated time information from ① to ④. In determining the synchronization time point, Equation 1 described above may be used.

Step S270 is an alternative signal correction step. Sensory substitution of visual information and multi-path delivery devices (100, 100') correct characteristics (eg, intensity) or types of alternative signals according to image information and user behavior. The apparatuses 100 and 100' for sensory substitution and multi-path transmission of visual information may recognize the user's behavior through analysis of the image information. In addition, the sensory substitution and multi-path transmission devices 100 and 100' of visual information may correct an alternative signal based on the location of the image information input device and the location of the alternative signal transmission device. For example, when a camera into which image information is input is located on a necklace, target information is location information and distance information of an object, and the location information is transmitted to the user through a vibration device attached to the hand, sensory substitution of visual information and the multi-path transmission devices 100 and 100' correct a tactile signal (vibration signal) transmitted to the vibration device based on a positional difference between the position of the camera and the position of the vibration device. In this case, the position of the camera and the position of the vibration device may be expressed as 3D coordinates, and a difference between the two positions may be expressed as a 3D vector.

Step S280 is an alternative signal output step. The sensory substitution and multi-path transmission device for visual information (100, 100') sets a single or multiple paths for each alternative signal by referring to the synchronization time of the alternative signal, and attaches the alternative signal to the user through the set transmission path. output to the device. The apparatuses 100 and 100' for sensory substitution and multi-path delivery of visual information may deliver single information to the user through a plurality of delivery routes or deliver a plurality of information to the user through different routes separated from each other. The apparatus 100 for transmitting sensory substitution and multipath of visual information sequentially outputs substitution signals according to synchronization signals.

Meanwhile, in the description with reference to FIG. 5 ( FIGS. 5A and 5B ), each step may be further divided into additional steps or combined into fewer steps, depending on an embodiment of the present invention. Also, some steps may be omitted if necessary, and the order of steps may be changed. In addition, even if other omitted contents, the contents of FIGS. 1 to 4 may be applied to the contents of FIG. 5 . In addition, the contents of FIG. 5 may be applied to the contents of FIGS. 1 to 4 .

The sensory substitution and multi-path transmission method of the above-described visual information has been described with reference to the flow chart presented in the drawings. For simplicity, the method is shown and described as a series of blocks, but the invention is not limited to the order of the blocks, and some blocks may occur in a different order or concurrently with other blocks than shown and described herein. and various other branches, flow paths, and sequences of blocks that achieve the same or similar results may be implemented. Also, not all blocks shown may be required for implementation of the methods described herein.

For reference, the terms '~unit' or '~module' used in the embodiments of the present invention may be software or hardware components such as FPGAs or ASICs, and perform certain roles, but are limited to software or hardware. is not Each operation of the module may be performed through one hardware or may be performed through different hardware. Operations of a plurality of modules may be performed through one hardware, and operations of one module may be performed through a plurality of hardware.

Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will variously modify and change the present invention within the scope not departing from the spirit and scope of the present invention described in the claims below. You will understand that it can be done.

100, 100': sensory substitution of visual information and multi-path transmission device
110: input module
111: user input unit
112: user input information processing unit
113: video input unit
120: sensory pathway determination module
130: image information segmentation module
131: form information generation unit
132: spatial information generator
133: global information generation unit
140, 140': target information extraction module
141: complex information generation unit
142: visual information extraction unit
150: alternative signal generation module
151: first information conversion unit
152: second information conversion unit
153: third information conversion unit
160: control and output module
161: time analysis unit
162: transmission signal synchronization unit
163: alternative signal correction unit
164: delivery route setting unit
165: multipath output unit

Claims (18)

A target information extraction module for extracting information corresponding to a type requested by a user from image information received from the outside based on user input information;
a sensory path determining module configured to determine at least one sensory path through which the extracted information is transmitted based on the user input information;
an alternative signal generation module for generating an alternative signal corresponding to each of the at least one sensory pathway by converting the extracted information; and
a control and output module for outputting the alternative signal through the user's sensory signal transmission device;
including,
The type is included in the user input information
A device for sensory substitution and multipath transmission of human visual information. The method of claim 1, wherein the at least one sensory pathway,
multiple sensory pathways,
The alternative signal generation module,
converting the extracted information to generate an alternative signal corresponding to each of the plurality of sensory pathways;
A device for sensory substitution and multipath transmission of human visual information. The method of claim 2, wherein the plurality of sensory pathways,
Including sensory pathways corresponding to at least two or more senses among sight, hearing, touch and heat
A device for sensory substitution and multipath transmission of human visual information. The method of claim 1, wherein the target information extraction module,
Analyzing the image information to generate overall visual information including at least one of shape information, spatial information, global information, and composite information;
Extracting information corresponding to the type from among the total visual information based on the user input information
A device for sensory substitution and multipath transmission of human visual information. The method of claim 2, wherein the control and output module,
Synchronizing alternative signals corresponding to each of the plurality of sensory pathways based on a frame of the image information, and outputting the synchronized alternative signals through the sensory signal transmission device.
A device for sensory substitution and multipath transmission of human visual information. The method of claim 1, wherein the control and output module,
Recognizing the user's behavior based on the image information, correcting characteristics of the alternative signal based on the user's behavior, and outputting the corrected alternative signal through the sensory signal transmission device.
A device for sensory substitution and multipath transmission of human visual information. The method of claim 1, wherein the control and output module,
Correcting characteristics of the substitute signal based on the position of the image information input device and the position of the sensory signal transmission device, and outputting the corrected substitute signal through the sensory signal transmission device.
A device for sensory substitution and multipath transmission of human visual information. The method of claim 1, wherein the user input information,
contains information about the strength of transmission;
The control and output module,
Correcting the strength of the alternative signal according to the transmission strength, and outputting the corrected replacement signal through the sensory signal transmission device.
A device for sensory substitution and multipath transmission of human visual information. The method of claim 1, wherein the control and output module,
Calculate the time required to convert the extracted information into the alternative signal in the alternative signal generating module;
The alternative signal generation module,
Determining whether to simplify a process of converting the extracted information into the alternative signal based on the time
A device for sensory substitution and multipath transmission of human visual information. Extracting information corresponding to a type requested by a user from image information received from the outside based on user input information;
determining at least one sensory pathway through which the extracted information is transmitted based on the user input information;
generating an alternative signal corresponding to each of the at least one sensory pathway by converting the extracted information; and
outputting the alternative signal through the user's sensory signal transmission device;
Sensory substitution and multi-path transmission method of visual information including. The method of claim 10, wherein the at least one sensory pathway,
multiple sensory pathways,
Generating the replacement signal,
Generating the extracted information into an alternative signal corresponding to each of the plurality of sensory pathways
Sensory substitution and multipath transmission method of human visual information. The method of claim 11, wherein the plurality of sensory pathways,
Including sensory pathways corresponding to at least two or more senses among sight, hearing, touch and heat
Sensory substitution and multipath transmission method of human visual information. The method of claim 10, after generating the replacement signal,
An alternative signal synchronization step of synchronizing the alternative signal based on the frame of the image information; a sensory substitution and multi-path transmission method of visual information. The method of claim 10, after generating the replacement signal,
Further comprising a substitute signal correction step of correcting the characteristics of the substitute signal based on the position of the image information input device and the position of the sense signal transmission device,
The outputting step is
Outputting the corrected alternative signal through the sensory signal transmission device
Sensory substitution and multipath transmission method of human visual information. The method of claim 10, wherein the step of extracting information corresponding to the type requested by the user comprises:
Analyzing the image information to generate overall visual information including at least one of shape information, spatial information, global information, and composite information;
Extracting information corresponding to the type from among the total visual information based on the user input information
Sensory substitution and multipath transmission method of human visual information. The method of claim 11, wherein the outputting step,
Synchronizing alternative signals corresponding to each of the plurality of sensory pathways based on a frame of the image information, and outputting the synchronized alternative signals through the sensory signal transmission device.
Sensory substitution and multipath transmission method of human visual information. The method of claim 10, wherein the user input information,
contains information about the strength of transmission;
Generating the replacement signal,
Correcting the intensity of the replacement signal by reflecting the transmission intensity
Sensory substitution and multipath transmission method of human visual information. 11. The method of claim 10, wherein generating the replacement signal comprises:
Calculate the time required to convert the extracted information into the alternative signal;
Determining whether to simplify the process of converting the extracted information into the alternative signal based on the time, and generating the alternative signal by converting the extracted information based on the determination result
Sensory substitution and multipath transmission method of human visual information.

Images