KR102169884B1 - Night vision system - Google Patents

Abstract

The present invention provides a night vision method for providing more forward image information to a driver by detecting vehicles, pedestrians, animals, etc. in front that cannot be seen by the driver's eyes outside the driver's field of view secured by a headlight during night driving, and It's about the system. According to an aspect of the present invention, a method for displaying night vision information includes: a first step of obtaining thermal energy information for at least one object through a Far Infra-Red (FIR) camera; A second step of extracting first heat energy information satisfying a preset condition from among the heat energy information; A third step of extracting a vertical edge from the thermal energy information; A fourth step of extracting second thermal energy information mapped to the extracted vertical edge from among the first thermal energy information; A fifth step of extracting the vertical one end and the other end point of the object included in the second thermal energy information; A sixth step of generating a region of interest (ROI) using the one end and the other end point; A seventh step of comparing an object included in the ROI with a plurality of classifiers related to the first object to be identified; An eighth step of extracting an ROI corresponding to at least one of the plurality of classifiers from the ROI; And a ninth step of displaying the extracted ROI through a display unit.

Description

Night vision system

The present invention provides a night vision method for providing more forward image information to a driver by detecting vehicles, pedestrians, animals, etc. in front that cannot be seen by the driver's eyes outside the driver's field of view secured by a headlight during night driving, and It's about the system.

Specifically, in the present invention, when the FIR (Far Infra-Red) camera installed in the vehicle detects the thermal energy of an object, it converts it into a digital image signal, and analyzes, corrects, and supplements the converted image signal through a specific algorithm to display It relates to a night vision method and system that supports the driver to know the situation ahead by displaying on or outputting an alarm.

In general, a thermal imaging camera using Far Infra-Red (FIR) detects a far infrared region rather than a visible region.

That is, the thermal imaging camera detects infrared thermal energy emitted from a subject, and acquires the detected result as a thermal image.

Due to these characteristics, thermal imaging cameras can be used to detect people with body temperature.

Meanwhile, a pedestrian detection device using an FIR camera has recently been applied to a vehicle.

FIR cameras applied to such vehicles must provide consistent detection performance regardless of the current vehicle driving environment.

That is, the FIR camera applied to a vehicle must provide consistent pedestrian detection performance even in a city center where diffuse reflection is severe due to ambient light sources or an urban exterior area where diffuse reflection is not severe due to the ambient light sources.

Therefore, the FIR camera applied to a conventional vehicle is basically added with a function of adjusting sensitivity in order to provide consistent detection performance.

This sensitivity control may adjust the sensitivity by adjusting the radiation temperature difference setting between objects in the photographed thermal image.

However, there is a limit to sensitivity control in urban areas where scattered reflections such as light sources emitted from surrounding buildings and light sources emitted from signboards attached to surrounding buildings, and particularly, diffuse reflections of lights from front and rear vehicles, are extreme.

Moreover, FIR cameras are expensive equipment, and in urban areas where diffuse reflections are severe as described above, they do not satisfy the price/detection performance required by users.

Of course, pedestrian detection using a CMOS camera, which is relatively inexpensive compared to FIR cameras, is also possible. However, since the CMOS camera does not acquire an image of a subject by detecting infrared heat energy emitted from the subject, there is no sensitivity control function using the difference in radiation temperature of the subject.

That is, it is impossible to detect pedestrians at night, especially in urban areas where diffuse reflection is severe, and it is not possible to clearly distinguish whether the object is a pedestrian or an animal, like an FIR camera.

On the other hand, in recent years, pedestrians (including bicycle occupants) occupy a significant portion of traffic accidents, and these accidents are accidents that must be prevented in that once an accident occurs, it leads to major accidents such as death.

Therefore, many studies on the protection of pedestrians (including bicycle occupants) are in progress.

Nevertheless, in the conventional pedestrian protection device, there is still a problem that the background of surrounding buildings, surrounding objects, etc., which should be excluded in order to accurately recognize the pedestrian, cannot be accurately distinguished from the pedestrian, and due to this, proper pedestrian protection cannot be provided. There is a situation.

In addition, a pedestrian detection apparatus using a conventional FIR camera performs pedestrian detection by setting a region in which the temperature is high when applied to pedestrian detection as a region of interest.

However, since the temperature of the pedestrian detection device using the FIR camera is not relatively constant depending on the environment such as day/night, it is difficult to expect consistent performance in a region where the temperature is high.

In addition, in addition to the pedestrian detection device using the FIR camera, various pedestrian detection devices have been proposed, but mainly include sensors mounted in the front of the vehicle, such as the inside of a bumper or a fender of a vehicle.

Pedestrian detection devices including such sensors can easily provide an output signal of an error, so that the safety device may be inappropriately deployed when a hit such as, for example, a cone of a road or an animal is hit.

Therefore, to solve the above problem and to provide more forward image information to the driver by detecting vehicles, pedestrians, and animals in front that are outside the driver's field of view secured by the headlight during night driving and cannot be seen by the driver's eyes. There is a need for a night vision method and system.

Korean Intellectual Property Office Registration No. 10-1410215 Korean Intellectual Property Office Registration No. 10-1284892

The present invention provides a night vision method for providing more forward image information to a driver by detecting vehicles, pedestrians, animals, etc. in front that cannot be seen by the driver's eyes outside the driver's field of view secured by a headlight during night driving, and We want to provide a system.

Specifically, in the present invention, when the FIR (Far Infra-Red) camera installed in the vehicle detects the thermal energy of an object, it converts it into a digital image signal, and analyzes, corrects, and supplements the converted image signal through a specific algorithm to display It is intended to provide users with a night vision method and system that supports the driver to know the situation ahead by displaying on or outputting an alarm.

However, the technical problems to be achieved in the present invention are not limited to the technical problems mentioned above, and other technical problems that are not mentioned are clearly to those of ordinary skill in the technical field to which the present invention belongs from the following description. It will be understandable.

A method for displaying night vision information according to an aspect of the present invention for achieving the above technical problem includes: a first step of obtaining thermal energy information for at least one object through a Far Infra-Red (FIR) camera; A second step of extracting first heat energy information satisfying a preset condition from among the heat energy information; A third step of extracting a vertical edge from the thermal energy information; A fourth step of extracting second thermal energy information mapped to the extracted vertical edge from among the first thermal energy information; A fifth step of extracting the vertical one end and the other end point of the object included in the second thermal energy information; A sixth step of generating a region of interest (ROI) using the one end and the other end point; A seventh step of comparing an object included in the ROI with a plurality of classifiers related to the first object to be identified; An eighth step of extracting an ROI corresponding to at least one of the plurality of classifiers from the ROI; And a ninth step of displaying the extracted ROI through a display unit.

In addition, the preset condition may be a condition having a brightness value equal to or greater than a preset value.

Further, the preset condition includes a first condition and a second condition, the brightness value of the first condition is greater than the brightness value of the second condition, and the fourth step is a first condition that satisfies the second condition. It is applied to thermal energy information, and the fifth step may be performed by using the first thermal energy information and the second energy information satisfying the first condition.

In addition, the second step and the third step may be performed simultaneously.

In addition, between the eighth and ninth steps, a step 8-1 of tracking an object included in the extracted ROI by using at least one of the one end and the other end point; further includes, and the th In step 9, information tracked in step 8-1 may be displayed.

In addition, the first to ninth steps are repeatedly performed, and the sixth step, which is repeatedly performed, is a vanishing line around the vertical axis of the object included in the extracted ROI from the eighth step previously performed. step 6-1 of setting line); Step 6-2 of setting a vanishing region based on the set vanishing line; A 6-3 step of comparing the vanishing area statistical information related to the first object with the vanishing area; And a 6-4 step of generating an ROI using the one end point and the other end point when the difference between the vanishing area statistical information and the vanishing area is within a preset range.

In addition, the first to ninth steps are repeatedly performed, and the eighth step, which is repeatedly performed, is a vanishing line around the vertical axis of the object included in the extracted ROI from the previously performed eighth step. line) setting; An 8-3 step of setting a vanishing region based on the set vanishing line; An 8-4th step of comparing the vanishing area statistical information related to the first object with the vanishing area; And a step 8-5 of extracting an ROI corresponding to at least one of the plurality of classifiers from the ROI when the difference between the vanishing area statistical information and the vanishing area is within a preset range. I can.

In addition, the first object includes a vehicle, a pedestrian, and an animal in front, and in the first step, the thermal energy information may be obtained at night.

In addition, after the ninth step, a tenth step of transmitting the extracted ROI to the outside; further comprising, a short-range communication or a long-distance communication is used for the transmission, and the short-range communication is Wi-Fi (Wireless Fidelity) , Bluetooth, Radio Frequency Identification (RFID), infrared communication (IrDA, infrared Data Association), Ultra Wideband (UWB), and at least one of ZigBee technology is used, and the long-distance communication is CDMA (code division multiple access) , At least one of frequency division multiple access (FDMA), time division multiple access (TDMA), orthogonal frequency division multiple access (OFDMA), and single carrier frequency division multiple access (SC-FDMA) techniques may be used.

On the other hand, a night vision information display device according to another aspect of the present invention for achieving the above technical problem, FIR (Far Infra-Red) camera for obtaining thermal energy information on at least one object; A first thermal energy information that satisfies a preset condition is extracted from among the thermal energy information, a vertical edge is extracted from the thermal energy information, and is mapped to the extracted vertical edge among the first thermal energy information. 2 Extract thermal energy information, extract vertical one end and the other end point of the object included in the second heat energy information, generate a region of interest (ROI) using the one end and the other end point, and identify the first object A control unit for comparing a plurality of classifiers related to the object included in the ROI and extracting an ROI corresponding to at least one of the plurality of classifiers from the ROI; And a display unit that displays the extracted ROI.

The present invention provides a night vision method for providing more forward image information to a driver by detecting vehicles, pedestrians, animals, etc. in front that cannot be seen by the driver's eyes outside the driver's field of view secured by a headlight during night driving, and System can be provided.

Specifically, in the present invention, when the FIR (Far Infra-Red) camera installed in the vehicle detects the thermal energy of an object, it converts it into a digital image signal, and analyzes, corrects, and supplements the converted image signal through a specific algorithm to display A night vision method and system that supports the driver to know the situation ahead by displaying on or outputting an alarm can be provided to the user.

However, the effects obtainable in the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those of ordinary skill in the art from the following description. I will be able to.

1A and 1B are diagrams for comparing night vision distances of a night vision system and a headlamp system according to the present invention.
2A and 2B are diagrams showing a specific state of mounting a night vision system in a vehicle in connection with the present invention.
3 shows a block diagram of a night vision system proposed by the present invention.
4 shows an example of a detailed block diagram of a night vision system according to the present invention described in FIG. 3.
5 is a flowchart illustrating a specific process of displaying information according to a specific algorithm by an FIR camera sensing thermal energy of an object through the night vision system according to the present invention.
6A to 6E are diagrams for explaining a detailed process of generating an ROI through an input image among the processes described in FIG. 5.
7 is a diagram illustrating a specific process of classifying a pedestrian according to machine learning among the processes described in FIG. 5.
8A and 8B are diagrams for explaining in detail a process of a false recognition removal algorithm using a multi-classifier among the processes described in FIG. 5.
9A and 9B are diagrams for specifically explaining a Head Point-based tracking process among the processes described in FIG. 5.
FIG. 10 is a flowchart illustrating another example in which an FIR camera detects thermal energy of an object and displays information according to a specific algorithm through the night vision system according to the present invention.
11 is a diagram illustrating a specific state in which the driver can easily check the surrounding situation through the night vision system according to the present invention in a situation in which the field of view such as fog, dust, etc. is restricted.
12 is a flowchart illustrating another process in which an FIR camera detects thermal energy of an object and displays information according to a specific algorithm through the night vision system according to the present invention.

In general, a vehicle checks route information to a desired destination using a navigation system, and conveniently searches for a destination according to route information guided through the navigation system.

Such a navigation system provides information such as route guidance, TV viewing, vehicle speed information display, and the like, but it is impossible to detect and warn a pedestrian or animal while driving on a steal.

While the vehicle is driving, the driver directly checks the appearance of pedestrians or animals with the naked eye, and controls the vehicle operation based on the detected information identified with the naked eye.

Here, most pedestrians or animals can be identified with the naked eye during the day, but human eyes are almost impossible to identify pedestrians or animals at night when the illumination is low.

Therefore, it is difficult to identify pedestrians or animals at night when the illumination is low, and accidents frequently occur in which pedestrians or animals collide with vehicles.

Recently, a pedestrian detection device using an FIR camera has been applied to a vehicle.

However, conventional pedestrian detection devices using FIR cameras have scattered reflections such as light sources emitted from surrounding buildings and light sources emitted from signboards attached to surrounding buildings, especially in urban areas where diffuse reflections of lights from front and rear vehicles are severe. There is a limit to the sensitivity control.

Moreover, FIR cameras are expensive equipment, and in urban areas where diffuse reflection is severe as described above, they do not satisfy the price-to-detection performance required by users.

That is, it is impossible to detect pedestrians at night, especially in urban areas where diffuse reflection is severe, and it is not possible to clearly distinguish whether an object is a pedestrian or an animal, like an FIR camera.

In addition, in the conventional pedestrian protection device using an FIR camera, there is still a problem in that the background of surrounding buildings, surrounding objects, etc., which should be excluded in order to accurately recognize the pedestrian, cannot be accurately distinguished from the pedestrian. I can't.

In addition, the conventional pedestrian detection apparatus using the FIR camera has a problem that it is difficult to expect consistent performance in a region where the temperature is high because the temperature is relatively not constant depending on the environment such as day/night.

Therefore, in the present specification, a night vision method and system that solves the above problem is provided.

The night vision system (NVS) proposed by the present invention detects vehicles, pedestrians, and animals in front that cannot be seen by the driver's eyes outside the driver's field of view secured by the headlight during night driving. It is a driver assistance system to provide more forward image information.

1A and 1B are diagrams for comparing night vision distances of a night vision system and a headlamp system according to the present invention.

Referring to FIG. 1A, the night vision system according to the present invention can secure a field of view 3 to 5 times wider than when the low beam of a general head lamp is turned on.

In addition, referring to FIG. 1B, the distance that can be visually identified by irradiating the headlight is limited to within 60m, whereas the recognition of a pedestrian in front through the night vision system according to the present invention is possible up to 130m, which is more effective in securing a front view.

Meanwhile, FIGS. 2A and 2B are diagrams showing a detailed state of mounting a night vision system in a vehicle in connection with the present invention.

As shown in FIG. 2A, when a Far Infra-Red (FIR) camera installed inside a vehicle grill detects the thermal energy of an object, the integrated ECU converts it into a digital image signal.

Thereafter, the converted image signal is analyzed, corrected and supplemented through an image algorithm, and the processed image signal is displayed on the display or a warning sound is output, so that the driver can know the situation ahead.

FIG. 2B is a diagram illustrating a specific state in which a pedestrian detected through a Far Infra-Red (FIR) camera is displayed through a display unit.

Prior to describing the specific operation of the present invention, a configuration of the night vision system proposed by the present invention will be preliminarily described.

3 is a block diagram of a night vision system proposed by the present invention.

Referring to FIG. 3, a night vision system 100 proposed by the present invention may include a far infrared camera 122, a controller 180, and a display unit 151.

Here, the far-infrared camera 122 is a camera that photographs at least one object using far infrared (Far Infrared, LongWave Infrared, FIR).

Far-infrared rays generally mean infrared rays having a wavelength of 8 μm or more, and because they are longer than visible rays, they are invisible, heat action is large, and penetration is strong.

The characteristics of the far infrared camera 122 according to the present invention will be described.

Since a camera using a general CCD or CMOS device detects and projects light in the visible light region, it is possible to acquire an image similar to that of a human eye.

On the other hand, the far infrared camera 122 projects light in an infrared band that humans cannot see.

Infrared refers to light in the band of 750nm to 1mm among the wavelengths of light. Among these infrared bands, the light of NIR (Near Infra-Red) refers to the wavelength of 700nm to 1400nm, and the light in the NIR band is not visible to the human eye. It can be detected with a CCD or CMOS device, and if a filter is used, only light in the NIR band can be detected.

In contrast, FIR light is also known as LWIR (Long Wavelength Infra-Red), and infrared rays represent a band of 8 μm to 15 μm of the wavelength of light.

In particular, since the wavelength of the FIR band changes according to temperature, there is an advantage of distinguishing the temperature.

The body temperature of a person (pedestrian) that is the target of the far-infrared camera 122 has a wavelength of 10 μm.

Also, the controller 180 controls the overall operation of the night vision system 100.

In particular, the information obtained through the far-infrared camera 122 is converted into a digital image signal, the converted image signal is analyzed, corrected, and supplemented through a specific algorithm, and the processed image signal is displayed on the display unit 151 It can be controlled to be output through ).

The controller 180 may be implemented in a recording medium that can be read by a computer or a similar device using software, hardware, or a combination thereof.

According to the hardware implementation, the controller 180 includes application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), and processors. processors), controllers, micro-controllers, microprocessors, and electrical units for performing other functions.

According to software implementation, the controller 180 may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein. The software code can be implemented as a software application written in an appropriate programming language. The software code may be stored in a memory 160 to be described later.

In addition, the display unit 151 provides a function of outputting the degree processed by the controller 180 to the outside.

Here, the display unit 151 is a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. It may include at least one of a (flexible display) and a 3D display.

Hereinafter, a detailed configuration of the night vision system 100 proposed by the present invention described in FIG. 3 will be described with reference to FIG. 4.

4, the night vision system 100 includes a wireless communication unit 110, an audio/video (A/V) input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, and It may include a memory 160, an interface unit 170, a control unit 180, a power supply unit 190, and the like.

However, since the components shown in FIG. 4 are not essential, a night vision system having more components or fewer components may be implemented.

Hereinafter, the components will be described in order.

The wireless communication unit 110 may include one or more modules that enable wireless communication between a night vision system and a wireless communication system or between a device and a network in which the device is located.

For example, the wireless communication unit 110 may include a mobile communication module 112, a wireless Internet module 113, a short-range communication module 114, a location information module 115, and the like.

The mobile communication module 112 transmits and receives a radio signal with at least one of a base station, an external device, and a server on a mobile communication network.

It may contain various types of data according to transmission/reception of text/multimedia messages.

The wireless Internet module 113 refers to a module for wireless Internet access, and may be built-in or external to the night vision system. As a wireless Internet technology, WLAN (Wireless LAN) (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access), and the like may be used.

The short-range communication module 114 refers to a module for short-range communication. Short range communication technology, such as Bluetooth, Radio Frequency Identification (RFID), infrared data association (IrDA), Ultra-WideBand (UWB), ZigBee, and WiFi (Wireless Fidelity, Wi-Fi). Can be used.

The location information module 115 is a module for obtaining a location of a night vision system, and a representative example thereof is a GPS (Global Position System) module.

Referring to FIG. 4, an audio/video (A/V) input unit 120 is for inputting an audio signal or a video signal, and may include a camera 121 and a microphone 122. The camera 121 processes image frames such as still images or moving pictures obtained by the image sensor in the photographing mode. The processed image frame may be displayed on the display unit 151.

The image frame processed by the camera 121 may be stored in the memory 160 or transmitted to the outside through the wireless communication unit 110. Two or more cameras 121 may be provided depending on the use environment.

The microphone 122 receives an external sound signal by a microphone in a recording mode, a voice recognition mode, and the like and processes it as electrical voice data. The processed voice data may be converted into a format transmittable to a mobile communication base station through the mobile communication module 112 and then output. Various noise removal algorithms may be implemented in the microphone 122 to remove noise generated in the process of receiving an external sound signal.

Next, the far-infrared camera 122 projects and photographs light in an infrared band that a person cannot see.

FIR light is also known as LWIR (Long Wavelength Infra Red), and infrared rays represent a band of 8 μm to 15 μm among the wavelengths of light, and the FIR band changes the wavelength according to temperature, so the temperature can be distinguished.

The body temperature of a person (pedestrian) that is the target of the far-infrared camera 122 has a wavelength of 10 μm, and through the far-infrared camera 122, it is possible to capture images and videos of a specific object even at night.

Next, the user input unit 130 generates input data for the user to control the operation of the night vision system. The user input unit 130 may include a key pad, a dome switch, a touch pad (positive pressure/power failure), a jog wheel, a jog switch, and the like.

The sensing unit 140 detects the current state of the night vision system, such as the open/closed state of the night vision system, the position of the night vision system, the presence of user contact, the orientation of the night vision system, and acceleration/deceleration of the night vision system. It generates a sensing signal to control the operation of.

The sensing unit 140 may sense whether the power supply unit 190 supplies power, whether the interface unit 170 is coupled to an external device, or the like.

The output unit 150 is for generating output related to visual, auditory, or tactile sense, and includes a display unit 151, an audio output module 152, an alarm unit 153, a haptic module 154, and a projector module ( 155), a head-up display (HUD), a head mounted display (HMD), and the like.

The display unit 151 displays (outputs) information processed by the night vision system.

The display unit 151 includes a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), and a flexible display. display) and a 3D display.

Some of these displays may be configured as a transparent type or a light-transmitting type so that the outside can be seen through them. This may be referred to as a transparent display, and a representative example of the transparent display is TOLED (Transparant OLED). The rear structure of the display unit 151 may also be configured as a light transmission type structure. With this structure, the user can see an object located behind the night vision system body through an area occupied by the display unit 151 of the night vision system body.

Two or more display units 151 may exist depending on the implementation type of the night vision system. For example, in a night vision system, a plurality of display units may be spaced apart or integrally disposed on one surface, or may be disposed on different surfaces.

When the display unit 151 and a sensor (hereinafter referred to as'touch sensor') for detecting a touch motion form a mutual layer structure (hereinafter, referred to as a'touch screen'), the display unit 151 It can also be used as an input device. The touch sensor may have, for example, a touch film, a touch sheet, a touch pad, or the like.

The touch sensor may be configured to convert a change in pressure applied to a specific portion of the display unit 151 or a capacitance generated at a specific portion of the display unit 151 into an electrical input signal. The touch sensor may be configured to detect not only a touched position and area, but also a pressure at the time of touch.

When there is a touch input to the touch sensor, a signal(s) corresponding thereto is transmitted to the touch controller. The touch controller processes the signal(s) and then transmits the corresponding data to the controller 180. As a result, the controller 180 can know whether an area of the display unit 151 is touched.

The proximity sensor 141 may be disposed in an inner area of the night vision system surrounded by the touch screen or near the touch screen. The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or an object existing in the vicinity using the force of an electromagnetic field or infrared rays without mechanical contact. Proximity sensors have a longer lifespan and higher utilization than contact sensors.

Examples of the proximity sensor include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive type proximity sensor, a magnetic type proximity sensor, an infrared proximity sensor, and the like. When the touch screen is capacitive, it is configured to detect the proximity of the pointer by a change in an electric field according to the proximity of the pointer. In this case, the touch screen (touch sensor) may be classified as a proximity sensor.

Hereinafter, for convenience of explanation, an action of allowing the pointer to be recognized as being positioned on the touch screen by approaching the touch screen without contacting the pointer is referred to as "proximity touch", and the touch The act of actually touching the pointer on the screen is referred to as "contact touch". A position at which a proximity touch is performed by a pointer on the touch screen means a position at which the pointer corresponds vertically to the touch screen when the pointer is touched.

The proximity sensor detects a proximity touch and a proximity touch pattern (eg, a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, a proximity touch movement state, etc.). Information corresponding to the sensed proximity touch operation and proximity touch pattern may be output on the touch screen.

The sound output module 152 may output audio data received from the wireless communication unit 110 or stored in the memory 160 in a recording mode, a voice recognition mode, or a broadcast reception mode. The sound output module 152 also outputs sound signals related to functions performed in the night vision system. The sound output module 152 may include a receiver, a speaker, a buzzer, and the like.

The alarm unit 153 outputs a signal for notifying the occurrence of an event of the night vision system.

The alarm unit 153 may output a signal for notifying the occurrence of an event in a form other than a video signal or an audio signal, for example, by vibration.

The video signal or audio signal may also be output through the display unit 151 or the audio output module 152, so that they 151 and 152 may be classified as part of the alarm unit 153.

The haptic module 154 generates various tactile effects that a user can feel. A typical example of the tactile effect generated by the haptic module 154 is vibration. The intensity and pattern of the vibration generated by the haptic module 154 can be controlled.

For example, different vibrations may be synthesized and output or may be sequentially output.

In addition to vibration, the haptic module 154 is used for stimulation such as an arrangement of pins that move vertically with respect to the contact skin surface, a blowing force or suction force of air through a jet or inlet, a grazing on the skin surface, contact of an electrode, electrostatic force, etc. It can generate various tactile effects, such as the effect by the effect and the effect by reproducing the feeling of cooling and warming using an endothermic or heat generating element.

The haptic module 154 may not only transmit a tactile effect through direct contact, but may also be implemented so that a user can feel the tactile effect through muscle sensations such as a finger or an arm. Two or more haptic modules 154 may be provided depending on the configuration aspect of the night vision system.

The projector module 155 is a component for performing an image project function using a night vision system, and is the same as or at least as an image displayed on the display unit 151 according to a control signal from the controller 180. Some may display different images on an external screen or wall.

Specifically, the projector module 155 generates an image to be output to the outside by using a light source (not shown) that generates light (for example, laser light) for outputting an image to the outside, and light generated by the light source. It may include an image generating means (not shown) for performing and a lens (not shown) for expanding the image to the outside at a predetermined focal length. In addition, the projector module 155 may include a device (not shown) capable of adjusting an image projection direction by mechanically moving a lens or the entire module.

The projector module 155 may be divided into a cathode ray tube (CRT) module, a liquid crystal display (LCD) module, a digital light processing (DLP) module, and the like according to the device type of the display means. In particular, the DLP module may be advantageous for miniaturization of the projector module 151 by expanding and projecting an image generated by reflecting light generated from a light source onto a digital micromirror device (DMD) chip.

Preferably, the projector module 155 may be provided on the side, front, or rear of the night vision system in the longitudinal direction. Of course, it is natural that the projector module 155 may be provided at any position of the night vision system as needed.

In addition, the head-up display (HUD) 156 refers to a device that projects the current vehicle speed, fuel level, and navigation directions information as a graphic image on the windshield of the driver.

The information acquired through the far-infrared camera 122 may be output through the head-up display 156.

In addition, a head mounted display (HMD) 157 is a representative device capable of outputting virtual reality information.

Virtual reality is a human-computer that creates a specific environment or situation through a computer into 3D content with a three-dimensional effect, and makes it as if the person using the 3D content is interacting with the actual surrounding situation and environment. It refers to the interface of

In general, the three-dimensional effect perceived by humans is the degree of change in the thickness of the lens according to the position of the object being observed, the difference in the angle between both eyes and the object, the difference in the position and shape of the object visible to the left and right eyes, and the parallax caused by the motion of the object. , And other effects of various psychological and memory effects, etc.

Among them, the most important factor that a person feels a three-dimensional effect is binocular disparity that occurs when the human eyes are about 6.5 cm apart in the horizontal direction. In other words, by binocular parallax, the object is viewed with a difference in angle to the object, and due to this difference, images entering each eye have different images. When these two images are transmitted to the brain through the retina, the brain will You can feel the original 3D stereoscopic image by precisely fusion of each other.

These three-dimensional 3D contents have already been widely used in various media fields and are receiving favorable reviews from consumers. For example, 3D movies, 3D games and experiential displays are typical.

In addition to the generalization of virtual reality technology 3D contents, development of a technology capable of providing a more immersive virtual reality service is required in various ways.

In general, an image display device uses a precise optical device to focus image light generated at a location very close to the eye so that a large virtual screen can be formed at a distance, allowing users to view an enlarged virtual image. Refers to an image display device.

In addition, the image display device is a see-close type in which the surrounding environment can not be seen and only the image light emitted from the display device can be viewed, and the image light emitted from the display device can be viewed through a window at the same time. It can be divided into see-through.

The head mounted display (HMD, 157) according to the present invention refers to various digital devices that can be worn on the head like glasses to receive multimedia contents. According to the trend of lightening and miniaturization of digital devices, various wearable computers are being developed, and HMDs are also widely used. The HMD 157 goes beyond a simple display function and is combined with an augmented reality technology and an N screen technology to provide various conveniences to users.

For example, when a microphone and a speaker are mounted on the HMD 157, the user can make a phone call while wearing the HMD 157. In addition, for example, when the far-infrared camera 122 is mounted on the HMD 157, the user can capture an image in a direction desired by the user while wearing the HMD 157.

In addition, the memory unit 160 may store a program for processing and control of the controller 180, and temporarily store input/output data (eg, messages, audio, still images, moving pictures, etc.). It can also perform a function for. The frequency of use of each of the data may also be stored in the memory unit 160. In addition, the memory unit 160 may store data on vibrations and sounds of various patterns output when a touch input on the touch screen is performed.

The memory 160 is a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), and RAM. (Random Access Memory, RAM), SRAM (Static Random Access Memory), ROM (Read-Only Memory, ROM), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic It may include at least one type of storage medium among a disk and an optical disk. The night vision system may operate in connection with a web storage that performs a storage function of the memory 160 over the Internet.

The interface unit 170 serves as a passage for all external devices connected to the night vision system. The interface unit 170 receives data from an external device or receives power and transmits it to each component inside the night vision system, or transmits data inside the night vision system to an external device. For example, a wired/wireless headset port, an external charger port, a wired/wireless data port, a memory card port, a port for connecting a device equipped with an identification module, an audio input/output (I/O) port, A video input/output (I/O) port, an earphone port, etc. may be included in the interface unit 170.

The identification module is a chip that stores various information for authenticating the right to use the night vision system, and includes a user identification module (UIM), a subscriber identification module (SIM), and a universal user authentication module (Universal Subscriber). Identity Module, USIM), etc. may be included. A device equipped with an identification module (hereinafter,'identification device') may be manufactured in the form of a smart card. Therefore, the identification device can be connected to the night vision system through the port.

The interface unit becomes a path through which power from the cradle is supplied to the night vision system when the night vision system is connected to an external cradle, or various command signals input from the cradle by a user are transmitted to the mobile device. It can be a passage. Various command signals or the power input from the cradle may be operated as signals for recognizing that the mobile device is correctly mounted on the cradle.

The controller 180 typically controls the overall operation of the night vision system.

The power supply unit 190 receives external power and internal power under the control of the controller 180 to supply power necessary for the operation of each component.

Various embodiments described herein may be implemented in a recording medium that can be read by a computer or a similar device using, for example, software, hardware, or a combination thereof.

According to hardware implementation, the embodiments described herein include application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), It may be implemented using at least one of processors, controllers, micro-controllers, microprocessors, and electrical units for performing other functions, in some cases herein. The described embodiments may be implemented by the controller 180 itself.

According to the software implementation, embodiments such as procedures and functions described in the present specification may be implemented as separate software modules. Each of the software modules may perform one or more functions and operations described herein. The software code can be implemented as a software application written in an appropriate programming language. The software code may be stored in the memory 160 and executed by the controller 180.

Meanwhile, based on the above-described configuration of the present invention, the controller 180 converts the object-related content (document, image, video, etc.) acquired by the far-infrared camera 122 into a digital image signal, and the converted image A detailed operation of analyzing, modifying, and supplementing a signal through a specific algorithm and controlling the display 151 to display the processed content will be described.

Method 1

5 is a flowchart illustrating a specific process of displaying information according to a specific algorithm by an FIR camera sensing thermal energy of an object through the night vision system according to the present invention.

6A to 6E are diagrams for explaining a specific process of generating an ROI through an input image among the processes described in FIG. 5.

Referring to FIG. 5, first, a step S100 of obtaining an image of thermal energy of at least one object through the far-infrared camera 122 is performed.

Referring to FIG. 6A, a specific state in which thermal energy images of a plurality of objects are obtained through the far-infrared camera 122 of the present invention is illustrated.

When step S100 is performed, the controller 180 performs step S110 of extracting an image of a high threshold having a brightness value greater than or equal to a preset value among the input images.

Here, the threshold value (threshold value) can be statistically estimated using the mean, standard deviation, histogram, etc. of the image, or can be set by the user inputting an arbitrary value.

This is based on the fact that the brightness value of ordinary people (pedestrians) is brighter than the surrounding background.

Referring to FIG. 6B, a certain area is photographed as a thermal energy image through the far-infrared camera 122, and a high-threshold bright object having a brightness value equal to or higher than a preset value is illustrated among the photographed images.

However, if the acquired image characteristics of pedestrians adjacent to and far away from the far-infrared camera 122 are different, and a plurality of pedestrians other than one pedestrian are adjacent or the weather is very cold or very hot, one threshold in step S110 With Threshold, it can be difficult to shoot without missing all pedestrians.

Therefore, unlike step S110, step S120 of extracting all object images brightly rising on a low threshold by lowering the brightness value (S120) is performed in parallel.

Here, the threshold value (threshold value) can be statistically estimated using the mean, standard deviation, histogram, etc. of the image, or can be set by the user inputting an arbitrary value.

However, in step S120, unnecessary noises other than targeting pedestrians may be frequently included and extracted.

Therefore, in order to solve this problem, step S130 of extracting the image input in step S100 based on an edge is performed together.

In the case of a person (pedestrian), since a VERTICAL edge component, that is, a vertical edge component, appears strongly due to a body characteristic, this is to remove unnecessary noise obtained in step S120 by using this.

The edge image can be estimated from a sobel filter, a canny edge detector, and a laplacian filter.

The controller 180 extracts all object images brightly on a low threshold in step S120, and multiplies the extracted images based on the edge in step S130.

Therefore, even if the object is extracted by exceeding a low threshold in step S120, the object without the vertical edge component disappears.

For example, when a pedestrian stands on a road, the pedestrian and the road are treated as one object and may be extracted as an image in step S120.

However, by step S130, since the road has no vertical edge component, only the image element of the pedestrian located on the road remains.

In addition, multiplying the steps S120 and S130 may perform a masking function to compensate for an object that has not been completely extracted in addition to extracting an object having a vertical edge component.

That is, in step S120, the pedestrian is not completely extracted as a single connected bright thermal image, but a shape with a cut in the middle may be extracted.

At this time, by applying the element from which the edge component is extracted in step S130 to step S120, masking in which the broken space is filled may be performed.

Referring to FIG. 6C, masking in which the broken space is filled may be performed by applying an element from which an edge component is extracted in step S130 and an element with a cut-off shape in step S120.

The controller may sequentially perform steps S110, S120, and S130, or may perform each step in parallel and simultaneously.

In steps S110 and S120 and S130, similar to the above-described masking operation, a labeling operation may be additionally performed.

Labeling is a work that attaches closely spaced chunks and segments, and can be seen as a work of tying the same object or separated objects due to a clear failure to acquire heat energy.

After the steps S110 to S130, the step of extracting the head point information (S140) is performed.

In step S140, the image extracted in step S110 and steps S120 and S130 are simultaneously applied to extract the head (head) point of the object from the extracted image.

That is, the uppermost region of the extracted image where the bright part ends may be determined as the head point region.

Here, head point extraction may be performed by skeletonization (thinning), local maximum extraction, and head recognition through learning.

Also, the step S150 of extracting a foot line of the object may be performed simultaneously with the step S140 or sequentially with the step S140.

In step S150, steps S120 and S130 are applied together and the extracted image is not used, but is performed through information in step S130 in which an edge map is extracted with only edge components.

In step S150, a point where each of the extracted images is brought up in the vertical direction from the bottom of the acquired image is found, and the point of encounter is determined as a footline of the object.

Here, the foot line (toe) information may be converted by performing dilation from the input image, and may be extracted based on a derivative value in the vertical direction (for example, a position that changes to 255->0).

The reason for performing steps S120, S130, S140, and S150 is to reduce the computational throughput when the controller 180 acquires a thermal image.

The elements that can be clearly identified as noise through steps S120, S130, S140 and S150 are not finally detected by checking objects present on all images obtained here, and are removed from the first half of the work. It is to proceed.

In the end, for the pedestrian recognition rate of the thermal image night vision system 100, a method of generating an ROI that is not sensitive to changes in season or weather without missing pedestrians is proposed.

Even through the steps S120 to S150, unwanted noise may be included, which may be removed through steps S160 to S180.

When the head point and the foot line are detected through steps S140 and S150, a region of interest (ROI) may be generated through them (S160).

Referring to FIG. 6D, a specific example of detecting a head and a foot line of a specific object is shown.

The ROI region generated in step S160 may be determined as a box having a specific ratio (eg, 2:1) based on the length of the head candidate and the corresponding toe candidate.

Referring to FIG. 6E, a detailed view of an ROI in the form of a box having a specific ratio based on the length of a foot line (toe candidate) corresponding to a head candidate is shown.

As a result, steps S110 to S160 include obtaining an input thermal image, estimating at least one threshold value based on the input image, and generating a first split image for dividing the image by the estimated threshold value. , Computing an edge image for the input image, estimating a threshold value based on the calculated edge image, generating a second segmented image for dividing the edge image by a threshold value estimated from the edge image, Generating a third segmented image by performing a multiplication operation on the first segmented image and the second segmented image, labeling the third segmented image, extracting head candidates for each label, first and second , Extracting a foot line (toe candidate) from the segmented image, and generating an ROI region by combining the head candidate point and the foot line (toe candidate).

Meanwhile, in generating the ROI in step S160, a vanishing line may be additionally used.

The vanishing line may serve to reduce misrecognition in step S180 to be described later, or may be used as a criterion for generating the ROI itself in step S160.

The vanishing line is a line on an invisible virtual space, and is used as a basis on the assumption that the center of the vertical axis of a pedestrian crosses the Vanishing line regardless of the distance.

That is, in order to solve the problem induced in that the size of the pedestrian is different and the distance between the pedestrian and the camera is different, the assumption that the center of the vertical axis of the pedestrian crosses the vanishing line regardless of the size and distance is used.

When the pedestrian recognition is performed from the input image of at least 1 frame or more acquired through the step S100, in step S160, the reference vanishing line may be estimated from the coordinates of the vertical axis of the recognized pedestrian area.

Also, before generating the ROI, the controller 180 may set a vanishing region based on the reference vanishing line and perform pedestrian recognition of the current frame based on the vanishing region.

At this time, the control unit 180 loads vanishing region statistics for the vertical axis center coordinates of the pedestrians stored in the memory 160 in advance, and calculates a difference between the statistical values with the vanishing region for the recognized vertical axis center coordinates of the pedestrians.

If the difference value is greater than a predetermined threshold value based on the difference in the statistical value, the controller 180 may determine that it is not an object for a pedestrian and may not generate the ROI itself.

For example, points of objects detected (detected) from previous pedestrians may be accumulated to continuously accumulate statistics of a Gaussian distribution, and the accumulated information may be stored in the memory 160 as a probability model.

The ROI can be treated as a box surrounding a person (pedestrian), and it is generated as an ROI only when the center of the box is less than 0.5, which is a preset difference in relation to the probability model.

If the number of ROIs generated through the vanishing line decreases in step S160, it is possible to efficiently reduce the process of distributed processing by the controller 180.

In generating the ROI in step S160, a method of using head point information extracted from a previous frame may also be applied.

That is, when the head point is extracted from the previous frame, the position of the head point in the next frame may be predicted by the controller 180.

Therefore, in the next frame, it is possible to reduce the calculation process used for ROI generation by acknowledging a specific area corresponding to the expected head point point as a head point, not using the rest, and using existing information.

If step S160 proceeds, N ROIs may be generated.

The controller 180 collects the generated N ROIs for parallel processing, and classifies pedestrians according to machine learning.

That is, the controller 180 stores a plurality of classifiers corresponding to pedestrians in the memory 160 in advance, and corresponds to a plurality of classifiers stored among pedestrian candidate objects included in the N ROIs. Only object images are extracted.

7 is a diagram illustrating a specific process of classifying a pedestrian according to machine learning among the processes described in FIG. 5.

Referring to FIG. 7, the training data set stored in the memory 160 is loaded by the control unit 180, and the control unit 180 includes a pedestrian candidate image included in the N ROIs and a plurality of loaded classifiers. Compared to each other, only corresponding objects are extracted.

Misrecognition may be removed by using a plurality of classifiers of different criteria in step S170.

8A and 8B are diagrams for specifically explaining a process of a false recognition removal algorithm using a multi-classifier among the processes described in FIG. 5.

Referring to FIGS. 8A and 8B, it can be seen that when different classifiers are applied, misrecognition patterns are different from each other.

In FIGS. 8A and 8B, red boxes are pedestrians estimated by the LBP-Cascade method, blue boxes are pedestrians estimated by Haar-Cascade methods, and green boxes indicate pedestrians estimated by LBP-Haar at the same time.

Therefore, it is possible to complement the missing parts by applying a plurality of different classifiers.

After step S170, the controller 180 performs a head point-based tracking step (S180).

In step S180, the controller 180 may use a particle filter, a mean shift, or a Kalman filter.

9A and 9B are diagrams for specifically explaining a Head Point-based tracking process among the processes described in FIG. 5.

9A is a flowchart illustrating a head point-based tracking step (S180), and the controller 180 first extracts a motion vector (S181).

Thereafter, the controller 180 performs a step (S182) of comparing the histogram with the tracking candidate.

After step S182, the controller 180 determines Best Matching (S183), and finally determines whether to track (S184).

9B is a diagram illustrating a specific state of the controller 180 tracking based on a head point.

Through such head point based tracking, the amount of calculation performed by the controller 180 can be drastically reduced.

The tracking in step S180 may be to achieve two purposes.

First, tracking is used to display an image displayed through the display unit 151 without interruption.

That is, when the recognition of the pedestrian through the control unit 180 is not continuously made, the pedestrian appears in a specific frame and may not appear in the other frame, and thus a flickering phenomenon may occur.

Therefore, even if a pedestrian is not recognized in a specific frame through the tracking in step S180, the content of the pedestrian in the previous frame is tracked and displayed on the display unit 151, thereby implementing a smooth image.

In this case, in the case of false recognition that is not a desired object, it exists only in a specific frame and does not appear continuously in other frames.

Therefore, such an object is treated as false recognition through tracking, and is not displayed on the display unit 151.

The desired object, the pedestrian, may not be detected in a few frames, but is detected in most frames. In this case, the controller 180 can control the object to be continuously displayed on the display unit 151 through tracking in this case. have.

Next, the purpose of step S180 is to remove false recognition.

Similar to step S160, a vanishing line may be used in step S180 to remove false positives.

As described above, the vanishing line is a line on an invisible virtual space, and is used on the basis of the assumption that the center of the vertical axis of the pedestrian crosses the vanishing line regardless of the distance. That is, in order to solve the problem induced in that the size of the pedestrian is different and the distance between the pedestrian and the camera is different, the assumption that the center of the vertical axis of the pedestrian crosses the vanishing line regardless of the size and distance is used.

When the pedestrian recognition is performed from the acquired input image of at least 1 frame, in step S180, a reference vanishing line may be estimated from the coordinates of the vertical axis of the recognized pedestrian area.

Also, the controller 180 may set a vanishing region based on a reference vanishing line, and perform pedestrian recognition of a current frame based on the vanishing region.

At this time, the control unit 180 loads vanishing region statistics for the vertical axis center coordinates of the pedestrians stored in the memory 160 in advance, and calculates a difference between the statistical values with the vanishing region for the recognized vertical axis center coordinates of the pedestrians.

If the difference value is greater than a predetermined threshold value based on the difference in statistics, the controller 180 determines that it is not an object for a pedestrian, treats it as a false recognition, and discards the object information.

For the purpose of removing false recognition in step S180, the foot line information acquired in step S150 may be additionally used.

That is, a foot line is held based on a box recognized as an ROI, and if the foot line is not a pedestrian, the shape change of the foot line is large based on the box, and thus the ROI may be treated as a false recognition and removed.

Thereafter, the controller 180 may control the finally extracted thermal image information to be output through the display unit 151 (S190).

2nd way

Meanwhile, in addition to the method of FIG. 5, a method of extracting thermal image information on a pedestrian may be applied in a different order.

FIG. 10 is a flowchart illustrating another example in which an FIR camera detects thermal energy of an object and displays information according to a specific algorithm through the night vision system according to the present invention.

In FIG. 10, steps S200 and S240 to S280 correspond to steps S100 and S150 to S190 of FIG. 5, respectively, and thus, repetitive descriptions will be omitted for simplicity of the specification.

The method of FIG. 10 is different from the method of FIG. 5 in steps S210 to S230.

That is, referring to FIG. 10, unlike the method in FIG. 5, when step S200 proceeds, the controller 180 does not distinguish between a high threshold and a low threshold, and extracts an image using only one threshold value. (S210).

Here, the threshold value (threshold value) can be statistically estimated using the mean, standard deviation, histogram, etc. of the image, or can be set by the user inputting an arbitrary value.

This is based on the fact that the brightness value of ordinary people (pedestrians) is brighter than the surrounding background.

In this case, in step S210, unnecessary noises other than targeting pedestrians may be frequently included and extracted.

Therefore, in order to solve this problem, step S220 of extracting the image input in step S200 based on an edge is performed together.

In the case of a person (pedestrian), since a VERTICAL edge component, that is, a vertical edge component, appears strongly due to body characteristics, this is used to remove unnecessary noise obtained in step S210.

The edge image can be estimated from a sobel filter, a canny edge detector, and a laplacian filter.

The controller 180 extracts all the object images brightly on the threshold in step S210 and multiplies the extracted images based on the edge in step S220.

Therefore, even if the object is extracted by passing the threshold in step S210, the object without the vertical edge component disappears.

In addition, multiplying the steps S210 and S220 may perform a masking function to compensate for an object that has not been completely extracted in addition to extracting an object having a vertical edge component.

That is, in step S210, the pedestrian may not be extracted as a completely connected bright thermal image, but may be extracted as a shape with a cut in the middle.

In this case, by applying the element from which the edge component is extracted in step S220 to step S210, masking in which the broken space is filled may be performed.

The controller may sequentially perform steps S210 and S220, or may perform each step in parallel and simultaneously.

In steps S210 and S220, similar to the above-described masking operation, a labeling operation may be additionally performed.

Labeling is a work that attaches closely spaced chunks and segments, and can be seen as a work of tying the same object or separated objects due to a clear failure to acquire heat energy.

After steps S210 and S220, a step of extracting head point information through the multiplied information (S230) is performed.

In step S230, the image extracted in step S210 and step S220 are simultaneously applied to extract the head (head) point of the object from the extracted image.

That is, the uppermost region of the extracted image where the bright part ends may be determined as the head point region.

Here, head point extraction may be performed by skeletonization (thinning), local maximum extraction, and head recognition through learning.

Thereafter, steps S240 to S280 are performed corresponding to each of steps S150 to S190 in FIG. 5.

11 shows the result of applying the night vision system according to the above-described algorithm.

FIG. 11A is an image acquired through a general camera 121 and shows a specific state in which pedestrians are not visible due to fog.

On the contrary, the image of the far-infrared camera 123 according to the algorithm proposed by the present invention recognizes four pedestrians, and the recognized result can be guided to the driver.

Meanwhile, FIG. 12 is a flowchart illustrating another process in which the FIR camera detects thermal energy of an object and displays information according to a specific algorithm through the night vision system according to the present invention.

Steps S300 to S390 in FIG. 12 correspond to each of the steps S100 to S190 in FIG. 5, but there are differences in steps S310, S320, and S340.

Referring to FIG. 12, step S310 is less frequent than step S330, but it may be extracted and included unnecessary in addition to targeting pedestrians.

Therefore, in the flowchart of FIG. 12, in order to solve this problem in step S310 as well, step S320 of extracting the image input in step S300 based on an edge may be performed.

In the case of a person (pedestrian), since a VERTICAL edge component, that is, a vertical edge component, appears strongly due to body characteristics, this is used to remove unnecessary noise obtained in step S310.

Here, the edge image can be estimated from a sobel filter, a canny edge detector, and a laplacian filter.

The controller 180 extracts all object images brightly on a high threshold in step S310, and multiplies the extracted images based on the edge in step S320.

Therefore, even if the object is extracted by exceeding a high threshold in step S310, the object without the vertical edge component disappears.

In addition, multiplying the steps S310 and S320 may perform a masking function to supplement an object that is not completely extracted, in addition to extracting an object having a vertical edge component.

That is, in step S310, the pedestrian is not completely extracted as a single connected bright thermal image, but may be extracted as a shape with a cut in the middle.

In this case, by applying the element from which the edge component is extracted in step S320 to step S310, masking in which the broken space is filled may be performed.

When the above-described configuration and method of the present invention is applied, when driving at night, more forward to the driver by detecting vehicles, pedestrians, and animals in front that cannot be seen by the driver's eyes outside the driver's field of view secured by the headlight A night vision method and system for providing image information may be provided to a user.

That is, in the present invention, when the FIR (Far Infra-Red) camera installed in the vehicle detects the thermal energy of an object, it converts it into a digital image signal, and analyzes, corrects and supplements the converted image signal through a specific algorithm to display A night vision method and system that supports the driver to know the situation ahead by displaying on or outputting an alarm can be provided to the user.

The embodiments of the present invention described above can be implemented through various means. For example, embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof.

In the case of hardware implementation, the method according to embodiments of the present invention includes one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), and Programmable Logic Devices (PLDs). , Field Programmable Gate Arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and the like.

In the case of implementation by firmware or software, the method according to the embodiments of the present invention may be implemented in the form of a module, procedure, or function that performs the functions or operations described above. The software code may be stored in a memory unit and driven by a processor. The memory unit may be located inside or outside the processor, and may exchange data with the processor through various known means.

Detailed description of the preferred embodiments of the present invention disclosed as described above has been provided to enable those skilled in the art to implement and implement the present invention. Although the above has been described with reference to preferred embodiments of the present invention, those skilled in the art will appreciate that various modifications and changes can be made to the present invention without departing from the scope of the present invention. For example, those skilled in the art may use the configurations described in the above-described embodiments in a manner that combines with each other. Thus, the invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The present invention may be embodied in other specific forms without departing from the spirit and essential features of the present invention. Therefore, the detailed description above should not be construed as restrictive in all respects and should be considered as illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention. The invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. In addition, the embodiments may be configured by combining claims that do not have an explicit citation relationship in the claims, or may be included as new claims by amendment after filing.

Claims (10)

A first step of obtaining thermal energy information for at least one object through a Far Infra-Red (FIR) camera;
A second step of extracting first thermal energy information that satisfies a condition having a brightness value equal to or greater than a preset value among the thermal energy information;
A third step of extracting a vertical edge from the thermal energy information;
A fourth step of extracting second thermal energy information mapped to the extracted vertical edge from among the first thermal energy information;
A fifth step of extracting the vertical one end and the other end point of the object included in the second thermal energy information;
A sixth step of generating a region of interest (ROI) using the one end and the other end point;
A seventh step of comparing an object included in the ROI with a plurality of classifiers related to the first object to be identified;
An eighth step of extracting an ROI corresponding to at least one of the plurality of classifiers from the ROI; And
Including; a ninth step of displaying the extracted ROI through a display unit,
The preset condition includes a first condition and a second condition,
The brightness value of the first condition is greater than the brightness value of the second condition,
The fourth step is applied to the first heat energy information satisfying the second condition,
The fifth step is performed by using the first heat energy information and the second energy information satisfying the first condition. delete delete The method of claim 1,
The second and third steps are performed at the same time. The method of claim 1,
Between the eighth and ninth steps,
Step 8-1 of tracking the object included in the extracted ROI by using at least one of the one end and the other end point; further comprising,
In the ninth step, the night vision information display method, characterized in that the information to be tracked in the 8-1 step is displayed. The method of claim 1,
The first to ninth steps are repeatedly performed,
The sixth step is repeatedly performed,
A 6-1 step of setting a vanishing line around a vertical axis of an object included in the extracted ROI from the eighth step previously performed;
Step 6-2 of setting a vanishing region based on the set vanishing line;
A 6-3 step of comparing the vanishing area statistical information related to the first object with the vanishing area; And
When the difference between the vanishing area statistical information and the vanishing area is within a preset range, a 6-4 step of generating an ROI using the one end point and the other end point; and night vision information display comprising: Way. The method of claim 1,
The first to ninth steps are repeatedly performed,
The eighth step is repeatedly performed,
An 8-2 step of setting a vanishing line around a vertical axis of an object included in the extracted ROI from the eighth step previously performed;
An 8-3 step of setting a vanishing region based on the set vanishing line;
An 8-4th step of comparing the vanishing area statistical information related to the first object with the vanishing area; And
8-5 step of extracting an ROI corresponding to at least one of the plurality of classifiers from the ROI when the difference between the vanishing area statistical information and the vanishing area is within a preset range; further comprising: Night vision information display method characterized by. The method of claim 1,
The first object includes a vehicle, a pedestrian, and an animal in front,
In the first step, the thermal energy information is obtained at night. The method of claim 1,
After the ninth step,
A tenth step of transmitting the extracted ROI to the outside; further comprising,
Short-range communication or long-distance communication is used for the transmission,
The short-range communication uses at least one of Wi-Fi (Wireless Fidelity), Bluetooth, RFID (Radio Frequency Identification), infrared communication (IrDA, infrared Data Association), UWB (Ultra Wideband), ZigBee technology,
The long-distance communication is CDMA (code division multiple access), FDMA (frequency division multiple access), TDMA (time division multiple access), OFDMA (orthogonal frequency division multiple access), SC-FDMA (single carrier frequency division multiple access) technology Night vision information display method, characterized in that using at least one of. A Far Infra-Red (FIR) camera for obtaining thermal energy information for at least one object;
Among the thermal energy information, first thermal energy information that satisfies a condition having a brightness value equal to or greater than a preset value is extracted, a vertical edge is extracted from the thermal energy information, and a vertical edge is extracted from the first thermal energy information. Extracting the mapped second heat energy information, extracting the vertical one end and the other end point of the object included in the second heat energy information, and generating a region of interest (ROI) using the one end and the other end point, A control unit for comparing a plurality of classifiers related to the first object to be identified with an object included in the ROI, and extracting an ROI corresponding to at least one of the plurality of classifiers from the ROI; And
Including; a display unit for displaying the extracted ROI,
The preset condition includes a first condition and a second condition,
The brightness value of the first condition is greater than the brightness value of the second condition,
The extraction of the second thermal energy information is applied to the first thermal energy information satisfying the second condition,
The extraction of the vertical one end and the other end point is performed by using the first heat energy information and the second energy information satisfying the first condition.

Images