KR20180113807A - Night Vision System using LiDAR(light detection and ranging) and RADAR(Radio Detecting And Ranging) - Google Patents

Abstract

The present invention provides a night vision system capable of providing forward image information to a driver by sensing a forward vehicle, a pedestrian, and an animal that cannot be seen by the driver′s eyes. A night vision information display method according to one aspect of the present invention includes: a first step of acquiring thermal energy information for at least one object through a far infra-red (FIR) camera; a second step of extracting first thermal energy information satisfying a predetermined condition among the thermal energy information; a third step of extracting a vertical edge from the thermal energy information; a fourth step of extracting second thermal energy information mapped to the extracted vertical edge of the first thermal energy information; a fifth step of extracting a vertical end point and the other end point of the object included in the second thermal energy information; a sixth step of generating a region of interest (ROI) using the one end point and the other end point; a seventh step of comparing a plurality of classifiers related to a first object to be identified with an object included in the ROI; an eighth step of extracting the ROI corresponding to at least one of the plurality of classifiers in the ROI; and displaying the extracted ROI through a display unit. A tenth step of acquiring additional information about the at least one object by using at least one of light detection and ranging (LIDAR) and radio detecting and ranging (RADAR) is performed between the first step and the fifth step. In the sixth step, the ROI may be generated by using the additional information and the one end point and the other end point.

Description

(Night Detection and Ranging) and RADAR (Night Detection and Ranging)

The present invention relates to a night vision method for detecting a forward vehicle, a pedestrian, and an animal which are out of sight of a driver secured by a headlight during nighttime driving and which can not be seen by the eyes of a driver, ≪ / RTI >

More particularly, the present invention relates to a method and apparatus for converting a digital image signal into a digital image signal when a FIR (Far Infra-Red) camera installed in a vehicle senses thermal energy of an object, reduces the amount of computation using information obtained through radars and radars, The present invention relates to a night vision method and system that analyze, correct, and supplement a signal through a specific algorithm (Algorithm) to display on a display or output an alarm so that a driver can know ahead of time.

Generally, a thermal imaging camera using Far Infra-Red (FIR) detects a far-infrared ray region rather than a visible light region.

That is, the infrared camera senses infrared heat energy emitted from the subject and acquires the sensed result as a thermal image.

This feature allows the thermal imaging camera to be used to sense people with body temperature.

Meanwhile, a pedestrian sensing apparatus using an FIR camera has recently been applied to a vehicle.

FIR cameras applied to these vehicles must provide consistent detection performance regardless of the current vehicle environment.

In other words, the FIR camera applied to the vehicle should provide consistent pedestrian detection performance even in an urban area with severe diffuse reflection due to the surrounding light sources or in a downtown area where diffuse reflection due to the surrounding light sources is not severe.

Therefore, the FIR camera applied to the conventional vehicle basically has the function of adjusting the sensitivity in order to provide consistent detection performance.

This sensitivity adjustment can adjust the sensitivity by adjusting the setting of the radiation temperature difference between the photographed images of the degraded image.

However, there is a limit to the sensitivity control in scattered reflections such as a light source emitting from a nearby building, a light source emitting from a signboard attached to a neighboring building, and particularly in a city where extreme diffuse reflection of the front and rear vehicle lights is severe.

In addition, the FIR camera is an expensive device, and thus, the user can not satisfy the price detection performance demanded by the user in a city with severe diffuse reflection.

Of course, it is also possible to detect pedestrians using CMOS cameras, which are relatively inexpensive compared to FIR cameras. However, since the CMOS camera is not a method of acquiring an image of a subject in such a manner that infrared heat energy emitted from the subject is sensed, there is no sensitivity control function using the temperature difference of the subject.

In other words, it is impossible to detect nighttime pedestrians, especially in urban areas with extreme diffuse reflections. As with FIR cameras, it is not possible to clearly distinguish whether an object is a pedestrian or an animal.

In recent years, pedestrians (including those on a bicycle) occupy a large part of traffic accidents, and pedestrian accidents must be prevented in the sense that once an accident occurs, it leads to a big accident such as death.

Therefore, there are many studies on the protection of pedestrians (including cyclists).

Nevertheless, in the conventional pedestrian protection apparatus, there is still a problem that the background of surrounding buildings, surrounding objects, etc., which should be excluded in order to correctly recognize the pedestrian, can not be distinguished from the pedestrian accurately, In fact.

In addition, the conventional pedestrian detection apparatus using the FIR camera performs the pedestrian detection by setting the region where the temperature is high when it is applied to the detection of the pedestrian as the region of interest.

However, since the temperature of the pedestrian detection device using the FIR camera is relatively constant depending on the environment such as day / night, it is difficult to expect a consistent performance in a region where the temperature is high.

In addition, various pedestrian detection devices have been proposed in addition to a pedestrian detection device using an FIR camera, but they include sensors mounted in front of a vehicle such as a bumper of a vehicle or an eye of a fender.

Pedestrian detection devices incorporating such sensors can readily provide an output signal of an error so that the safety device may be improperly deployed when struck by a hitting such as a cone of a road or an animal.

Accordingly, it is an object of the present invention to solve the above-mentioned problems and to provide a system and a method for detecting a vehicle, a pedestrian, an animal, etc. ahead of the driver's sight range secured by a headlight during night driving, There is a need for a night vision method and system.

Korean Intellectual Property Office Registration No. 10-1410215 Korean Intellectual Property Office Registration No. 10-1284892

The present invention relates to a night vision method for detecting a forward vehicle, a pedestrian, and an animal which are out of sight of a driver secured by a headlight during nighttime driving and which can not be seen by the eyes of a driver, System.

More particularly, the present invention relates to a method and apparatus for converting a digital image signal into a digital image signal when a FIR (Far Infra-Red) camera installed in a vehicle senses thermal energy of an object, reduces the amount of computation using information obtained through radars and radars, The present invention relates to a method and system for night vision that analyzes, corrects, and compensates a signal through a specific algorithm (Algorithm), displays it on a display, or outputs an alarm so that a driver can know ahead of time.

The present invention also provides a method and apparatus for reducing the amount of computation in the generation of a region of interest (ROI) using information obtained through radar and radar, Vision method and system to the user.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory and are not intended to limit the invention to the precise form disclosed. It can be understood.

According to an aspect of the present invention, there is provided a night vision information display method including: a first step of acquiring thermal energy information for at least one object through a Far Infra-Red (FIR) camera; A second step of extracting first thermal energy information satisfying a predetermined condition among the thermal energy information; A third step of extracting a vertical edge from the thermal energy information; A fourth step of extracting second thermal energy information mapped to the extracted vertical edge of the first thermal energy information; A fifth step of extracting a vertical one end point and another end point of the object included in the second thermal energy information; A sixth step of generating a region of interest (ROI) using the one end and the other end; A seventh step of comparing a plurality of classifiers related to a first object to be identified with an object included in the ROI; An eighth step of extracting an ROI corresponding to at least one of the plurality of classifiers in the ROI; And displaying the extracted ROI through a display unit. The method of claim 1, further comprising, during the first through fifth steps, displaying the extracted ROI through at least one of light detection and ranging (LiDAR) and radio detecting and ranging (RADAR) The method may further include acquiring additional information about at least one object. In the sixth step, ROI may be generated using the additional information and the one end and the other end together.

In the sixth step, the arrangement of the at least one object is confirmed using the additional information, and only one end and the other end point corresponding to the identified arrangement among the one and the other end points extracted in the fifth step are used The ROI can be generated.

In addition, the LiDAR obtains information about the at least one object located within a preset distance from the FIR camera, and the RADAR is for the at least one object located outside the predetermined distance from the FIR camera Information can be obtained.

In addition, the predetermined condition may be a condition having a brightness value equal to or greater than a predetermined value.

In addition, the predetermined condition may include a first condition and a second condition, the brightness value of the first condition being greater than the brightness value of the second condition, and the fourth step may include a first condition And the fifth step is performed using the first thermal energy information satisfying the first condition and the second energy information.

In addition, the second step and the third step may be performed simultaneously.

Between the eighth step and the ninth step, an object included in the extracted ROI is tracked using at least one of the one end and the other end; And in the ninth step, information to be tracked in the eighth step may be displayed.

The first through ninth steps are repeatedly performed, and the sixth step repeatedly performs the vanishing operation on the vertical axis of the object included in the ROI extracted from the eighth step, 6-1 < / RTI > (6-2) setting a vanishing region based on the set vanishing line; 6-3) comparing the vanishing area statistical information related to the first object with the vanishing area; And (6) if the difference between the vanishing region statistical information and the vanishing region is within a predetermined range, generating the ROI using the one end and the other end point; . ≪ / RTI >

In addition, the first through ninth steps are repeatedly performed, and the eighth step repeatedly performs a step of performing a vanishing operation on the vertical axis of the object included in the ROI extracted from the eighth step, 8-2 < / RTI > 8-3) setting a vanishing region on the basis of the set vanishing line; Comparing the vanishing area statistical information associated with the first object and the vanishing area; And extracting an ROI corresponding to at least one of the plurality of classifiers from the ROI when the difference between the vanishing region statistical information and the vanishing region is within a predetermined range, .

In addition, the first object includes a vehicle ahead, a pedestrian and an animal, and in the first step, the thermal energy information can be obtained at night.

According to another aspect of the present invention, there is provided an apparatus for displaying night vision information, comprising: a Far Infra-Red (FIR) camera for acquiring thermal energy information on at least one object; Light detection and ranging (LiDAR) and Radio Detecting And Ranging (RADAR) to obtain additional information about the at least one object; Extracting first thermal energy information satisfying a preset condition among the thermal energy information, extracting a vertical edge from the thermal energy information, mapping the extracted first thermal energy information to the extracted " vertical edge " Extracts a vertical end point and an end point of the object included in the second thermal energy information, generates an ROI using the additional information, the end point and the end point together, A control unit for comparing a plurality of classifiers related to one object with an object included in the ROI and extracting an ROI corresponding to at least one of the plurality of classifiers in the ROI; And a display unit displaying the extracted ROI.

The present invention relates to a night vision method for detecting a forward vehicle, a pedestrian, and an animal which are out of sight of a driver secured by a headlight during nighttime driving and which can not be seen by the eyes of a driver, System can be provided.

More particularly, the present invention relates to a method and apparatus for converting a digital image signal into a digital image signal when a FIR (Far Infra-Red) camera installed in a vehicle senses thermal energy of an object, reduces the amount of computation using information obtained through radars and radars, A night vision method and system can be provided to the user to analyze, correct and supplement the signal through a specific algorithm (Algorithm), display it on a display or output an alarm so that the driver can know ahead of time.

The present invention also provides a method and apparatus for reducing the amount of computation in the generation of a region of interest (ROI) using information obtained through radar and radar, Vision method and system to the user.

It should be understood, however, that the effects obtained by the present invention are not limited to the above-mentioned effects, and other effects not mentioned may be clearly understood by those skilled in the art from the following description It will be possible.

FIGS. 1A and 1B are diagrams for comparing night vision distances of a night vision system and a headlamp system according to the present invention. FIG.
Figs. 2A and 2B are views showing a concrete state in which a vehicle is equipped with a night vision system, in the context of the present invention. Fig.
FIG. 3 is a block diagram of a night vision system proposed by the present invention.
FIG. 4 illustrates an exemplary block diagram of a night vision system according to an embodiment of the present invention. Referring to FIG.
FIG. 5 is a flowchart illustrating a detailed process of displaying information according to a specific algorithm by detecting thermal energy of an FIR camera through a night vision system using a radar and a radar according to the present invention.
6A to 6E are diagrams for explaining a specific process of generating an ROI through an input image among the processes described with reference to FIG.
FIG. 7 is a view for explaining a concrete process of classifying a pedestrian according to machine learning among the processes described in FIG. 5. FIG.
8A and 8B are diagrams for explaining the procedure of the false recognition algorithm using the multi-classifier among the processes described in FIG.
FIGS. 9A and 9B are views for specifically explaining the HeadPoint-based Tracking process among the processes described in FIG.
10 is a flowchart for explaining another example in which the FIR camera senses thermal energy of an object and displays information according to a specific algorithm through a night vision system using a radar and a radar according to the present invention.
FIG. 11 is a view showing a specific situation in which the driver can easily check the surrounding situation through the night vision system according to the present invention in a situation where the field of view such as fog and dust is limited.
12 is a flowchart illustrating another process of displaying information according to a specific algorithm by detecting thermal energy of an FIR camera through a night vision system using a radar and a radar according to the present invention.

Generally, in a vehicle, a navigation system (navigation system) is used to confirm route information to a desired destination, and navigates to a destination conveniently according to route information that is navigated through navigation.

Such a navigation system provides information such as route guidance, television viewing, vehicle speed information display, etc., but it is impossible to detect a pedestrian or an animal while driving on the stern and warn it.

During the driving of the vehicle, the driver visually confirms the appearance of pedestrians or animals, and controls the vehicle operation based on the sensed information detected by the naked eye.

In the daytime, it is possible to identify most pedestrians and animals with the naked eye, but human eyes can not distinguish pedestrians or animals at night with low illumination.

Therefore, it is difficult to identify a pedestrian or an animal at a low illuminance at night, and an accident occurs frequently in which a vehicle collides with a pedestrian or an animal.

Recently, pedestrian sensing devices using FIR cameras have been applied to vehicles.

However, the conventional pedestrian sensing apparatus using the FIR camera has a problem in that scattered reflections of a light source emitted from a neighboring building and a light emitted from a signboard attached to a neighboring building, and particularly, There is a limit to the sensitivity control.

In addition, the FIR camera is an expensive apparatus, and does not satisfy the price detection performance required by the user in a city with severe diffuse reflection as described above.

In other words, it is impossible to detect nighttime pedestrians, especially in urban areas with extreme diffuse reflections. As with FIR cameras, it is not possible to clearly distinguish whether the object is a pedestrian or an animal.

Further, in the conventional pedestrian protection apparatus using the FIR camera, there is still a problem that the background of surrounding buildings, surrounding objects, etc., which should be excluded in order to correctly recognize the pedestrian, can not be distinguished from the pedestrian accurately. I can not.

In addition, the conventional pedestrian detection apparatus using the FIR camera has a problem that it is difficult to expect consistent performance in a region where the temperature is high, because the temperature is relatively constant depending on the environment such as night and day.

Accordingly, the present invention provides a night vision method and system for solving the above problems.

The night vision system (NVS) proposed by the present invention detects a vehicle, a pedestrian, and an animal ahead of the driver's sight beyond the visual range of the driver secured by the headlight during night driving, It is a driver assistance system for providing more forward image information.

FIGS. 1A and 1B are diagrams for comparing night vision distances of a night vision system and a headlamp system according to the present invention. FIG.

Referring to FIG. 1A, the night vision system according to the present invention is capable of securing a field of vision three to five times wider than that when the downward light of a general head lamp is turned on.

In addition, referring to FIG. 1B, the distance that can be visually recognized by irradiating the headlight is limited to within 60m, while the recognition of a forward pedestrian through the night vision system according to the present invention is possible up to 130m, which is more effective in securing a forward view.

Figs. 2A and 2B are diagrams showing concrete details of mounting a night vision system on a vehicle in accordance with the present invention. Fig.

As shown in FIG. 2A, when an FIR (Far Infra-Red) camera installed inside the vehicle grill senses thermal energy of an object, the integrated ECU converts the digital image signal.

Thereafter, the converted video signal is analyzed, corrected and supplemented through an image algorithm, and the processed video signal is displayed on the display or a warning sound is output, thereby allowing the driver to know the forward situation in advance.

FIG. 2B shows a specific view of a pedestrian detected through a Far Infra-Red (FIR) camera through a display unit.

Before describing the specific operation of the present invention, the configuration of the night vision system proposed by the present invention will be described in detail.

FIG. 3 is a block diagram of a night vision system proposed by the present invention.

Referring to FIG. 3, the night vision system 100 proposed by the present invention may include a far infrared ray camera 122, a control unit 180, and a display unit 151.

The far infrared ray camera 122 is a camera that photographs at least one object using Far Infrared (Far Infrared, FIR).

Far infrared rays usually mean infrared rays having a wavelength of 8 탆 or more, and are longer than wavelengths of visible light, so they are invisible to the naked eye, have a large thermal effect, and have high penetration.

Characteristics of the far-infrared camera 122 according to the present invention will be described.

Since a camera using a general CCD or a CMOS device serves to sense and project light in a visible light region, it is possible to acquire an image similar to that of a human eye.

On the other hand, the far-infrared camera 122 projects the light in the infrared band that is not seen by a person.

Infrared light refers to light in the wavelength range of 750 nm to 1 mm among the wavelengths of light. Among these infrared bands, the light of NIR (near infra-red) refers to the wavelength of 700 to 1400 nm and the light of NIR band is not visible to human eyes It can be detected by CCD or CMOS device, and it can detect only NIR band light by using filter.

On the other hand, the light of the FIR is also referred to as LWIR (Long Wavelength Infra-Red), and the infrared ray represents a band of 8 μm to 15 μm in the wavelength of light.

In particular, the FIR band has the advantage of being able to distinguish the temperature because the wavelength varies according to the temperature.

The body temperature of the person (pedestrian) which is the object of the far-infrared camera 122 has a wavelength of 10 mu m.

In addition, the control unit 180 controls the overall operation of the night vision system 100.

Particularly, the information obtained through the far infrared ray camera 122 is converted into a digital image signal, the converted image signal is analyzed, corrected and supplemented through a specific algorithm, and the processed image signal is displayed on the display unit 151 ).

The control unit 180 may be implemented in a recording medium that can be read by a computer or similar device using software, hardware, or a combination thereof.

According to a hardware implementation, the control unit 180 may be implemented as an application specific integrated circuit (ASIC), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays processors, controllers, micro-controllers, microprocessors, and other electronic units for performing other functions.

According to the software implementation, the control unit 180 may be implemented with separate software modules. Each of the software modules may perform one or more of the functions and operations described herein. Software code can be implemented in a software application written in a suitable programming language. The software codes may be stored in the memory 160, which will be described later.

Also, the display unit 151 provides a function of outputting the degree processed by the control unit 180 to the outside.

Here, the display unit 151 may be a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED) a flexible display, and a 3D display.

Hereinafter, with reference to FIG. 4, a specific configuration of the night vision system 100 proposed by the present invention described in FIG. 3 will be described.

4, the night vision system 100 includes a wireless communication unit 110, an A / V input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, A memory 160, an interface unit 170, a control unit 180, a power supply unit 190, and the like.

However, the components shown in Fig. 4 are not essential, so that a night vision system having components with fewer components or fewer components may be implemented.

Hereinafter, the components will be described in order.

The wireless communication unit 110 may include one or more modules that enable wireless communication between the night vision system and the wireless communication system or between the device and the network in which the device is located.

For example, the wireless communication unit 110 may include a mobile communication module 112, a wireless Internet module 113, a short distance communication module 114, a location information module 115, and the like.

The mobile communication module 112 transmits and receives a radio signal to at least one of a base station, an external device, and a server on a mobile communication network.

And various types of data according to transmission / reception of text / multimedia messages.

The wireless Internet module 113 refers to a module for wireless Internet access, and may be embedded in a night vision system or externally. WLAN (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access) and the like can be used as wireless Internet technologies.

The short-range communication module 114 refers to a module for short-range communication. Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IRDA), Ultra-WideBand (UWB), ZigBee, Wireless Fidelity, Wi-Fi Can be used.

The position information module 115 is a module for obtaining the position of the night vision system, and a representative example thereof is a Global Position System (GPS) module.

Referring to FIG. 4, an A / V (Audio / Video) input unit 120 is for inputting an audio signal or a video signal, and may include a camera 121 and a microphone 122. The camera 121 processes an image frame such as a still image or a moving image obtained by the image sensor in the photographing mode. The processed image frame can be displayed on the display unit 151. [

The image frame processed by the camera 121 may be stored in the memory 160 or transmitted to the outside through the wireless communication unit 110. [ Two or more cameras 121 may be provided depending on the use environment.

The microphone 122 receives an external sound signal by a microphone in a recording mode, a voice recognition mode, or the like, and processes it as electrical voice data. The processed voice data can be converted into a form that can be transmitted to the mobile communication base station through the mobile communication module 112 and output. Various noise reduction algorithms may be implemented in the microphone 122 to remove noise generated in receiving an external sound signal.

Next, the far-infrared camera 122 projects the infrared ray of the band that the person can not see.

The light of the FIR is also referred to as LWIR (Long Wavelength Infra Red). Infrared light has a wavelength of 8 μm to 15 μm in the wavelength of the light. In the FIR band, the temperature can be distinguished because the wavelength varies with temperature.

The body temperature of the person (pedestrian) which is the object of the far infrared ray camera 122 has a wavelength of 10 mu m and it is possible to photograph images, moving images, and the like of a specific object even at night through the far infrared ray camera 122. [

Next, the user input unit 130 generates input data for controlling the operation of the night vision system by the user. The user input unit 130 may include a key pad dome switch, a touch pad (static / static), a jog wheel, a jog switch, and the like.

The sensing unit 140 senses the current state of the night vision system, such as the open / close state of the night vision system, the position of the night vision system, the presence or absence of user contact, the orientation of the night vision system, And generates a sensing signal for controlling the operation of the sensor.

The sensing unit 140 may sense whether the power supply unit 190 is powered on, whether the interface unit 170 is connected to an external device, and the like.

In addition, the sensing unit 140 according to the present invention may include LiDAR (light detection and ranging) 141.

Here LiDAR (141) is also called rider after the initials of light detection and ranging.

In principle, it is a distance measuring device up to a target, in which a microwave is replaced by a laser beam in a conventional radar.

By using laser light, the distance measurement accuracy, azimuth resolution, SN ratio, and the like are improved as compared with the radar 142.

Since the wavelength is short, fine particles in the air can be detected.

The laser Raman radar detects specific molecules using Raman scattering and is used to monitor air and marine pollution.

In the present invention, the LiDAR 141 recognizes an object located in front of a vehicle or the like, and at the same time, the control unit 180 greatly reduces the amount of computation in generating ROI (Region of Interest) It can be made known in advance.

In addition, the sensing unit 140 according to the present invention may include a radio detecting and ranging (RADAR)

The RADAR 142 is an abbreviation of Radio Detection and Ranging, which emits an electromagnetic wave of a microwave (microwave, 10 cm to 100 cm wavelength) to an object, receives an electromagnetic wave reflected from the object, Direction, altitude, and so on.

Since the microwave used in the RADAR 142 has a long wavelength and has a straightness like light and is not reflected by the ionosphere, the radio wave emitted from the directional antenna travels straight to the target and then reflects back.

At this time, it is possible to find the distance, direction and altitude of the target by measuring the time of the reflected electromagnetic waves. Through this information, the position of the target, the topography, and the formation of the cloud are found.

By using the radar 142, it is possible to measure the distance more precisely than the optical method, and it is possible to study the surface state of the satellite.

Because the radar 142 can detect cloud droplets, ice crystals, raindrops, hail, etc., meteorology can also detect and track storms hundreds of kilometers away using ground or radar information.

As the circuits and auxiliary devices of the radar 142 are miniaturized and portable, it is possible to use a radar gun or an optical radar sensor device for speed sensing used by the police, Technology.

In the present invention, the RADAR 142 recognizes an object located in front of the vehicle or the like, and at the same time, the control unit 180 greatly reduces the amount of computation in generating ROI (Region of Interest), and transmits the forward information having higher accuracy to the driver It can be made known in advance.

The output unit 150 generates an output related to a visual, auditory or tactile sense and includes a display unit 151, an audio output module 152, an alarm unit 153, a haptic module 154, Module 155, a head-up display (HUD), a head mounted display (HMD), and the like.

The display unit 151 displays (outputs) information processed in the night vision system.

The display unit 151 may be a liquid crystal display (LCD), a thin film transistor-liquid crystal display (TFT LCD), an organic light-emitting diode (OLED), a flexible display display, and a 3D display.

Some of these displays may be transparent or light transmissive so that they can be seen through. This can be referred to as a transparent display, and a typical example of the transparent display is TOLED (Transparent OLED) and the like. The rear structure of the display unit 151 may also be of a light transmission type. With this structure, the user can see an object located behind the night vision system body through the area occupied by the display unit 151 of the night vision system body.

There may be two or more display units 151 depending on the implementation of the night vision system. For example, in a night vision system, a plurality of display portions may be spaced apart from one another, or may be disposed integrally with each other, or may be disposed on different surfaces, respectively.

(Hereinafter, referred to as a 'touch screen') in which a display unit 151 and a sensor for sensing a touch operation (hereinafter, referred to as 'touch sensor') form a mutual layer structure, It can also be used as an input device. The touch sensor may have the form of, for example, a touch film, a touch sheet, a touch pad, or the like.

The touch sensor may be configured to convert a change in a pressure applied to a specific portion of the display unit 151 or a capacitance generated in a specific portion of the display unit 151 into an electrical input signal. The touch sensor can be configured to detect not only the position and area to be touched but also the pressure at the time of touch.

If there is a touch input to the touch sensor, the corresponding signal (s) is sent to the touch controller. The touch controller processes the signal (s) and transmits the corresponding data to the controller 180. Thus, the control unit 180 can know which area of the display unit 151 is touched or the like.

The proximity sensor 141 may be disposed within an interior region of the night vision system that is enclosed by the touch screen or near the touch screen. The proximity sensor refers to a sensor that detects the presence or absence of an object approaching a predetermined detection surface or a nearby object without mechanical contact using the force of an electromagnetic field or infrared rays. The proximity sensor has a longer life span than the contact sensor and its utilization is also high.

Examples of the proximity sensor include a transmission type photoelectric sensor, a direct reflection type photoelectric sensor, a mirror reflection type photoelectric sensor, a high frequency oscillation type proximity sensor, a capacitive proximity sensor, a magnetic proximity sensor, and an infrared proximity sensor. And to detect the proximity of the pointer by the change of the electric field along the proximity of the pointer when the touch screen is electrostatic. In this case, the touch screen (touch sensor) may be classified as a proximity sensor.

Hereinafter, for convenience of explanation, the act of recognizing that the pointer is positioned on the touch screen while the pointer is not in contact with the touch screen is referred to as "proximity touch & The act of actually touching the pointer on the screen is called "contact touch. &Quot; The position where the pointer is proximately touched on the touch screen means a position where the pointer is vertically corresponding to the touch screen when the pointer is touched.

The proximity sensor detects a proximity touch and a proximity touch pattern (e.g., a proximity touch distance, a proximity touch direction, a proximity touch speed, a proximity touch time, a proximity touch position, a proximity touch movement state, and the like). Information corresponding to the detected proximity touch operation and the proximity touch pattern may be output on the touch screen.

The audio output module 152 may output audio data received from the wireless communication unit 110 or stored in the memory 160 in a recording mode, a voice recognition mode, a broadcast receiving mode, or the like. The sound output module 152 also outputs sound signals related to functions performed in the night vision system. The audio output module 152 may include a receiver, a speaker, a buzzer, and the like.

The alarm unit 153 outputs a signal for notifying the occurrence of an event of the night vision system.

The alarm unit 153 may output a signal for notifying the occurrence of an event in a form other than the video signal or the audio signal, for example, vibration.

The video signal or the audio signal may be output through the display unit 151 or the audio output module 152 so that they may be classified as a part of the alarm unit 153.

The haptic module 154 generates various tactile effects that the user can feel. A typical example of the haptic effect generated by the haptic module 154 is vibration. The intensity and pattern of the vibration generated by the hit module 154 can be controlled.

For example, different vibrations may be synthesized and output or sequentially output.

In addition to the vibration, the haptic module 154 may include a pin arrangement vertically moving with respect to the contact skin surface, a spraying force or suction force of the air through the injection port or the suction port, a touch on the skin surface, contact of an electrode, And various tactile effects such as an effect of reproducing a cold sensation using an endothermic or exothermic element can be generated.

The haptic module 154 can be implemented not only to transmit the tactile effect through the direct contact but also to allow the user to feel the tactile effect through the muscular sensation of the finger or arm. The haptic module 154 may include two or more haptic modules 154 according to the configuration of the night vision system.

The projector module 155 is a component for performing an image project function using the night vision system and is configured to have the same or at least the same image displayed on the display unit 151 in accordance with a control signal of the controller 180 Some can display other images on an external screen or wall.

Specifically, the projector module 155 includes a light source (not shown) that generates light (for example, laser light) for outputting an image to the outside, a light source And a lens (not shown) for enlarging and outputting the image at a predetermined focal distance to the outside. Further, the projector module 155 may include a device (not shown) capable of mechanically moving the lens or the entire module to adjust the image projection direction.

The projector module 155 can be divided into a CRT (Cathode Ray Tube) module, an LCD (Liquid Crystal Display) module and a DLP (Digital Light Processing) module according to the type of the display means. In particular, the DLP module may be advantageous for miniaturization of the projector module 151 by enlarging and projecting an image generated by reflecting light generated from a light source on a DMD (Digital Micromirror Device) chip.

Preferably, the projector module 155 may be provided longitudinally on the side, front, or back of the night vision system. Of course, it is needless to say that the projector module 155 may be provided at any position of the night vision system, if necessary.

In addition, the head-up display (HUD) 156 refers to a device for projecting the current vehicle speed, remaining fuel amount, navigation route information, and the like in a vehicle or the like as a graphic image on a window portion in front of the driver.

The information obtained through the far infrared camera 122 may also be output through the head-up display 156.

In addition, a head mounted display (HMD) 157 is a typical device capable of outputting virtual reality information.

Virtual reality is a process of creating a 3D environment in which a specific environment or situation is created through a computer and creating a human-computer interaction that makes the person using the 3D content appear as if they are interacting with the actual surroundings and environment. And the like.

Generally, the three-dimensional sensation perceived by a person depends on the degree of thickness change of the lens according to the position of the observed object, the angle difference between the eyes and the object, the position and shape difference of the object on the left and right eyes, , And various other psychological and memory effects.

Among them, binocular disparity is the most important factor for a person to feel the stereoscopic effect, as the eyes of a person are about 6.5 cm apart in the horizontal direction. In other words, the binocular parallax causes the angle of the object to be viewed with the difference. Due to this difference, the images coming into each eye have different phases. When these two images are transmitted to the brain through the retina, The 3D stereoscopic image of the original can be felt.

These stereoscopic 3D contents have already been widely used in various media fields and have been well received by consumers. For example, 3D movies, 3D games, and experience displays are examples.

As described above, in addition to the universalization of virtual reality technology 3D contents, there is a need to develop a technology capable of providing a more immersive virtual reality service.

2. Description of the Related Art Generally, an image display device forms a focal point so that a virtual large-sized screen can be formed at a long distance by using a precision optical device, which is generated in a position very close to an eye, so that a user can view an enlarged virtual image An image display device.

In addition, the image display device is configured to include a see-close type in which only the image light emitted from the display device can not be seen in the surrounding environment, and a see-close mode in which the image light emitted from the display device (See-through) that can be seen.

The head mounted display (HMD) 157 according to the present invention refers to various digital devices such as glasses that are worn on the head to receive multimedia contents. Various wearable computers (Wearable Computers) have been developed in accordance with the trend of weight reduction and miniaturization of digital devices, and HMDs are also widely used. The HMD 157 can be combined with augmented reality technology, N-screen technology, etc., beyond a simple display function, to provide various convenience to the user.

For example, when a microphone and a speaker are mounted on the HMD 157, the user can perform a telephone conversation while wearing the HMD 157. [ For example, when the far infrared ray camera 122 is mounted on the HMD 157, the user can capture an image of a desired direction in a state in which the HMD 157 is worn by the user.

In addition, the memory unit 160 may store a program for processing and controlling the control unit 180 and temporarily store the input / output data (e.g., message, audio, still image, For example. The frequency of use of each of the data may be stored in the memory unit 160 as well. In addition, the memory unit 160 may store data on vibration and sound of various patterns output when the touch is input on the touch screen.

The memory 160 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), a RAM (Random Access Memory), SRAM (Static Random Access Memory), ROM (Read Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM A disk, and / or an optical disk. The night vision system may operate in association with a web storage that performs the storage function of the memory 160 on the Internet.

The interface unit 170 serves as a path for communication with all external devices connected to the night vision system. The interface unit 170 receives data from an external device, supplies power to each component in the night vision system, or allows data in the night vision system to be transmitted to an external device. For example, a wired / wireless headset port, an external charger port, a wired / wireless data port, a memory card port, a port for connecting a device having an identification module, an audio I / O port, A video input / output (I / O) port, an earphone port, and the like may be included in the interface unit 170.

The identification module is a chip for storing various information for authenticating the usage right of the night vision system and includes a user identification module (UIM), a subscriber identity module (SIM), a universal user authentication module Identity Module, USIM), and the like. Devices with identification modules (hereinafter referred to as "identification devices") can be manufactured in a smart card format. Thus, the identification device can be connected to the night vision system through the port.

When the night vision system is connected to an external cradle, the interface unit may be a path through which power from the cradle is supplied to the night vision system or various command signals input from the cradle by the user are transmitted to the mobile device It can be a passage. The various command signals or the power source input from the cradle may be operated as a signal for recognizing that the mobile device is correctly mounted on the cradle.

The controller 180 typically controls the overall operation of the night vision system.

The power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power necessary for operation of the respective components.

The various embodiments described herein may be embodied in a recording medium readable by a computer or similar device using, for example, software, hardware, or a combination thereof.

According to a hardware implementation, the embodiments described herein may be implemented as application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays May be implemented using at least one of a processor, controllers, micro-controllers, microprocessors, and other electronic units for performing other functions. In some cases, The embodiments described may be implemented by the control unit 180 itself.

According to a software implementation, embodiments such as the procedures and functions described herein may be implemented with separate software modules. Each of the software modules may perform one or more of the functions and operations described herein. Software code can be implemented in a software application written in a suitable programming language. The software code is stored in the memory 160 and can be executed by the control unit 180. [

On the other hand, based on the above-described configuration of the present invention, the control unit 180 converts object-related contents (document, image, moving image, etc.) acquired by the far-infrared camera 122 through thermal energy into a digital image signal, The specific operation in which the signal is analyzed, corrected and supplemented through a specific algorithm (Algorithm), and the display 151 is controlled to display the processed content will be described.

First Method

FIG. 5 is a flowchart illustrating a detailed process of displaying information according to a specific algorithm by sensing thermal energy of an object through an FIR camera through a night vision system according to the present invention.

6A to 6E are diagrams for explaining a specific process of generating an ROI through an input image among the processes described with reference to FIG.

Referring to FIG. 5, first, an image of thermal energy of at least one object is obtained through a far infrared ray camera 122 (S100).

Referring to FIG. 6A, a detailed view of a thermal energy image of a plurality of objects is obtained through the far-infrared camera 122 of the present invention.

In operation S100, the control unit 180 extracts a threshold image having a brightness value greater than a predetermined value from the input image (S110).

Here, the threshold value (threshold value) can be statistically estimated using the average, standard deviation, histogram, etc. of the image, or can be set by the user to input an arbitrary value.

This is based on the fact that the brightness value of the ordinary person (pedestrian) is brighter than the surrounding background.

Referring to FIG. 6B, a certain area is photographed with a thermal energy image through a far-infrared camera 122, and a high threshold bright object having a brightness value of a preset value or more among the photographed images is shown.

However, if the obtained image characteristics of the pedestrian far away from the camera 122 outside the original are different from each other, and a plurality of pedestrians other than one pedestrian are adjacent to each other, or the weather is extremely cold or very hot, With the threshold, it can be difficult to shoot all the pedestrians without missing them.

Accordingly, unlike step S110, the step S120 of extracting all the object images which are brighter on the lower threshold by lowering the brightness value is performed in parallel.

Here, the threshold value (threshold value) can be statistically estimated using the average, standard deviation, histogram, etc. of the image, or can be set by the user to input an arbitrary value.

However, in step S120, unnecessary noise other than the targeted pedestrian may be frequently included and extracted.

Therefore, in order to solve such a problem, step S100 of extracting the input image based on an edge is performed together with step S100.

This is because, in the case of a person (pedestrian), the outer edge component of the vertical edge, that is, the edge outer edge component of the vertical edge, is strongly displayed in terms of body characteristics, so that unnecessary noise obtained in step S120 is removed using this edge.

The edge image can be estimated from the sobel filter, the canny edge detector, and the laplacian filter.

The control unit 180 extracts all the object images brightly rising on the low threshold in step S120 and multiplies the extracted images on the basis of the edge in step S130.

Therefore, even if the object is extracted as an object by passing a low threshold in step S120, the object having no vertical edge component disappears.

For example, when a pedestrian stands on the road, the pedestrian and the road are treated as one object and can be extracted as an image in step S120.

However, in step S130, since the road does not have a vertical edge component, only the image elements of the pedestrian located on the road remain.

In addition, multiplying S120 and S130 may be performed by performing a masking function to complement an object that has not been completely extracted, in addition to extracting an object having a vertical edge component.

That is, in step S120, the pedestrian may not be extracted as a single bright light image, but may be extracted as a broken shape.

At this time, by applying the extracted edge component in step S130 to step S120, masking that the broken space is filled can be performed.

Referring to FIG. 6C, it is possible to perform masking in which the broken space is filled by applying an element in which an intermediate shape is broken and an edge component extracted in step S130 is extracted to step S120.

The control unit may perform step S110, step S120 and step S130 in order, or may perform the respective steps in parallel at the same time.

In steps S110, S120, and S130, a labeling operation may be additionally performed, similar to the above-described masking operation.

Labeling work can be seen as a task of sticking closely spaced chunks or segments, which tie together objects that are separated by the same object or thermal energy failing to acquire clearly.

The step of extracting head point information (S140) is performed after steps S110 to S130.

In step S140, the image extracted in step S110 and the steps S120 and S130 are simultaneously applied to extract the head point of the object from the extracted image.

That is, the uppermost region where the bright portion ends of the extracted image can be determined as the head point region.

Head point extraction can be performed by skeletonization, local maximum extraction, head recognition through learning, and so on.

In addition, step S150 may be performed simultaneously with step S140 or sequentially extracting a foot line of the object in step S140.

In operation S150, the extracted image is extracted using only the edge components, but the edge map is extracted in operation S130.

In step S150, the image is ascended in the vertical direction at the lowermost end of the acquired image, and the point of meeting with the extracted images is found, and the point of meeting is determined as the foot line of the object.

Here, the foot line (toe) information extraction can be performed by performing dilation among inputted images and extracting based on the differential value in the vertical direction (for example, a position changing from 255 to 0).

The reason why the steps S120, S130, S140, and S150 are performed is to reduce the calculation throughput when the controller 180 acquires the thermal image.

Instead of finally detecting the pedestrian by confirming the objects existing on all images acquired here, the elements that can be clearly discriminated by noise through steps S120, S130, S140 and S150 are removed in advance in the first part of the work, .

As a result, the ROI generation method is not sensitive to changes in seasons or weather without missing a pedestrian for the pedestrian recognition rate of the thermal night vision system 100. [

Despite steps S120 through S150, unwanted noise may be included, which may be removed through steps S160 through S180.

In order to efficiently reduce the amount of computation in future generation of ROIs, the LiDAR 141 may include information that recognizes an object located in front of the vehicle or the like, and information in which the RADAR 142 is located in front of the vehicle (S400) may be added to the information of the object.

That is, when the thermal camera 123, the radar 142, and the LiDAR 141 are used separately, it is difficult to achieve 100% effect according to the object recognition. However, the thermal camera 123, the radar 142, The present invention is applied to a method capable of improving object recognition by using information acquired in the present invention.

More specifically, in the present invention, the localization is first confirmed in the image through the information obtained through the Radar 142 and the LiDAR 141 through step S400, and then the information acquired through the thermal camera 123 (Pedestrian, vehicle, animal) by analyzing the image based on the image.

That is, in this technique, the radar 142 is used for a long distance, the LiDAR 141 is used for a medium distance or a short distance, and the image of the thermal image obtained through the infrared camera 123 is analyzed, , Vehicles, and animals.

In addition, when the thermal camera 123 is used alone, a low resolution result can be obtained. However, when the Radar 142 and the LiDAR 141 are used together, accurate object recognition becomes possible even at a low resolution.

As a result, the distance information to the object is obtained through the Radar 142 or the LiDAR 141, and the recognition of the object is confirmed through the infrared camera 123, so that the risk factors such as the vehicle and the pedestrian can be judged It becomes.

5, the head point and foot line are detected through steps S140 and S150, and the LiDAR 141 and the RADAR 142 information are obtained through step S400. Region of Interest) (S160).

Referring to FIG. 6D, a specific example of detecting a head and a foot line of a specific object is shown.

The ROI area generated in step S160 may be determined as a box of a specific ratio (for example, 2: 1) based on the head candidate and the length of the foot candidate corresponding thereto.

Referring to FIG. 6E, the specific shape of the ROI is shown in a box-shaped form at a specific ratio based on the length of the head candidate and the corresponding foot line (toe candidate).

As a result, steps S110 to S160 may include obtaining an input thermal image, estimating at least one threshold based on the input image, generating a first divided image for dividing the image by the estimated threshold, Calculating an edge image for the input image, estimating a threshold value based on the calculated edge image, generating a second divided image for dividing the edge image by a threshold value estimated from the edge image, Generating a third divided image by performing a multiplication operation on the first divided image and the second divided image, labeling the third divided image, extracting a head candidate for each of the Label, Extracting a foot line (toe candidate) from the three-divided image, and combining the head candidate point and the foot line (toe candidate) to generate an ROI region.

In step S160, a vanishing line may be additionally used in generating the ROI.

The vanishing line may serve to reduce false positives in step S180 described later or may be used as a criterion for generating the ROI itself in step S160.

The vanishing line is an invisible virtual space line, and the center of the vertical axis of the pedestrian is used based on the assumption that it passes through the vanishing line regardless of the distance.

In other words, in order to solve the problem derived from the fact that the pedestrian size is different and the distance between the pedestrian and the camera is different, the center of the vertical axis of the pedestrian is assumed to pass the vanishing line regardless of the size and the distance.

If pedestrian recognition is performed from at least one frame of the input image obtained in step S100, the reference vanishing line can be estimated from the coordinates of the center of ordinate of the recognized pedestrian area in step S160.

Also, the control unit 180 may set the vanishing region based on the reference vanishing line and perform the pedestrian recognition of the current frame based on the vanishing region before generating the ROI.

At this time, the controller 180 loads the vanishing region statistics for the coordinates of the longitudinal axis of the pedestrians stored in the memory 160, and calculates the difference between the statistics of the vanishing region and the coordinates of the longitudinal axis of the recognized pedestrians.

If the difference value is larger than a predetermined threshold value based on the difference of the statistic, the controller 180 determines that the object is not an object for the pedestrian, and may not generate the ROI itself.

For example, the points of objects detected from previous pedestrians can be accumulated to continuously accumulate the statistics of the Gaussian distribution, and the accumulated information can be stored in the memory 160 as a probability model.

The ROI can be treated as a box that surrounds a person (pedestrian), and is created as an ROI only when the center of the box is 0.5 or less, which is a predetermined difference in relation to the probability model.

If the number of ROIs generated through the vanishing line is decreased in step S160, the process of distributing the ROI by the controller 180 can be effectively reduced.

In the ROI generation in step S160, a method of using the head point information extracted from the previous frame may also be applied.

That is, when extracted from the previous frame to the head point, the position of the head point in the next frame can be predicted by the control unit 180. [

Therefore, in the next frame, it is also possible to reduce the calculation process used for ROI generation by using the existing information without recognizing the specific region corresponding to the expected head point point as a head point and using the rest as the head point.

When step S160 is performed, N ROIs can be generated.

The control unit 180 collects the generated N ROIs for parallel processing, and performs the pedestrian classification according to the machine learning.

That is, the controller 180 stores a plurality of classifiers corresponding to the pedestrians in advance in the memory 160, and stores a plurality of classifiers corresponding to the plurality of stored classifiers of the pedestrian candidate objects included in the N ROIs Only the object image is extracted.

FIG. 7 is a view for explaining a concrete process of classifying a pedestrian according to machine learning among the processes described in FIG. 5. FIG.

Referring to FIG. 7, the training data set stored in the memory 160 is loaded by the controller 180, and the controller 180 stores a plurality of classifiers loaded with the pedestrian candidate image included in the N ROIs And only the corresponding objects are extracted.

In step S170, a plurality of classifiers having different criteria may be used to remove false positives.

8A and 8B are diagrams for explaining the procedure of the false recognition algorithm using the multi-classifier among the processes described in FIG.

Referring to FIGS. 8A and 8B, when different classifiers are applied, it can be seen that the aspects of false positives are different from each other.

8A and 8B, the red box is a pedestrian estimated by the LBP-Cascade method, the blue box is a pedestrian estimated by the Haar-Cascade method, and the green box represents a pedestrian estimated by LBP-Haar at the same time.

Therefore, it is possible to complement the missing parts by applying a plurality of different classifiers.

After step S170, the controller 180 performs a head-point-based tracking step (S180).

In step S180, the controller 180 may use a particle filter, a mean shift, a Kalman filter, and the like.

FIGS. 9A and 9B are views for specifically explaining the HeadPoint-based Tracking process among the processes described in FIG.

FIG. 9A is a flowchart illustrating a head point-based tracking step (S180). The controller 180 first extracts a motion vector (S181).

Thereafter, the control unit 180 performs a step S182 of comparing the histogram with respect to the tracking candidate.

After step S182, the control unit 180 determines Best Matching (S183), and finally determines whether or not it is tracking (S184).

FIG. 9B shows a concrete state in which the control unit 180 performs tracking based on the head point.

With this headpoint basis tracking, the amount of calculation performed by the control unit 180 can be drastically reduced.

The tracking in step S180 may be for achieving two purposes.

First, tracking is used to display images displayed through the display unit 151 without interruption.

That is, when the pedestrian recognition through the controller 180 can not be continuously performed, the pedestrian may appear in a specific frame, and may not appear in the different frame and flicker may occur.

Therefore, even if the pedestrian is not recognized in the specific frame through tracking in step S180, the contents of the pedestrian in the previous frame are tracked and displayed on the display unit 151, so that a smooth image can be realized.

In this case, if the object is not a desired object, it exists only in a specific frame, and does not appear continuously in different frames.

Therefore, such an object is treated as an error through tracking, and is not displayed on the display unit 151.

The pedestrian, which is a desired object, may not be sensed in a few frames but is sensed in most frames, so that the control unit 180 can continuously control the object to be displayed on the display unit 151 through tracking have.

Next, the purpose of step S180 is to remove the cognitive impairment.

As in step S160, a vanishing line can also be used to remove false positives in step S180.

As described above, the vanishing line is an invisible virtual space line, and the center of the vertical axis of the pedestrian is used based on the assumption that it passes through the vanishing line regardless of the distance. In other words, the pedestrian's center of the vertical axis is assumed to pass through the vanishing line regardless of its size and distance in order to solve the problem that the pedestrian size is different and the distance between the pedestrian and the camera is different.

When the pedestrian recognition is performed from the obtained input image of at least one frame, in step S180, the reference vanishing line can be estimated from the coordinates of the vertical axis of the recognized pedestrian area.

In addition, the controller 180 may set the vanishing region based on the reference vanishing line and perform the pedestrian recognition of the current frame based on the vanishing region.

At this time, the controller 180 loads the vanishing region statistics for the coordinates of the longitudinal axis of the pedestrians stored in the memory 160, and calculates the difference between the statistics of the vanishing region and the coordinates of the longitudinal axis of the recognized pedestrians.

If the difference value is larger than a predetermined threshold value based on the difference of the statistical values, the controller 180 determines that the object is not an object for a pedestrian, treats it as false, and discards the object information.

The foot line information obtained in step S150 may be used for the purpose of eliminating the observer in step S180.

That is, if the foot line is held based on the box recognized as the ROI, and if the pedestrian is not the pedestrian, the shape of the foot line changes greatly with respect to the box, so that the ROI can be treated as false and can be removed.

Thereafter, the control unit 180 may control the finally extracted thermal image information to be outputted through the display unit 151 (S190).

Second Method

In addition to the method of FIG. 5, a method of extracting thermal image information for a pedestrian may be applied in a different order.

10 is a flowchart for explaining another example in which the FIR camera senses thermal energy of an object through a night vision system according to the present invention and displays information according to a specific algorithm.

In FIG. 10, steps S200 and S240 to S280 correspond to steps S100, S150, and S190 in FIG. 5, respectively, and therefore, a repetitive description thereof will be omitted for simplicity.

The method in FIG. 10 differs from the method in FIG. 5 in steps S210 to S230.

Referring to FIG. 10, in contrast to the method shown in FIG. 5, when the step S200 is performed, the controller 180 does not distinguish between a high threshold and a low threshold but only one threshold And extracts an image using the image (S210).

Here, the threshold value (threshold value) can be statistically estimated using the average, standard deviation, histogram, etc. of the image, or can be set by the user to input an arbitrary value.

This is based on the fact that the brightness value of the ordinary person (pedestrian) is brighter than the surrounding background.

At this time, in step S210, unnecessary noise other than the targeted pedestrian may be frequently included and extracted.

Accordingly, in order to solve this problem, step S220 is performed in which the input image is extracted on the basis of an edge.

This is because, in the case of a person (pedestrian), the outer edge component of the vertical edge, that is, the edge edge component of the vertical edge, is strongly displayed in terms of body characteristics, and thus unnecessary noise obtained in step S210 is removed by using this.

The edge image can be estimated from the sobel filter, the canny edge detector, and the laplacian filter.

The control unit 180 extracts all the object images brighter on the threshold in step S210 and multiplies the extracted images on the basis of the edge in step S220.

Therefore, even though the object is extracted by passing the threshold in step S210, the object having no vertical edge component disappears.

The step S210 and the step S220 may be performed by extracting an object having a vertical edge component and performing a masking function to complement an object that has not been completely extracted.

That is, in step S210, the pedestrian may not be extracted as a single bright light image, but may be extracted as a broken shape.

At this time, by applying the extracted element of the edge component in step S220 to step S210, the masking in which the broken space is filled can be performed.

The control unit may perform steps S210 and S220 in sequence, or may perform the respective steps in parallel at the same time.

In steps S210 and S220, a labeling operation may be additionally performed similarly to the above-described masking operation.

Labeling work can be seen as a task of sticking closely spaced chunks or segments, which tie together objects that are separated by the same object or thermal energy failing to acquire clearly.

After steps S210 and S220, the head point information is extracted through the multiplied information (S230).

In step S230, the image extracted in step S210 and the step S220 are simultaneously applied to extract the head point of the object from the extracted image.

That is, the uppermost region where the bright portion ends of the extracted image can be determined as the head point region.

Head point extraction can be performed by skeletonization, local maximum extraction, head recognition through learning, and so on.

In order to efficiently reduce the amount of computation in future generation of ROIs, the LiDAR 141 may include information that recognizes an object located in front of the vehicle or the like, and information in which the RADAR 142 is located in front of the vehicle (S400) may be added to the information of the object.

That is, when the thermal camera 123, the radar 142, and the LiDAR 141 are used separately, it is difficult to achieve 100% effect according to the object recognition. However, the thermal camera 123, the radar 142, The present invention is applied to a method capable of improving object recognition by using information acquired in the present invention.

More specifically, in the present invention, the localization is first confirmed in the image through the information obtained through the Radar 142 and the LiDAR 141 through step S400, and then the information acquired through the thermal camera 123 (Pedestrian, vehicle, animal) by analyzing the image based on the image.

That is, in this technique, the radar 142 is used for a long distance, the LiDAR 141 is used for a medium distance or a short distance, and the image of the thermal image obtained through the infrared camera 123 is analyzed, , Vehicles, and animals.

In addition, when the thermal camera 123 is used alone, a low resolution result can be obtained. However, when the Radar 142 and the LiDAR 141 are used together, accurate object recognition becomes possible even at a low resolution.

As a result, the distance information to the object is obtained through the Radar 142 or the LiDAR 141, and the recognition of the object is confirmed through the infrared camera 123, so that the risk factors such as the vehicle and the pedestrian can be judged It becomes.

The process returns to FIG. 10, and then steps S240 to S280 are performed corresponding to steps S150 to S190 in FIG.

11 shows a result of applying the night vision system according to the above-described algorithm.

11 (a) shows a specific image in which a pedestrian is not seen due to fog, which is an image obtained through a general camera 121

On the other hand, the image of the far-infrared camera 123 according to the algorithm proposed by the present invention recognizes four pedestrians and can guide the recognized result to the driver.

12 is a flowchart illustrating another process of displaying information according to a specific algorithm by sensing thermal energy of an object through an FIR camera through a night vision system according to the present invention.

Steps S300 to S390 in FIG. 12 correspond to steps S100 to S190 in FIG. 5, respectively, but differ in steps S310, S320, and S340.

Referring to FIG. 12, the step S310 may be unnecessarily included in the target pedestrian other than the step S330.

Therefore, in order to solve this problem in step S310 in the flowchart of FIG. 12, step S320 of extracting the input image based on an edge in step S300 may be performed together.

This is because, in the case of a person (pedestrian), the outer edge component of the vertical edge, that is, the edge outer edge component of the vertical edge, is strongly displayed in terms of body characteristics, and thus unnecessary noise obtained in step S310 is removed using the extracted edge component.

Here, the edge image can be estimated from the sobel filter, the canny edge detector, and the laplacian filter.

The control unit 180 extracts all the object images brightly rising on the high threshold in step S310 and multiplies the extracted images on the basis of the edge in step S320.

Therefore, in step S310, an object having no vertical edge component disappears even if it is extracted as an object by passing a high threshold value.

Multiplication of steps S310 and S320 may be performed not only for extracting an object having a vertical edge component but also for masking an object that has not been completely extracted.

That is, in step S310, the pedestrian may not be extracted as a single bright light image, but may be extracted as a broken shape.

At this time, by applying the extracted element of the edge component in step S320 to step S310, masking that the broken space is filled can be performed.

In the embodiment of FIG. 12, in order to efficiently reduce the amount of calculation in the generation of ROI (region of interest), in the present invention, LiDAR 141 transmits information recognizing an object located in front of the vehicle, (S400) of acquiring information that recognizes an object located in front of the vehicle or the like may be added.

That is, when the thermal camera 123, the radar 142, and the LiDAR 141 are used separately, it is difficult to achieve 100% effect according to the object recognition. However, the thermal camera 123, the radar 142, The present invention is applied to a method capable of improving object recognition by using information acquired in the present invention.

More specifically, in the present invention, the localization is first confirmed in the image through the information obtained through the Radar 142 and the LiDAR 141 through step S400, and then the information acquired through the thermal camera 123 (Pedestrian, vehicle, animal) by analyzing the image based on the image.

That is, in this technique, the radar 142 is used for a long distance, the LiDAR 141 is used for a medium distance or a short distance, and the image of the thermal image obtained through the infrared camera 123 is analyzed, , Vehicles, and animals.

In addition, when the thermal camera 123 is used alone, a low resolution result can be obtained. However, when the Radar 142 and the LiDAR 141 are used together, accurate object recognition becomes possible even at a low resolution.

As a result, the distance information to the object is obtained through the Radar 142 or the LiDAR 141, and the recognition of the object is confirmed through the infrared camera 123, so that the risk factors such as the vehicle and the pedestrian can be judged It becomes.

In operation S360, a region of interest (ROI) may be generated through a combination of the detected head point and the foot line and the LiDAR 141 and RADAR 142 information obtained in operation S400.

When the structure and method of the present invention described above are applied, it is possible to detect a forward vehicle, a pedestrian, and an animal that are out of sight of the driver and can not be seen by the driver's eyes, A night vision method and system for providing image information can be provided to the user.

That is, according to the present invention, when a Far Infra-Red (FIR) camera installed in a vehicle senses heat energy of an object, it converts it into a digital image signal, analyzes the converted image signal through a specific algorithm, A night vision method and system can be provided to the user to support the driver to know the forward situation in advance.

The present invention also provides a method and apparatus for reducing the amount of computation in the generation of a region of interest (ROI) using information obtained through radar and radar, Vision method and system to the user.

The above-described embodiments of the present invention can be implemented by various means. For example, embodiments of the present invention may be implemented by hardware, firmware, software, or a combination thereof.

In the case of hardware implementation, the method according to embodiments of the present invention may be implemented in one or more Application Specific Integrated Circuits (ASICs), Digital Signal Processors (DSPs), Digital Signal Processing Devices (DSPDs), Programmable Logic Devices Field programmable gate arrays (FPGAs), processors, controllers, microcontrollers, microprocessors, and the like.

In the case of an implementation by firmware or software, the method according to embodiments of the present invention may be implemented in the form of a module, a procedure, or a function for performing the functions or operations described above. The software code can be stored in a memory unit and driven by the processor. The memory unit may be located inside or outside the processor, and may exchange data with the processor by various well-known means.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The foregoing detailed description of preferred embodiments of the invention disclosed herein is provided to enable any person skilled in the art to make and use the present invention. While the present invention has been particularly shown and described with reference to preferred embodiments thereof, it will be understood by those skilled in the art that various changes and modifications may be made therein without departing from the spirit and scope of the invention as defined by the appended claims. For example, those skilled in the art can utilize each of the configurations described in the above-described embodiments in a combination of them. Accordingly, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

The present invention may be embodied in other specific forms without departing from the spirit or essential characteristics thereof. Accordingly, the above description should not be construed in a limiting sense in all respects and should be considered illustrative. The scope of the present invention should be determined by rational interpretation of the appended claims, and all changes within the scope of equivalents of the present invention are included in the scope of the present invention. The present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein. In addition, claims that do not have an explicit citation in the claims may be combined to form an embodiment or be included in a new claim by amendment after the filing.

Claims (11)

A first step of acquiring thermal energy information for at least one object through a Far Infra-Red (FIR) camera;
A second step of extracting first thermal energy information satisfying a predetermined condition among the thermal energy information;
A third step of extracting a vertical edge from the thermal energy information;
A fourth step of extracting second thermal energy information mapped to the extracted vertical edge of the first thermal energy information;
A fifth step of extracting a vertical one end point and another end point of the object included in the second thermal energy information;
A sixth step of generating a region of interest (ROI) using the one end and the other end;
A seventh step of comparing a plurality of classifiers related to a first object to be identified with an object included in the ROI;
An eighth step of extracting an ROI corresponding to at least one of the plurality of classifiers in the ROI; And
Displaying the extracted ROI through a display unit; , ≪ / RTI &
Between the first step and the fifth step,
A step 10) of obtaining additional information about the at least one object through at least one of light detection and ranging (LiDAR) and radio detaching and ranging (RADAR); Further comprising:
In the sixth step,
And the ROI is generated using the additional information and the one end and the other end point together. The method according to claim 1,
In the sixth step,
Confirm the placement of the at least one object using the additional information,
Wherein the ROI is generated using only one end point and the other end point corresponding to the identified arrangement among the one end point and the other end point extracted in the fifth step. The method according to claim 1,
Wherein the LiDAR obtains information about the at least one object located within a preset distance from the FIR camera,
Wherein the RADAR obtains information on the at least one object located outside a preset separation distance from the FIR camera. The method according to claim 1,
Wherein the preset condition is a condition having a brightness value equal to or greater than a predetermined value. 5. The method of claim 4,
Wherein the predetermined condition includes a first condition and a second condition,
Wherein the brightness value of the first condition is greater than the brightness value of the second condition,
Wherein the fourth step is applied to the first thermal energy information satisfying the second condition,
Wherein the fifth step is performed using the first thermal energy information and the second energy information satisfying the first condition. The method according to claim 1,
Wherein the second step and the third step are simultaneously performed. The method according to claim 1,
Between the eighth step and the ninth step,
An eighth step of tracking an object included in the extracted ROI using at least one of the one end and the other end; Further comprising:
And the ninth step displays the information to be tracked in the eighth step. The method according to claim 1,
The first to ninth steps are repeatedly performed,
The sixth step, which is repeatedly performed,
A sixth step of setting a vanishing line around the vertical axis of the object included in the extracted ROI from the eighth step performed previously;
(6-2) setting a vanishing region based on the set vanishing line;
6-3) comparing the vanishing area statistical information related to the first object with the vanishing area; And
If the difference between the vanishing region statistical information and the vanishing region is within a preset range, generating an ROI using the one end and the other end; And displaying the night vision information. The method according to claim 1,
The first to ninth steps are repeatedly performed,
In the eighth step,
An 8-2 step of setting a vanishing line around the vertical axis of the object included in the extracted ROI from the eighth step performed previously;
8-3) setting a vanishing region on the basis of the set vanishing line;
Comparing the vanishing area statistical information associated with the first object and the vanishing area; And
Extracting an ROI corresponding to at least one of the plurality of classifiers from the ROI when the difference between the vanishing region statistical information and the vanishing region is within a preset range; And displaying the night vision information. The method according to claim 1,
Wherein the first object comprises a forward vehicle, a pedestrian and an animal,
Wherein the thermal energy information is obtained at night in the first step. A Far Infra-Red (FIR) camera that obtains thermal energy information for at least one object;
Light detection and ranging (LiDAR) and Radio Detecting And Ranging (RADAR) to obtain additional information about the at least one object;
The first thermal energy information extracting unit extracts first thermal energy information satisfying a predetermined condition among the thermal energy information, extracts a vertical edge from the thermal energy information, and maps the first thermal energy information to the extracted vertical edge Extracting two thermal energy information, extracting vertical and other end points of the object included in the second thermal energy information, generating an ROI using the additional information, the one end and the other end point together, A controller for comparing a plurality of classifiers associated with the classifier and an object included in the ROI and extracting an ROI corresponding to at least one of the plurality of classifiers in the ROI; And
A display unit for displaying the extracted ROI; The night vision information display device comprising:

Images