KR20200027789A - Image sysntehsis device and driving assitance system including the same - Google Patents

Abstract

According to an exemplary embodiment of the present disclosure, an image integration device can comprise: a first camera module acquiring a visible light image from a field and generating first image data based on the visible light image; a second camera module acquiring an infrared image from the field and generating second image data based on the infrared image; a third camera module acquiring depth information of the field and generating third image data based on the depth information; and an image integration part generating first, second, and third weights corresponding to the first, second, and third image data respectively, and generating integrated image data from the first, second, and third image data based on the first, second, and third weights.

Description

Image integration device and driving assistance system including the same {IMAGE SYSNTEHSIS DEVICE AND DRIVING ASSITANCE SYSTEM INCLUDING THE SAME}

The technical idea of the present disclosure relates to image data processing, and more particularly, to an apparatus for integrating image data received from three different cameras and a driving assistance system using the same.

A general camera can detect visible light incident from a field in order to generate the same image that a human eye sees, that is, a visible light image. However, such a visible light camera may not be able to generate an effective image in a situation in which it is difficult to detect visible light such as a dark night or haze. Unlike a camera for photographing a person or a landscape, in the case of a camera used to photograph situations occurring in the field of field such as CCTV, black box, etc., usefulness may be deteriorated while a valid image cannot be generated.

To solve this problem, an infrared camera that detects infrared rays incident from the field of view can be used with the visible light camera. However, the infrared camera may provide an effective image in a low-light environment, but there is a limitation that the effective image cannot be provided even in a case where the temperature of the par-clock is uniform or heat sensing is difficult.

The technical idea of the present disclosure provides an image integration device capable of generating an accurate image of a field of field even in a poor environment for assistance of a user.

In order to achieve the above object, an image integrating apparatus according to an aspect of the technical idea of the present disclosure is a first that acquires a visible light image from a filed and generates first image data based on the visible light image Camera module, a second camera module that acquires an infrared image from the depth of field and generates second image data based on the infrared image, a third camera that acquires depth information about the depth of field and generates third image data based on the depth information The module generates first, second, and third weights corresponding to the first, second, and third image data, respectively, and based on the first, second, and third weights, the first, second, and third And an image integrator generating integrated image data from the image data.

According to an exemplary embodiment of the present disclosure, the image integrator divides the field of view into at least one region and corresponds to one region based on a single first weight, a single second weight, and a single third weight in one region. Can generate a portion of the integrated image data.

According to an exemplary embodiment of the present disclosure, the image integrator may acquire illumination information of the depth of field based on the first image data, and divide the depth of field into at least one area based on the illumination level information.

According to an exemplary embodiment of the present disclosure, the image integrator may acquire temperature information of the field based on the second image data, and divide the field into at least one area based on the temperature information.

According to an exemplary embodiment of the present disclosure, the image integrator may divide the depth of field into at least one area based on the depth information.

According to an exemplary embodiment of the present disclosure, each of the at least one region may be defined by a simple closed curve.

According to an exemplary embodiment of the present disclosure, the image integrator estimates a velocity of at least one region based on sequentially generated third image data, and a third corresponding to the at least one region based on the estimated velocity. The weight can be changed.

According to an exemplary embodiment of the present disclosure, the image integrator is integrated to include a window that highlights at least one region based on at least one of a first weight, a second weight, and a third weight corresponding to the at least one region. You can create image data.

According to an exemplary embodiment of the present disclosure, the image integrator may merge two or more regions based on first weights, second weights, and third weights corresponding to two or more regions.

According to an exemplary embodiment of the present disclosure, the image integrator acquires illuminance information of the depth of field based on the first image data, acquires temperature information of the depth of field based on the second image data, and illuminance information, temperature information, and Each of the first weight, the second weight, and the third weight may be generated based on at least one of the depth information.

According to an exemplary embodiment of the present disclosure, the second camera module may generate second image data from infrared rays in a wavelength band of about 8 μm to about 12 μm, which are received from a field of field.

A driving assistance system according to an aspect of the technical spirit of the present disclosure may include an image integrating device that generates integrated image data, and a display device that displays an integrated image based on the integrated image data, wherein the image integrating device includes: First, second and third camera modules for acquiring visible light images, infrared images, and distance information about the vehicle surroundings from the vehicle surroundings, and first received from the first, second, and third camera modules, respectively , An image integration unit generating integrated image data based on the second and third image data.

According to an exemplary embodiment of the present disclosure, the image integrator generates first, second, and third weights corresponding to the first, second, and third image data, respectively, and the first, second, and third Integrated image data may be generated based on the weight.

According to an exemplary embodiment of the present disclosure, the image integrator divides the field of view into at least one region and corresponds to one region based on a single first weight, a single second weight, and a single third weight in one region. Can generate a portion of the integrated image data.

According to an exemplary embodiment of the present disclosure, the image integrating unit acquires illumination information about the ambient light around the vehicle and driving light information about an area where the driving light of the vehicle is irradiated, and based on the illumination information and the driving light information, When the ambient light around the vehicle is low, the first weight corresponding to the area where the driving light of the vehicle is irradiated may be increased, and the first weight corresponding to the area excluding the area where the driving light of the vehicle is irradiated may be decreased.

According to an exemplary embodiment of the present disclosure, the image integrator may further include an output device capable of generating a warning signal based on the distance information, and generating a signal recognizable by the driver based on the warning signal.

By the image integrating device and the driving assistance system according to the exemplary embodiment of the present disclosure, an image accurately representing the depth of field can be obtained even in a situation in which the field of view is difficult to detect, as well as in a poor viewing condition.

In addition, the image integration device and the driving assistance system according to the exemplary embodiment of the present disclosure not only assist the driver's vision, but also predict and warn of danger, thereby preventing driver's safety as well as pedestrian safety and property damage. You can.

The effects obtainable in the exemplary embodiments of the present disclosure are not limited to the above-mentioned effects, and other effects not mentioned are common knowledge in the art to which the exemplary embodiments of the present disclosure belong from the following description. It can be clearly drawn and understood by those who have it. In other words, unintended effects of practicing exemplary embodiments of the present disclosure may also be derived by one of ordinary skill in the art from the exemplary embodiments of the present disclosure.

1 is a block diagram illustrating an image integration apparatus according to an exemplary embodiment of the present disclosure.
2A to 2C show examples of an image photographed with a depth of field according to example embodiments of the present disclosure.
3 shows an example of the image integrator of FIG. 1 in accordance with an exemplary embodiment of the present disclosure.
4 is a diagram showing an example of a depth of field according to an exemplary embodiment of the present disclosure.
5 shows an example of a field of view including divided regions according to an exemplary embodiment of the present disclosure.
6 shows an example of an integrated image creation operation in accordance with an exemplary embodiment of the present disclosure.
7 is a block diagram illustrating a driving assistance system according to an exemplary embodiment of the present disclosure.
8 shows an example of a display of the display device of FIG. 7 in accordance with an exemplary embodiment of the present disclosure.
9A and 9B are diagrams for describing an operation of the image integrator according to an exemplary embodiment of the present disclosure.
10 is a diagram illustrating a method for generating an integrated image according to an exemplary embodiment of the present disclosure.
11 is a diagram illustrating a vehicle including an integrated imaging device according to an exemplary embodiment of the present disclosure.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. The embodiments of the present invention are provided to more fully describe the present invention to those skilled in the art. The present invention can be applied to various changes and may have various forms, and specific embodiments will be illustrated in the drawings and described in detail. However, this is not intended to limit the present invention to a specific disclosure form, and it should be understood that all modifications, equivalents, and substitutes included in the spirit and scope of the present invention are included. In describing each drawing, similar reference numerals are used for similar components. In the accompanying drawings, the dimensions of the structures are shown to be enlarged or reduced than actual to clarify the present invention.

Terms used in the present application are only used to describe specific embodiments, and are not intended to limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this application, the terms "include" or "have" are intended to indicate the presence of features, numbers, steps, actions, elements, parts or combinations thereof described in the specification, one or more other features. It should be understood that the presence or addition possibilities of fields or numbers, steps, actions, components, parts or combinations thereof are not excluded in advance.

Unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as commonly understood by a person skilled in the art to which the present invention pertains. Terms such as those defined in a commonly used dictionary should be interpreted as having meanings consistent with meanings in the context of related technologies, and are not to be interpreted as ideal or excessively formal meanings unless explicitly defined in the present application. .

1 is a block diagram illustrating an image integrating apparatus 100 according to an exemplary embodiment of the present disclosure. As shown in FIG. 1, the image integrating apparatus 100 may generate integrated image data IMG representing the field 5 by photographing the field 5. The image integration device 100 may include a first camera module 110, a second camera module 120, and a third camera module 130, and may include an image integration unit 140.

The first camera module 110 may acquire a visible light image from the depth of field 5 and generate first image data IMG1 based on the obtained visible light image. In the present specification, the first camera module 110 may be referred to as a visible light camera or a visible light camera module. Visible light, for example, may refer to an electromagnetic wave or a range of wavelengths having a wavelength between about 450 nm and about 650 nm. In some embodiments, the first camera module 110 may include a lens, an aperture, an image sensor, an image processor, and the like, and the image sensor included in the first camera module 110 may be electrically operated by sensing household light. A pixel array in which a plurality of pixels generating a signal are arranged may be included. The image processor may generate the first image data IMG1 corresponding to the visible light image of the depth of field 5 by processing the signal received from the pixel array.

The second camera module 120 may acquire an infrared image from the depth of field 50 and generate second image data IMG2 based on the acquired infrared image. In the present specification, the second camera module 120 may be referred to as an infrared camera or infrared camera module. Infrared light may refer to a range of electromagnetic waves or wavelengths, for example, having a wavelength between about 750 nm and 1 mm. In some embodiments, the second camera module 120 may include a lens, aperture, image sensor, image processor, and the like, and the image sensor included in the second camera module 120 may detect an infrared signal to detect electrical signals. It may include a pixel array in which a plurality of pixels generating. For example, as illustrated in FIG. 1, the image sensor included in the second camera module 120 may detect, for example, far infrared rays having a wavelength between about 8 μm and 12 μm. Since all objects above the absolute temperature can emit infrared rays and the amount of infrared radiation is determined according to the temperature, the second camera module 120 can be used to measure the depth of field 5 even at low illumination conditions, such as at night or inside a tunnel. Subjects included in may generate second image data IMG2 corresponding to the identified infrared image.

The third camera module 130 may acquire depth information about the depth of field 5 and generate third image data IMG3 based on the depth information. In the present specification, the third camera module 130 may be referred to as a depth camera or depth camera module. Depth information may refer to distance information of subjects included in the depth of field 5 from the third camera module 130 (or the image integration device 100). In some embodiments, the third camera module 130 may include a light source, a lens, a pixel array, and a controller, and the pixel array detects light received by reflecting light output from the light source to the depth of field 5. You can. For example, the light source may include a light emitting device (LED) or a laser device such as a vertical cavity surface emitting laser (VCSEL). The light source may output light modulated according to a modulated signal generated by the controller, and the pilcell array may provide the generated electrical signal to the controller by sensing the light reflected from the subject. The controller may detect a variation occurring in the received signal, and estimate a Time Of Flight (TOF) based on the variation. The controller may generate depth information of the depth of field 5 according to the estimated TOF, and may generate third image data IMG3 including depth information of the depth of field 5. Accordingly, the third camera module 130 may acquire distance information to the subject, not only in an opaque condition for a watch, but also for a subject that is difficult to sense heat, for example, a subject that is adjacent to a high-temperature heating element. The data IMG3 may include such distance information (or depth information).

The image integration unit 140 may control the first camera module 110, the second camera module 120, and the third camera module 130. In addition, the image integration unit 140 includes first image data IMG1, second image data IMG2, and first images from the first camera module 110, the second camera module 120, and the third camera module 130. Each of the three image data IMG3 can be received, and the integrated image data IMG can be generated by integrating the first image data IMG1, the second image data IMG2, and the third image data IMG3. . The first image data IMG1 output by the first camera module 110 in a good watch state can satisfactorily represent the depth of field 5, while the second camera module 120 outputs in a poor watch state. The second image data IMG2 and / or the third image data IMG3 output by the third camera module 130 may be used to represent the depth of field 5. Accordingly, the integrated image data IMG can accurately represent the field 5 even under various conditions.

In some embodiments, the image integrator 140 may generate the first weight W1, the second weight W2, and the third weight W3, and the first weight W1 and the second weight W The integrated image data IMG may be generated from the first image data IMG1, the second image data IMG2, and the third image data IMG3 based on the W2) and the third weight W3. Details of the image integration unit 140 will be described later with reference to FIG. 3 and the like.

2A to 2C show examples of an image photographed with a depth of field according to example embodiments of the present disclosure. Specifically, FIGS. 2A to 2C show a visible light image and an infrared image taken with a field of view when the watch is in poor condition, and the infrared image is shown at the bottom right of the visible light image frame. As described above with reference to FIG. 1, the first image data IMG1 output by the first camera module 110 while the watch is in poor condition may not represent the field 5 satisfactorily.

When the illuminance of the depth of field is low, the subject may not be identifiably displayed in the visible light image. For example, as shown in FIG. 2A, an object that does not emit visible light in a visible light image photographed at a dark night, for example, a person, etc., may not be identifiably displayed in the visible light image. On the other hand, as shown in the lower right of FIG. 2A, in the case of an infrared image, the subject may be clearly displayed due to the temperature (ie, body temperature) of the subject.

When the difference in illuminance is large in the depth of field, the subject may not be identifiably displayed in the visible light image. For example, as illustrated in FIG. 2B, in a visible light image photographed while escaping from a tunnel, a subject existing outside the tunnel with high illumination may not be identifiably displayed in the visible light image. On the other hand, as shown in the lower right of FIG. 2B, in the case of an infrared image, the subject may be clearly displayed due to the temperature of the subject (ie, the vehicle temperature).

When the field of view is opaque, the subject may not be identifiably displayed in the visible light image. For example, as shown in FIG. 2C, when a haze occurs between a camera module (eg, 110 in FIG. 1) and a field of view, a subject located at the rear of the haze in the visible light image, such as a person, can see the visible light image It may not be displayed identifiably in. On the other hand, as shown in the lower right of FIG. 2C, in the case of an infrared image, the subject may be clearly displayed due to the temperature (ie, body temperature) of the subject. As described above, even when the field of view cannot be accurately represented by only the visible light image, the infrared image can accurately represent the subject included in the field of view.

In addition, as described above with reference to FIG. 1, the third image data IMG3 output by the third camera module 130 may include depth information about the depth of field 5. Depth information about the subjects in the field can be obtained, and the acquired depth information can be expressed in the form of an image. For example, each of the pixels included in the image may have depth information, and pixels corresponding to the same depth may be shown as one region bounded by a contour line, and thus include depth information. In the image, an object, such as a person, can be displayed identifiably.

3 shows an example of the image integrator 140 of FIG. 1 according to an exemplary embodiment of the present disclosure. As described above with reference to FIG. 1, the image integration unit 140 ′ of FIG. 3 includes integrated image data IMG from the first image data IMG1, the second image data IMG2, and the third image data IMG3. ). As described above with reference to FIGS. 2A to 2C, when using not only a visible light image but also an infrared image and depth information, it may be possible to generate an image that more accurately identifies a subject. The image integration unit 140 'uses the first image data IMG1 corresponding to the visible light image, the second image data IMG2 corresponding to the infrared image, and the third image data IMG3 including depth information. Even in adverse conditions, it is possible to generate integrated image data (IMG) that accurately represents the depth of field.

In some embodiments, the operations of the image integrator 140 ′ may be performed by at least one processor executing program code including instructions corresponding to the operations. The instructions can be stored in memory, and the processor is a data processing device implemented in hardware, including circuitry that is physically structured to perform desired operations, including operations represented as instructions contained in code and / or programs, for example. You can. In some embodiments, a data processing device implemented in hardware includes, as non-limiting examples, a microprocessor, central processing unit (CPU), processor core, multi-core processor, multiprocessor, application specific integrated circuit (ASIC), and FPGA (Field Programmable Gate Array).

The region dividing unit 142 may divide a field into a plurality of regions R1, ..., Rn. Each of the plurality of regions R1, ..., Rn may be a set of pixels having a common characteristic, and may be defined by a single closed curve, as described below with reference to FIG. 5. As described later, pixels included in one region can be processed by weights of the same weight set. In some embodiments, the region division unit 142 may receive the first weight W1, the second weight W2, and the third weight W3 generated by the weight generator 144, and the first Two or more regions may be merged based on the weight W1, the second weight W2, and the third weight W3. For example, when the first weight W1, the second weight W2, and the third weight W3 are adjacent to the mutually adjacent regions, both regions may be merged with each other.

In some embodiments, the region dividing unit 142 may acquire illuminance information of the depth of field based on the first image data IMG1, and may divide the depth of field into at least one region based on the acquired illuminance information. have. For example, the region dividing unit 142 may compare the illuminance represented by each pixel of the first image data IMG1 with at least one reference value, and divide pixels equally classified according to at least one reference value into one region Can be defined. Accordingly, the dark portion of FIG. 2A may be divided into one region, and the bright portion of FIG. 2B may be divided into one region, and these regions may be used for the visible light image corresponding to the first image data IMG1. Integrated image data (IMG) may be generated to be more dependent on the pixel value and / or depth of the infrared image than the pixel values. In some embodiments, the illuminance compared to the at least one reference value may be normalized illuminance with respect to the illuminance of the entire subject, and the at least one reference value may also be determined based on the illuminance distribution (eg, variance, etc.) of the entire subject. have.

In some embodiments, the region dividing unit 142 may acquire temperature information of the field based on the second image data IMG2, and may divide the field into at least one region based on the obtained temperature information. have. For example, the region dividing unit 142 may compare the temperature represented by each pixel of the second image data IMG2 with at least one reference value, and pixels that are equally classified according to the at least one reference value into one region Can be defined. Accordingly, a subject having a certain range of temperature, such as a body temperature or a driving vehicle, may be divided into one area, and these areas are relatively integrated with pixel values of an infrared image corresponding to the second image data IMG2. Image data (IMG) may be generated. In some embodiments, the temperature compared to the at least one reference value may be a temperature normalized to the temperature of the entire subject, and the at least one reference value may also be determined based on the temperature distribution (eg, dispersion, etc.) of the entire subject. have.

In some embodiments, the region dividing unit 142 may acquire depth information of the depth of field based on the third image data IMG3, and may divide the depth of field into at least one region based on the acquired depth information. have. For example, the region dividing unit 142 may compare a depth corresponding to each pixel of the third image data IMG3 with at least one reference value, and pixels that are equally classified according to at least one reference value in one region Can be defined as Accordingly, very close pedestrians, vehicles, and other structures may be divided into one area, and these areas may generate integrated image data (IMG) in which depth information corresponding to the third image data (IMG3) is relatively emphasized. have.

In some embodiments, the region dividing unit 142 may estimate the velocity of the region based on the third image data IMG3 sequentially generated in the past, and compare the estimated velocity with at least one reference value. The depth of field may be divided into at least one region. For example, the region dividing unit 142 may divide pixels in which the depth (or distance) decreases or increases at substantially the same slope as one region. An example of the operation of the region division unit 142 will be described later with reference to FIG. 4.

The weight generator 144 corresponds to the first weight W1 corresponding to the first image data IMG1, the second weight W2 corresponding to the second image data IMG2, and the third image data IMG3. A third weight W3 can be generated. The weight may indicate the importance of the corresponding image data, for example, the larger the value of the weight, the larger the corresponding image data may have a great influence on the integrated image data (IMG). The weight generator 144 may generate weights corresponding to each of the plurality of regions R1, ..., Rn, and pixels included in one region may include a first weight W1 and a second weight ( W2) and the third weight W3. For example, the weight generator 144 reduces the first weight to reduce the influence of the first image data IMG1 on the divided region as a very light portion or a very dark portion according to the illuminance. (W1). In addition, the weight generator 144 may generate an increased second weight W2 to increase the influence of the second image data IMG2 on the divided region as having a certain range of temperature (eg, body temperature). You can. In addition, the weight generation unit 144 may generate an increased third weight W3 to increase the influence of the third image data IMG3 on the divided region as having a certain distance or less or a certain speed or more. You can. An example of the operation of the weight generator 144 will be described with reference to FIG. 5.

In some embodiments, at least one of the first camera module 110, the second camera module 120, and the third camera module 130 of FIG. 1 may be disabled. That is, any combination of the first camera module 110, the second camera module 120, and the third camera module 130 may be used. For example, the first camera module 110, the second camera module 120, and the third camera may be considered in order to reduce power consumption of the image integrator 100 or to consider an environment in which the image integrator 100 is used. At least one of the modules 130 may be disabled according to external control. To this end, the disabled camera module among the first camera module 110, the second camera module 120, and the third camera module 130 may be turned off, and the weight generator 144 may have a first weight Among the (W1), the second weight (W2), and the third weight (W3), it may be determined that a predetermined constant value, for example, zero, corresponds to the turned-off camera module. In addition, in some embodiments, the weight generator 144 compensates for the omission of the pixel value by the disabled camera module, so that the first weight W1, the second weight W2, and the third weight ( W3) It is also possible to equally increase the value of the weight value corresponding to the enabled camera module.

The integrated image generating unit 146 may include the first image data IMG1, the second image data IMG2, and the third image based on the first weight W1, the second weight W2, and the third weight W3. The integrated image data IMG can be generated from the data IMG3. An example of the operation of the integrated image generator 146 will be described with reference to FIG. 6.

4 is a diagram showing an example of a depth of field according to an exemplary embodiment of the present disclosure. The depth of field 50 may include a plurality of subjects 51 to 56, and the image integration unit 140 of FIG. 1 may generate integrated image data IMG for accurately identifying the subjects 51 to 56. You can. Hereinafter, FIG. 4 will be described with reference to the image integrator 140 ′ of FIG. 3 as an example of the image integrator 140 of FIG. 1.

In some embodiments, the image integrator 140 ′ may be used to assist in driving the vehicle. Accordingly, it is possible to provide the driver with an image that identifiably displays obstacles or hazards even when the watch is in poor condition, or to automatically control driving (eg, deceleration, braking, steering, etc.) according to the identified obstacles or hazards. have. To this end, the image integrator 140 ′ may receive image data (eg, IMG1, IMG2, and IMG3) from cameras (eg, 110, 120, and 130 of FIG. 1) photographing the front of the vehicle.

Referring to FIG. 4, in some embodiments, subjects 51, 52, and 53 that are relatively short distance from the vehicle may be divided into independent regions according to depth information obtained from the third image data IMG3. have. Further, in some embodiments, subjects 54 and 55 having a specific range of temperature, such as body temperature, even when relatively distant, may be divided into independent regions according to temperatures obtained from the second image data IMG2. You can. In addition, the subject 56 having a constant speed, such as a driving vehicle, may be divided into independent regions according to the speed information obtained from the series of third image data IMG3 even when it is relatively far.

5 shows an example of a field of view including divided regions according to an exemplary embodiment of the present disclosure. As shown in FIG. 5, the depth of field 60 may be divided into seven regions R1 to R7. As described above with reference to FIG. 3, the region dividing unit 142 of the image integration unit 140 ′ may divide the field 60 into seven regions R1 to R7 in various ways. In the following, FIG. 5 will be described with reference to FIG. 3.

Referring to FIG. 5, each of the seven regions R1 to R7 may have a set of weights. For example, the first region R1 may have a set including the first weight W1 ₁ , the second weight W2 ₁ , and the third weight W3 ₁ , and the second region R2 is It may have a set including the first weight W1 ₂ , the second weight W2 ₂ , and the third weight W3 ₂ .

In some embodiments, the region dividing unit 142 may be divided into regions according to the first image data IMG1, regions divided according to the second image data IMG2, and the regions divided according to the third image data IMG3. The divided regions may be respectively generated, or the divided regions may be generated based on at least two of the first image data IMG1, the second image data IMG2, and the third image data IMT3. As shown in FIG. 5, the seven regions R1 to R7 may be defined by a simple closed curve.

6 shows an example of an integrated image creation operation in accordance with an exemplary embodiment of the present disclosure. As described above with reference to FIG. 3, the integrated image generation unit 146 may include first image data IMG1 and second based on the first weight W1, the second weight W2, and the third weight W3. Integrated image data IMG may be generated from the image data IMG2 and the third image data IMG3.

Referring to FIG. 6, the first image data IMG1, the second image data IMG2, and the third image data IMG3 may be resized to correspond to an image of the same size. For example, at least one of the first camera module 110, the second camera module 120, and the third camera module 130 of FIG. 1 may generate image data corresponding to images of different sizes from other camera modules. Accordingly, the image data generated by the corresponding camera module may be converted to correspond to the resized image.

In images of the same size, a weighted sum of the values of pixels at positions corresponding to each other can be calculated, and the weighted sum can be determined as the value of the pixel at the corresponding position in the combined image. For example, as illustrated in FIG. 6, values of pixels corresponding to the same positions in the first image data IMG1, the second image data IMG2, and the third image data IMG3, respectively, are P1, P2, Speaking of P3, the pixel value P in the integrated image data IMG can be calculated as shown in [Equation 1] below.

As described above with reference to FIG. 3 and the like, the first weight W1, the second weight W2, and the third weight W3 include first image data IMG1, second image data IMG2, and third Since the importance of the image data IMG3 can be represented respectively, the integrated image data IMG according to Equation 1 can accurately represent the subject even when the watch is in poor condition.

7 is a block diagram illustrating a driving assistance system 200 according to an exemplary embodiment of the present disclosure. The driving assistance system 200 may be mounted on a vehicle, as will be described later with reference to FIG. 11, and provide an image of a driver displaying an identifiable subject even in a situation in which the watch is poor. As illustrated in FIG. 7, the driving assistance system 200 may include an image integration device 210, a display device 220, and an output device 230. Hereinafter, the image integration device 210 is assumed to include the components included in the image integration device 100 of FIG. 1, and FIG. 7 will be described with reference to FIG. 1.

As described above with reference to the drawings, the image integration device 210 may generate integrated image data (IMG). The display device 220 may display an image corresponding to the integrated image data IMG received from the image integration device 210. For example, the display device 220 may include any display module such as a liquid crystal display (LCD), an organic light-emitting diode (OLED), or the like.

In some embodiments, the image integrating device 210 may generate a warning signal WAR based on depth information (or distance information) obtained from the third image data IMG3. For example, the image integrating device 210 may generate an activated warning signal WAR when the distance of the area is smaller than a predetermined reference value, and an activated alert when the speed of the area increases in a direction closer to the vehicle. It is also possible to generate a signal WAR.

The output device 230 may output a signal recognized by the driver in response to the activated warning signal WAR. For example, the output device 230 may include a lamp, a light emitting device (LED), and the like, and may output a visual signal. In addition, the output device 230 may include a speaker, a buzzer, and the like, and may output an audible signal. In addition, the output device 230 may include a motor or the like, and may output vibration. The driver may recognize a signal output from the output device 230, and accordingly, the driver may cope with a situation that is not recognized while driving.

In some embodiments, the warning signal WAR output by the image integrating device 210 may be output outside the driving assistance system 200. The warning signal WAR output to the outside of the driving assistance system 200 may be used to automatically control driving (eg, deceleration, braking, steering, etc.), and thus the driving assistance system 200 may be configured to It can be used as a part. In addition, the integrated image data (IMG) generated by the image integration device 210 may be accumulated and stored in storage, such as a flash memory, and accordingly, the integrated image data (IMG) may function as a black box image. have.

8 shows an example of a display of the display device 220 of FIG. 7 in accordance with an exemplary embodiment of the present disclosure. Specifically, FIG. 8 shows a display output by the display apparatus 220 from the integrated image data IMG generated by photographing the same depth of field as in FIG. 4. In the following, FIG. 8 will be described with reference to FIG. 7.

8, a plurality of objects 91 to 96 may be displayed on the display 90. In some embodiments, the image integrating device 210 may generate integrated image data (IMG) by adding a highlight for a divided area (or object). For example, as illustrated in FIG. 8, a vehicle 92 or a pedestrian 93 located at a position that can intersect with a vehicle traveling direction among objects close to a vehicle equipped with the driving assistance system 200 is a window. It can be emphasized by the fields (WIN1, WIN2), the windows (WIN1, WIN2) can move along the vehicle 92 or the pedestrian 93. The driver can easily recognize the cause of the danger by the windows WIN1 and WIN2 displayed on the display device 220 as well as the danger situation recognition by the output device 230.

9A and 9B are diagrams for describing an operation of the image integrator according to an exemplary embodiment of the present disclosure. Specifically, FIGS. 9A and 9B show areas 11a and 11b to which the driving lights of the vehicle are irradiated, respectively. Hereinafter, FIGS. 9A and 9B will be described with reference to the driving assistance system 200 of FIG. 7.

Referring to FIG. 9A, when the driving light is adjusted downward, the area 11a to which the driving light is irradiated may be limited to a relatively wide and low area in the field. On the other hand, when the driving light is adjusted upward, the area 11b to which the driving light is irradiated may be limited to a relatively narrow and high area in the field. Information on the irradiation direction of the driving lights may be provided to the driving assistance system 200.

In some embodiments, the image integration device 210 of the driving assistance system 200 may obtain driving light information for an area in which the driving light of the vehicle is irradiated, and the driving light is irradiated based on the driving light information Weights applied to areas and other areas can be adjusted. For example, the image integration device 210 may obtain the illumination information about the illumination around the vehicle, and when the illumination around the vehicle is low, the difference in illumination between the area where the driving light is irradiated and other areas, such as at night or in a tunnel, is different. You can recognize that there is. The image integration device 210 increases the first weight W1 corresponding to the area where the driving light of the vehicle is irradiated when the light intensity around the vehicle is low, while responding to the area other than the area where the driving light of the vehicle is irradiated. The first weight W1 can be reduced.

10 is a diagram illustrating a method for generating an integrated image according to an exemplary embodiment of the present disclosure. For example, the method of FIG. 10 may be performed by the image integrator 140 ′ of FIG. 3, and FIG. 10 will be described below with reference to FIG. 3.

In step S20, an operation of acquiring image data may be performed. For example, the image integration unit 140 'includes first image data IMG1 corresponding to a visible light image, second image data IMG2 corresponding to an infrared image, and third image data IMG3 including depth information. ) Can be obtained from multiple cameras.

In step S40, an operation of dividing into at least one region may be performed. For example, the region division unit 142 of the image integration unit 140 ′ may divide the image into at least one region in various ways, as described above with reference to FIGS. 3 to 5. The divided area may be a single object or an object, or a background that is not important for the identification of the object.

In step S60, an operation for generating weights may be performed. For example, the weight generation unit 144 of the image integration unit 140 'may include first weights corresponding to the first image data IMG1, the second image data IMG2, and the third image data IMG3, respectively. W1), the second weight W2 and the third weight W3 may be generated. In addition, the weight generator 144 may generate a unique first weight W1, a second weight W2, and a third weight W3 for each of the divided regions in step S40.

In step S80, an operation of generating an integrated image may be performed. For example, the integrated image generation unit 146 of the image integration unit 140 'is based on the first weight W1, the second weight W2, and the third weight W3, the first image data IMG1. , Integrated image data IMG may be generated from the second image data IMG2 and the third image data IMG3. As described above with reference to FIG. 6, the values of the pixels of the integrated image data IMG are weighted of the values of the pixels of the first image data IMG1, the second image data IMG2, and the third image data IMG3. It can be calculated as a sum.

11 is a diagram illustrating a vehicle 300 including an integrated imaging device according to an exemplary embodiment of the present disclosure. As described above with reference to FIG. 7, the image integrating device 210 can be used as a component of the driving assistance system 200.

Referring to FIG. 11, the first integrated image device 310 may be mounted to acquire the front of the vehicle as a field of view, while the second integrated image device 320 may be mounted to acquire the rear of the vehicle as the field of view. . In addition, although not shown in FIG. 11, the integrated image device may be mounted on the side of the vehicle, or may be mounted on the head or stern of the vehicle, respectively. Accordingly, the driver of the vehicle 300 can easily recognize a dangerous situation even when the clock is poor, and autonomous driving of the vehicle 300 can be realized.

As described above, exemplary embodiments have been disclosed in the drawings and the specification. Although embodiments have been described using specific terms in this specification, they are used only for the purpose of describing the technical spirit of the present disclosure and are not used to limit the scope of the disclosure as defined in the meaning or claims. . Therefore, those of ordinary skill in the art will understand that various modifications and other equivalent embodiments are possible therefrom. Therefore, the true technical protection scope of the present disclosure should be defined by the technical spirit of the appended claims.

Claims (16)

A first camera module configured to acquire a visible light image from a filed and generate first image data based on the visible light image;
A second camera module configured to acquire an infrared image from the field and generate second image data based on the infrared image;
A third camera module configured to acquire depth information for the depth of field and generate third image data based on the depth information; And
First, second, and third weights corresponding to each of the first, second, and third image data are generated, and the first, second, and third weights are generated based on the first, second, and third weights. And an image integrator configured to generate integrated image data from the image data. The method according to claim 1,
The image integrator divides the depth of field into at least one area, and a portion of the integrated image data corresponding to the one area based on a single first weight, a single second weight, and a single third weight in one area The image integration device, characterized in that further configured to generate. The method according to claim 2,
And the image integration unit is further configured to acquire illumination information of the field based on the first image data, and to divide the field into the at least one region based on the illumination information. The method according to claim 2,
And the image integrating unit is further configured to acquire temperature information of the field based on the second image data, and to divide the field into the at least one region based on the temperature information. The method according to claim 2,
And the image integrating unit is further configured to divide the depth of field into the at least one area based on the depth information. The method according to claim 2,
Each of the at least one region is defined by a single closed curve (image closed device). The method according to claim 2,
The image integration unit is further configured to estimate the velocity of the at least one region based on the sequentially generated third image data, and to change a third weight corresponding to the at least one region based on the estimated velocity. Image integration device characterized in that. The method according to claim 2,
The image integration unit is further configured to generate the integrated image data to include a window that emphasizes the at least one region based on at least one of a first weight, a second weight, and a third weight corresponding to the at least one region. Image integration device, characterized in that configured. The method according to claim 2,
The image integrating unit is further configured to merge the two or more regions based on first weights, second weights, and third weights corresponding to two or more regions. The method according to claim 1,
The image integration unit acquires illuminance information of the depth of field based on the first image data, obtains temperature information of the depth of field based on the second image data, and obtains the illuminance information, the temperature information, and the depth information. And configured to generate each of the first weight, the second weight, and the third weight based on at least one of them. The method according to claim 1,
And the second camera module is configured to generate the second image data from infrared rays in a wavelength band of about 8 μm to about 12 μm received from the field. An image integrating device configured to generate integrated image data; And
A display device configured to display an integrated image based on the integrated image data,
The image integration device,
First, second, and third camera modules for acquiring visible light images, infrared images, and distance information about the vehicle surroundings from the vehicle surroundings; And
And an image integrator configured to generate the integrated image data based on first, second, and third image data received from the first, second, and third camera modules, respectively. The method according to claim 12,
The image integration device generates first, second, and third weights corresponding to each of the first, second, and third image data, and based on the first, second, and third weights, the integrated image Driving assistance system characterized in that it is further configured to generate data. The method according to claim 13,
The image integrator divides the field of view into at least one region, and a portion of the integrated image data corresponding to the one region based on a single first weight, a single second weight, and a single third weight in one region. Driving assistance system, characterized in that further configured to generate. The method according to claim 14,
The image integration unit acquires illumination information about the illumination around the vehicle and driving lamp information about an area where the driving lights of the vehicle are irradiated, and based on the illumination information and the driving lamp information, the illumination around the vehicle If is low, driving is further configured to increase the first weight corresponding to the area where the driving light of the vehicle is irradiated and decrease the first weight corresponding to the area except the area where the driving light of the vehicle is irradiated. Auxiliary system. The method according to claim 12,
The image integration unit is further configured to generate a warning signal based on the distance information,
And an output device configured to generate a signal recognizable by the driver based on the warning signal.

Images