KR101907657B1 - Apparatus securing exposure time of image sensor - Google Patents

Abstract

An apparatus for securing the exposure time of an image sensor according to the present invention secures an exposure time for exposing the image sensor according to a measured exposure angle based on the attitude information of an apparatus for securing an exposure time sensed by using an attitude information sensor in order to solve a problem that an image sensor module becomes large, and a problem that it is difficult to ensure proper exposure time of the image sensor due to the change of light amount when a device having an image sensor to secure the exposure time of the image sensor moves suddenly. Accordingly, even if the posture of the apparatus having the image sensor is suddenly changed, the time of exposure of the image sensor to external light can be controlled in real time, and the brightness of an image can be adjusted. The apparatus includes an image sensor part, an image processing part, an exposure angle measuring part, and an exposure time obtaining part.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to an image sensor,

The present invention relates to an exposure time securing apparatus for an image sensor. More particularly, the present invention relates to an exposure time securing apparatus for an image sensor that secures exposure time for exposure of an image sensor to external light.

As with all cameras, an image sensor that acquires external image data also has a field of view (FOV). Therefore, the angle of view from the image sensor to the outside is not infinite.

If you perform combat missions while acquiring image data that includes a target from an image sensor mounted on the aircraft, the aircraft may rotate the body during flight or move to another location to view other targets from the target Suddenly I can look at the sun.

If the aircraft is looking at various targets on the ground or over the ground, the amount of light reflected from each target will be different, so different exposure times corresponding to each target will be required. The exposure time represents the time during which the image sensor is exposed from the light incident from outside in order to acquire image data from the incident light.

If the aircraft suddenly rushes during a combat mission, there is a problem that it is difficult to achieve the mission because it is difficult to secure the proper exposure time of the image sensor due to sudden light intensity change. Therefore, there is a need for a technique for ensuring a clear image in real time even if the amount of incident light suddenly changes while the image sensor is exposed.

In recent years, each image sensor module is often equipped with a circuit for automatically controlling the exposure time in an auto exposure mode, but the image sensor module becomes large.

The present invention solves the problem that the image sensor module becomes large and secures the exposure time of the image sensor so as to solve the problem that it is difficult to secure a proper exposure time of the image sensor due to the change of the light amount when the air vehicle suddenly starts running during the combat mission The present invention provides an exposure time securing apparatus for measuring an exposure angle of light incident on an image sensor based on attitude information of the apparatus and securing an exposure time in which the image sensor is exposed according to the measured exposure angle in real time.

According to an aspect of the present invention, there is provided an exposure time securing apparatus for an image sensor, comprising: an image sensor unit for acquiring image data using the image sensor; An image processor for calculating a representative value from the pixels included in the image data obtained from the image sensor; An exposure angle measuring unit for measuring an exposure angle, which is an angle between a predetermined reference direction and a direction of light incident to the image sensor unit, based on attitude information of the exposure time securing apparatus sensed using the attitude information sensor; And an exposure time acquiring unit for acquiring an exposure time, which is a time during which the image sensor unit is exposed to the light so that the calculated representative value becomes a predetermined data value, according to the measured exposure angle.

And a memory for storing the exposure time acquired according to the exposure angle.

Wherein the control unit controls the exposure angle measuring unit to measure the exposure angle in real time and controls the exposure time acquiring unit to acquire an exposure time according to the exposure angle measured in real time so that the exposure time acquired according to the exposure angle stored in the memory And a controller for comparing the obtained exposure time with the exposure time measured in real time to update the exposure time stored in the memory.

When the attitude information is changed according to the movement of the exposure time securing device, the exposure angle measuring unit controls to measure the current exposure angle on the basis of the attitude information of the exposure time securing apparatus in which the attitude information is changed, A control unit for determining an exposure angle corresponding to the currently measured exposure angle, loading an exposure time corresponding to the determined exposure angle in the memory, and controlling the exposure of the image sensor unit to the light during the loaded exposure time; As shown in FIG.

The representative value may be an average value of pixel values of pixels included in the image data obtained from the image sensor and operating according to the incident light.

The predetermined data value may be an intermediate value indicating a number corresponding to the middle of the number of colors that the pixels included in the image data can express.

Wherein the exposure angle measuring unit comprises: a plurality of posture information sensor units for collecting measurement data on the posture of the exposure time securing apparatus through at least two posture information sensors; An attitude information sensor fusion unit for acquiring state variables for the attitude of the exposure time securing device by fusing and processing the collected measurement data; A filter unit for estimating state information on the attitude of the exposure time securing apparatus by applying a filter for estimating the state of the nonlinear dynamic system to the state variable; And an attitude information calculation unit for calculating attitude information according to motion of the exposure time securing apparatus in real time based on the estimated state information.

Wherein the plurality of attitude information sensor units comprises: a first attitude information sensor unit for collecting acceleration values (X, Y, Z) of the X-axis, Y-axis and Z-axis of the exposure time securing device from one three-axis accelerometer sensor; And a second attitude information sensor unit for collecting the angular velocity values (p, q, r) of the exposure time securing device from two biaxial gyro sensors.

Wherein the posture information sensor fusion unit obtains a change rate of the Euler angle of the exposure time securing device using a relational expression between the angular velocity of the exposure time securing device and the Euler angle, Wherein the Euler angle is defined by the rotation angle pie (?) With respect to the X-axis direction, the rotation angle theta (?) With respect to the Y-axis direction, and the rotation angle? Can be obtained as a state variable.

Wherein the predetermined reference direction is a relationship in which the X-axis, the Y-axis, and the Z-axis are all vertical, the Z-axis is perpendicular to the ground, and the rotation- (Pitch angle) is 0 degree, and the exposure angle may be the rotation angle theta ([theta]).

Wherein the filter unit applies an extended Kalman filter to the state variable to measure a predicted value and a predicted error covariance of the state information of the exposure time securing apparatus, calculates a Kalman gain, and uses the Kalman gain and the predicted estimated value The state information of the image sensor unit may be estimated, and the error covariance may be obtained using the Kalman gain and the prediction error covariance.

Wherein the filter unit updates the pie and theta obtained from the three-axis accelerometer among the sensor measurement values, obtains the state variable according to the update of the sensor measurement value, It is possible to repeat the process of estimating the state information of the secure device.

According to an embodiment of the present invention, even when the posture of the apparatus on which the image sensor is mounted is suddenly changed to secure the exposure time of the image sensor, the time during which the image sensor is exposed to external light can be controlled in real time. Therefore, the brightness of the image can be adjusted so that the image is vivid in real time without adding any additional hardware.

The effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned can be clearly understood to those of ordinary skill in the art from the following description.

FIGS. 1A through 1E illustrate that the CMOS sensor according to an embodiment of the present invention acquires an image according to the exposure time when the camera is exposed to the outside.
FIG. 2A is a block diagram schematically showing the configuration of an exposure time securing apparatus for an image sensor according to an embodiment of the present invention.
FIG. 2B illustrates a method of transmitting image data according to an exemplary embodiment of the present invention.
2C is a graph illustrating exposure time according to an embodiment of the present invention.
3 is a block diagram schematically illustrating the configuration of an exposure angle measuring unit included in the exposure time securing apparatus of an image sensor according to an embodiment of the present invention.
4A and 4B illustrate a specific method for securing an exposure time according to an exposure angle of an exposure time securing apparatus of an image sensor according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS The advantages and features of the present invention and the manner of achieving them will become apparent with reference to the embodiments described in detail below with reference to the accompanying drawings. The present invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete, and will fully convey the scope of the invention to those skilled in the art. Is provided to fully convey the scope of the invention to those skilled in the art, and the invention is only defined by the scope of the claims. Like reference numerals refer to like elements throughout the specification.

Unless defined otherwise, all terms (including technical and scientific terms) used herein may be used in a sense commonly understood by one of ordinary skill in the art to which this invention belongs. Also, commonly used predefined terms are not ideally or excessively interpreted unless explicitly defined otherwise.

The terms " first ", " second ", and the like in the present specification are for distinguishing one element from another element, and the scope of the right should not be limited by these terms. For example, the first component may be referred to as a second component, and similarly, the second component may also be referred to as a first component.

In the present description, the identification codes (e.g., a, b, c, etc.) in each step are used for convenience of explanation, and the identification codes do not describe the order of each step, Unless you specify a specific order, it can happen differently from the order specified. That is, each step may occur in the same order as described, may be performed substantially concurrently, or may be performed in reverse order.

As used herein, the expressions " have, " " comprise, " " comprise, " or " comprise may " refer to the presence of a feature (e.g., a numerical value, a function, And does not exclude the presence of additional features.

In addition, the term 'portion' as used herein refers to a hardware component such as software or a field-programmable gate array (FPGA) or an ASIC, and 'part' performs certain roles. However, 'part' is not meant to be limited to software or hardware. &Quot; to " may be configured to reside on an addressable storage medium and may be configured to play one or more processors. Thus, by way of example, 'parts' may refer to components such as software components, object-oriented software components, class components and task components, and processes, functions, , Subroutines, segments of program code, drivers, firmware, microcode, circuitry, data structures and variables. The functions provided in the components and components may be further combined with a smaller number of components and components or further components and components.

FIGS. 1A through 1E illustrate that the CMOS sensor according to an embodiment of the present invention acquires an image according to the exposure time when the camera is exposed to the outside.

Referring to FIGS. 1A and 1B, a CMOS sensor and a CMOS sensor, which are one type of image sensor, are exposed to the outside, and pixels included in the CMOS sensor read out an image. The CMOS sensor includes a plurality of pixels . As the CMOS sensor is exposed to sunlight or reflected light from an object, each pixel can be exposed to light and output to a digital value via an analog to digital converter (ADC) to read out the image data.

 Image sensors are semiconductors that convert light energy into electrical energy and display it on a display or store it in a storage device. There are various types of image sensors generally used, such as a charge coupled device (CCD), a complementary metal oxide semiconductor (CMOS), a contact image sensor (CIS), and the like.

A CCD image sensor is a sensor that converts light into electric charge to acquire an image. It is composed of a circuit in which a plurality of condensers are interconnected in pairs. Each capacitor in the circuit carries charge accumulated in its surrounding accumulator do. Specifically, a CCD chip is a chip in which a lot of photodiodes are gathered. When a light is irradiated to each photodiode, the CCD image sensor generates electrons according to the amount of light, reconstructs an amount of electrons representing the brightness of each of the photodiodes, Information.

The CMOS image sensor, which uses a photodiode in the same way as a CCD image sensor, but has different manufacturing methods and methods for reading signals, is a CMOS solid-state image sensor. The CMOS image sensor can be manufactured in a small size and has low power, so it is widely used in various fields such as mobile phones. The CMOS image sensor has an amplifier for each unit cell and has a feature that electric noise is less generated by reading the photo-converted electric signal. CMOS image sensor A CMOS image sensor is composed of several pixels. The pixels constituting the CMOS image sensor are composed of one photodiode and one pixel. And may be composed of four transistors.

Referring to FIG. 1C, the CMOS image sensor is an analog-to-digital converter (ADC) which is exposed to light reflected from an object and collected during a period of time after a frame overhead time (FOT) It is possible to READOUT the image data expressed in digital form. That is, the CMOS sensor can read out the image data, and the CMOS sensor can be exposed to external light and read out again.

However, here, the CMOS sensor reads the image data for one frame, analyzes the value of the image data, and determines a proper exposure time. In other words, the image can be obtained by exposing the CMOS sensor to light during the exposure time analyzed by the image data of 100 ms or earlier. However, in general, the image data acquired by the CMOS sensor is a real- Since a delay of more than 100 ms has occurred, it is not appropriate to apply the analyzed exposure time to the current image.

Referring to FIG. 1D, when the CMOS sensor is exposed for a long time from light incident from the outside, the pixel values included in the CMOS sensor may be saturated, and the image data may be entirely brightened by the saturated pixel values. Specifically, the bright portions of the image data are equally bright, making it difficult to distinguish between a portion that is slightly bright or a portion that is bright.

In this case, since the amount of light incident on the CMOS sensor is large, the pixel values included in the image sensor are saturated and the image may appear more whiter.

On the other hand, referring to FIG. 1E, in contrast to FIG. 1D, when the image sensor is briefly exposed from the light incident from the outside, it is difficult to distinguish between a part that is slightly dark and a part that is dark.

Accordingly, when the airplane suddenly rotates and looks at a dark place where no light is shining, the amount of light incident on the CMOS sensor is small, so that pixel values included in the image sensor may appear darker.

FIG. 2A is a block diagram schematically showing the configuration of an exposure time securing apparatus for an image sensor according to an embodiment of the present invention.

FIG. 2B illustrates a method of transmitting image data according to an exemplary embodiment of the present invention.

2C is a graph illustrating exposure time according to an embodiment of the present invention.

2A, an exposure time securing apparatus 200 for an image sensor includes an image sensor unit 210, an exposure angle measuring unit 230, an image processing unit 220, an exposure time obtaining unit 240, a memory 250, And a control unit 260.

The apparatus 200 for securing the exposure time of the image sensor according to an exemplary embodiment of the present invention may be a drone, a robot, or a missile, which is an unmanned aerial vehicle for acquiring images using an image sensor. However, the present invention is not limited thereto. The apparatus 200 for securing the exposure time of the image sensor described above may be configured to acquire images using an image sensor while moving It can be applied to all devices.

In addition, the apparatus 200 for securing the exposure time of the image sensor according to another embodiment of the present invention may be mounted on the outside of a moving object such as a drone, a robot, or a missile, which is an unmanned aerial vehicle.

The image sensor included in the image sensor unit 210 is exposed to external light, and light is incident on the image sensor 210, so that the image sensor unit 210 can acquire image data from the incident light.

The image sensor unit 210 according to an exemplary embodiment of the present invention may be an image sensor such as the CCD or the CMOS CIS described above. The image sensor unit 210 may be configured to reflect light or an object When exposed to light, the preprocessing unit included in the image sensor unit 210 converts the light incident on each pixel included in the image sensor unit 210 into an electrical signal, amplifies and compresses the converted electrical signal, and preprocesses . In addition, the analog-to-digital converter included in the image sensor unit 210 may convert the preprocessed analog signal into a digital signal. Therefore, each pixel included in the image data, which is a set of pixels obtained from the image sensor unit 210 and composed of pixel values, according to an embodiment of the present invention, can be represented by digital values such as 8BIT, 10BIT, 12BIT, have.

The image processing unit 220 may calculate a representative value from the pixels included in the image data obtained from the image sensor unit 210.

2B, the image sensor unit 210 and the image processing unit 220 according to an exemplary embodiment of the present invention may convert image data obtained from the image sensor unit 210 into low-voltage differential signaling LVDS). Specifically, FIG. 2B shows that the image processing unit 220 receives 8-bit, 10-bit, and 12-bit pixel data using a low voltage difference signal, respectively.

A method of transmitting signals by LVDS is a method of receiving image data by comparing two voltage differences transmitted from a receiving side by transmitting two different low voltages at a transmitting side in order to transmit digital information at high speed through a copper line, According to an embodiment of the present invention, the image sensor unit 210 may transmit two different low voltages, and may compare the two voltage differences in the image processing unit 220 to receive image data with noise removed at a high speed .

Accordingly, when the image processing unit 220 receives the image data from the image sensor unit 210 using the LVDS according to an embodiment of the present invention, the image data can be received quickly and the low voltage signal is used, And since the difference signal is used, the noise-eliminated image data can be received.

Referring again to FIG. 2A, the image processing unit 220 according to an exemplary embodiment of the present invention may include a representative (representative) including all values that can be statistically calculated from the pixels included in the image data acquired from the image sensor unit 210 Value can be calculated.

The representative value according to another exemplary embodiment of the present invention is a representative value of the image data of the pixels included in the image data obtained from the image sensor unit 210 as the light is incident on the image sensor included in the image sensor unit 210 May be an average value of the image brightness determined from the pixels operable to represent the image. Specifically, the image processing unit 220 can read the pixel values converted into the digital values included in the image data obtained from the image sensor unit 210, and process the read pixel values to determine an average value. The read pixel values described above may be read only in pixel values that are operable to represent image data, but are not limited thereto and may read pixel values in various combinations.

The exposure angle measuring unit 230 measures an angle between the predetermined reference direction and the direction of light incident on the image sensor unit based on the attitude information of the apparatus for securing the exposure time of the image sensor mounted with the sensed image sensor using the attitude information sensor. Can be measured.

A specific method by which the exposure angle measuring unit 230 measures the exposure angle will be described later with reference to FIG.

The exposure time obtaining unit 240 measures an exposure time which is a time when the image sensor unit 210 is exposed to light so that the representative value calculated by the image processing unit 220 becomes a preset data value, Can be obtained according to the exposure angle.

The predetermined data value may be an intermediate value indicating the number of the pixels included in the image data obtained from the image sensor unit 210, the number of which corresponds to the middle of the number of colors. More specifically, according to an embodiment of the present invention, the size of the image data obtained from the image sensor unit 210 may be 640 X 480, and the number of pixels is 307200 when the image size is 640 X 480 , And each pixel is 12 bits. If the pixel is 12 bits, the pixel can represent 4096 (= 212) colors, and the above-mentioned intermediate value may be 2048, or a value close to 2048. [

Referring to FIG. 2C, FIG. 2C is a diagram illustrating an example of a case where an image size obtained from the image sensor unit 210 according to an exemplary embodiment of the present invention is 640 X 480 and each pixel is 12 bits, And the exposure time is 20 ms, which is the median value. The exposure time is 33 ms. Accordingly, the exposure time acquisition unit 240 may acquire the exposure time 33 ms to make the average value calculated by the image processing unit 220 2048, which is the predetermined data value, to be 33 ms according to the exposure angle measured by the exposure angle measurement unit 230 . However, the above-described image size and the BIT of the pixels are examples for explaining one embodiment of the present invention, but the present invention is not limited thereto and may be various image sizes and the number of pixels.

Referring again to FIG. 2A, the memory 250 may store the exposure time acquired in the exposure time acquiring unit 240 according to the exposure angle.

The controller 260 according to an exemplary embodiment of the present invention may control the image processing unit 220 to continuously add brightness values of pixels included in one frame of image data and to provide an average value.

The control unit 260 may control the exposure angle measuring unit 230 to measure an exposure angle which is an angle between a preset reference direction and a direction of light incident on the image sensor unit 210, Can be controlled to obtain the exposure time according to the exposure angle measured in real time.

The control unit 260 may control the exposure time acquiring unit 240 to acquire the exposure time according to the exposure angle measured in real time so that the average value of the image brightness can be continuously maintained at the intermediate value of the data.

The control unit 260 can compare the exposure time acquired according to the exposure angle stored in the memory 250 and the exposure time acquired according to the exposure angle measured in real time, The control unit 260 can update the exposure time stored in the memory 250 to the exposure time measured in real time.

The controller 260 according to an embodiment of the present invention can update the exposure time stored in the memory 250 in real time through the synchronous serial communication. Synchronous serial communication is a method of using a separate clock line to indicate that data is transferred when data is transmitted, so that the communication partner can share the clock line to know when data is coming into the data line. I2C, SPI (Serial Peripheral Interface) communication is a kind of synchronous communication method, and SPI (Serial Peripheral Interface) communication is a serial communication of data, It is a kind of communication protocol that allows information to be known.

According to an embodiment of the present invention, the control unit 260 may update the exposure time stored in the memory 250 in real time through the SPI communication, but the present invention is not limited thereto. It is possible to update the exposure time stored in the storage unit 250 with the exposure time measured in real time.

According to another embodiment of the present invention, when the device 200 securing the exposure time of the image sensor mounted with the image sensor is suddenly changed as the exposure time securing device 200 of the image sensor moves , The control unit 260 can control the exposure angle measuring unit 230 to measure the current exposure angle based on the posture information of the exposure time securing apparatus 200 on which the image sensor with the above described posture information changed is mounted. The control unit 260 may determine the exposure angle corresponding to the measured exposure angle while changing the posture information among the exposure angles stored in the memory 250. The controller 260 may control the exposure time corresponding to the determined exposure angle to the memory 250, Can be loaded. In addition, the controller 260 may control the image sensor unit 210 to be exposed to light during the exposure time loaded from the memory 250.

3 is a block diagram schematically illustrating the configuration of an exposure angle measuring unit included in the exposure time securing apparatus of an image sensor according to an embodiment of the present invention.

3, the exposure angle measuring unit 230 may include a plurality of attitude information sensor units 231, an attitude information sensor fusion unit 232, a filter unit 233, and an attitude information calculating unit 234 have.

The plurality of posture information sensor units 231 can collect measurement data for an apparatus that secures the exposure time of the image sensor on which the image sensor is mounted through at least two posture information sensors.

In an embodiment of the present invention, the apparatus for securing the exposure time of the image sensor may be a drone, a robot, or a missile, which is an unmanned aerial vehicle for acquiring images using an image sensor. In the present specification, The present invention is not limited thereto and can be applied to various apparatuses that acquire images using an image sensor while moving.

The plurality of attitude information sensor units 231 may include a first attitude information sensor unit 231a and a second attitude information sensor unit 231b according to an embodiment of the present invention, The number of sensor portions may be more than two.

The first attitude information sensor unit 231a is constituted by one three-axis accelerometer sensor and can collect the acceleration values (X, Y, Z) of the X axis, Y axis and Z axis of the exposure time securing apparatus, The sensor unit 231b can collect the angular velocity values (p, q, r) of the exposure time securing apparatus from the two biaxial gyro sensors.

The attitude information sensor fusion unit 232 can obtain state variables for the attitude of the exposure time securing apparatus by fusing and processing data collected from the attitude information sensor units 231a and 231b.

Specifically, the attitude information sensor fusion unit 232 can determine the rate of change of the Euler angle of the image sensor unit using the relational expression between the angular velocity of the moving time securing apparatus and the Euler angle, The angle can be calculated. The attitude information sensor fusion unit 232 acquires the Euler angles calculated on the basis of the attitude information sensor fusion based on the rotation angle pie? In the X axis direction, the rotation angle theta? In the Y axis direction, Can be determined as a state variable composed of the rotation angle ψ.

Here, the attitude information sensor fusion unit 232 may use a series of digital filtering algorithms to supplement the shortcomings of the individual attitude information sensors and output accurate dynamic attitude information. That is, the purpose of the attitude information sensor fusion is to input data collected from the attitude information sensor units 231a and 231b, apply digital filtering, complement each other's values, and output accurate and useful dynamic attitude result values .

The posture information sensor fusion unit 232 can obtain the rate of change of the Euler angle by the following equation (1).

[Equation 1]

는 오일러 각의 변화율이다. Here, (p, q, r) is the value of each speedometer of the exposure time securing device on which the image sensor is mounted,

Is the rate of change of the Euler angles.

And, to be obtained for the state variables (Φ _t, θ _t, Ψ _t) represented by the Euler angles of the Equation (2) by integrating the rate of change of the above-described Eulerian angles.

&Quot; (2) "

Filter unit 233 is to apply a filter to estimate the state of the non-linear yeokhakgye for the state variables (Φ _t, θ _t, Ψ _t) is determined from Equation (2) to estimate the state information of the exposure time maintenance apparatus have. For this, the filter unit 233 treats the noise as a random variable in the measurement data collected from the plurality of attitude information sensor units 231, and obtains the state variable such that the expected value of the state information estimation error is minimized by the filter have.

Specifically, the exemplary filter unit 233 according to an embodiment of the present invention may be applied to the extended Kalman filter for the state variables (Φ _t, θ _t, Ψ _t).

The Kalman filter is a recursive filter that tracks the state of a linear dynamics system that contains noise. Specifically, it is a filter that recursively processes input data including noise, and can perform optimal statistical prediction of the current state. However, the Kalman filter is basically a filter designed on the assumption that the system is linear. When applied to a nonlinear system, there is a problem that the error becomes rather large or divergence occurs. Accordingly, in an embodiment of the present invention, an extended Kalman filter having a relatively small amount of calculation and capable of state estimation in a nonlinear system can be applied.

The Kalman gain can be calculated by obtaining a predictive value and a prediction error covariance of the state information of the exposure time securing apparatus through the extended Kalman filter. The state information of the exposure time securing apparatus is estimated using the calculated Kalman gain and the predicted value , The error covariance can be obtained using the Kalman gain and the prediction error covariance.

The filter unit 233 updates the pie? And theta? Obtained from the three-axis accelerometer among the measured values measured from the plurality of attitude information sensor units 231, obtains a state variable according to the updated measurement value , And the process of estimating the state information of the image sensor unit by applying the extended Kalman filter is repeated.

Here, the extended Kalman filter will be described in detail as follows.

The extended Kalman filter has a state change model and a measurement model as shown in Equation (3).

&Quot; (3) "

와

는 평균이 0이고, 분산이 각각

,

인 독립 가우시안 백색 잡음이다. 마르코프 프로세스는 상태벡터가 결정될 확률이 과거의 다른 상태 벡터들과 무관하고 그 직전의 결과인 X_t-1에 따라 결정된다는 가정이다. 다음은 수학식 4는 상기 상태방정식에 대한 확장 칼만 필터 알고리즘이다.Where X _t is a state vector following the Markov process,

Wow

The average is 0, and the variance is

,

Independent Gaussian white noise. The Markov process assumes that the probability that the state vector is determined is independent of other state vectors of the past and is determined according to X _t-1 , which is the result of the previous state vector. Equation (4) is an extended Kalman filter algorithm for the above state equation.

&Quot; (4) "

.는 평균값,

는 분산값,

는 제어값,

는 측정값,

는 칼만 게인, 확장 칼만 필터는 t-1 순간까지의 관측값에 근거하여 X(t|t-1)을 추정하며 알고리즘은 반복형으로 표현된다.

는 시스템에 대한 노이즈의 공분산 값이고

는 센서 측정 값에 대한 노이즈의 공분산 값이다. here,

The average value,

Is a variance value,

Is a control value,

Lt; / RTI >

The Kalman gain and the extended Kalman filter estimate X (t | t-1) based on the observations up to the instant t-1, and the algorithm is expressed in iterative form.

Is the covariance of the noise for the system

Is the covariance of the noise to the sensor measurement.

(10-1) and (10-2) are predictive parts and the equation (10-3) expresses Kalman gain. The physical meaning of the Kalman gain is that the larger the number is, the more accurate the measured value, and the smaller the number, the better the predicted value.

(10-4), (10-5) is the part that updates the measured value. The extended Kalman filter can be applied to nonlinear systems because the Taylor series and Jacobian are applied to the extended Kalman filter. The Taylor series linearizes the nonlinear system, and Jacobian creates a simple approximation of the line by approximating complex entangled formulas using derivatives. The extended Kalman filter can be applied to nonlinear systems, so the Euler angles are used without the use of a quaternion coordinate system.

In the case where the light incident on the image sensor unit included in the exposure time securing apparatus of the image sensor according to the embodiment of the present invention is directly incident from the sea, the image sensor unit looks at the sky. In an embodiment of the present invention, (Theta) with respect to the Y-axis direction, the rotation angle [theta] (theta) with respect to the Y-axis direction when the image sensor portion looks at the ground when the light reflected from the tank is incident, Only the rotation angle [theta] (theta) with respect to the Y axis direction among the rotation angles [psi] between the Z axis directions can be obtained in real time. However, depending on the special situation or environment, the rotation angle ψ (psi) or the rotation angle Φ (pie) with respect to the X axis direction in the Z axis direction can be acquired in real time. . ≪ / RTI >

Therefore, the state variable for acquiring the posture information of the apparatus for securing the exposure time of the image sensor on which the image sensor is mounted according to the embodiment of the present invention includes the rotation angle? (Pi) with respect to the X axis direction, The rotation angle? (Theta), and the rotation angle? (Pivot) with respect to the Z-axis direction.

And the above-described first equation (10-1) can be applied to the relation of the angular speed and Euler angles, but defense type, the first type to predict the extended Kalman filter 10-1 is _t u is not deleted is part of controlling And can be expressed by the following equation (5).

&Quot; (5) "

The above equation can be solved with an Euler angle representing the posture of the exposure time securing device of the image sensor.

를 적분함으로서 구할 수 있고, 이 식을 적분하면 이미지 센서의 노출시간 확보 장치의 위치 정보를 나타내는 상태모델(state model)이 될 수 있다. 단 상술한 예시는, 본 발명의 일 실시 예를 설명하기 위한 예시일 뿐 이에 한정되는 것은 아니며, θ 각도만을 고려하고 Φ와 ψ는 사용하지 않는 경우로 설정할 수 있고, 또한, Φ, θ 및 ψ 각도 모두를 고려할 수도 있다.To explain one embodiment of the present invention, only Φ and θ angle of the exposure time securing apparatus of the image sensor are considered, and ψ is set to the case of not using it. The angles? And?

, And integrating this equation can be a state model representing the position information of the exposure time securing device of the image sensor. However, the present invention is not limited to this, and it is possible to set Φ and ψ not to use Φ, θ and ψ in consideration of θ angle only, You can also consider both angles.

통용될 수 있다. 여기서, dt는 미소시간을 나타낸다.The posture information state model of the exposure time securing apparatus of the image sensor is as shown in Equation (1), and the equation obtained by integrating Equation (1) is the same as Equation (2). In the equations (1) and (2), the following equation (5)

Can be used. Here, dt represents the minute time.

The attitude information measurement model of the image sensor section is expressed by the following Equation (6).

&Quot; (6) "

Since Equation (6) is linear, it can be used as a measurement model as shown in Equation (7) without using Jacobian.

&Quot; (7) "

Then, the measurement model H is obtained by the following equation (8).

&Quot; (8) "

Jacobian is used to linearly approximate the system through differentiation, Jacobian can be obtained as shown in equation (9).

&Quot; (9) "

을 구한다.Using the Jacobian obtained above, the predicted covariance value of the extended Kalman filter is calculated using the second equation (10-2) below.

.

The filter unit 233 can calculate the measurement model as the estimated value using the Kalman gain K _t using equation (10-3), and obtain the covariance value of the attitude information sensor measurement noise using the predicted covariance and the Kalman gain.

을 구한 뒤 다시 이 값을

값으로 입력하여 확장 칼만 필터 알고리즘을 반복적으로 수행한다.The filter unit 233 starts from START and finally

And then the value

Value to perform the extended Kalman filter algorithm iteratively.

In addition, it can be confirmed that the gyro and accelerometer value, which are the updated attitude information sensor values, are updated in the course of performing, and the Jacobian value can be continuously updated.

The posture information calculation unit 234 can calculate the posture information according to the movement of the exposure time securing apparatus of the image sensor in real time based on the state information of the exposure time securing apparatus of the image sensor.

Accordingly, the exposure angle measuring unit can measure an exposure angle, which is an angle between a preset reference direction and a direction of light incident on the image sensor unit based on the real-time calculated posture information in accordance with the movement of the exposure time securing apparatus of the image sensor. For example, it is possible to calculate in real time how much the rotation angle [theta] with respect to the Y-axis direction has changed as the posture of the exposure time securing device is changed according to the movement of the exposure time securing device of the image sensor. However, the present invention is not limited thereto. For example, the rotation angle? (Psi) with respect to the Z-axis direction or the rotation about the X-axis direction It is possible to calculate in real time how much Φ (pi) has changed. The X-axis, the Y-axis, and the Z-axis according to an embodiment of the present invention are respectively in a right angle relation.

4A and 4B illustrate a specific method for securing an exposure time according to an exposure angle of an exposure time securing apparatus of an image sensor according to an embodiment of the present invention.

4A and 4B, an apparatus for securing exposure time of an image sensor according to an embodiment of the present invention is a drone 400, which is an unmanned aerial vehicle, and the above- The apparatus for securing the exposure time of the image sensor may be a robot or a missile and may be applied to all devices that acquire images using an image sensor while moving.

4A, the coordinate axes of the drones 400 on which the image sensor 410 is mounted are as shown in FIG. 4A, and the state variables of the drones 400 on which the image sensors 410 are mounted are as shown in FIG. (?) With respect to the Y-axis direction, and a rotation angle? With respect to the Z-axis direction, as described in the first embodiment of the present invention, In the case of the drone 400 mounted with the image sensor 410 according to the present invention, Φ is a roll angle, θ is a pitch angle when the Y axis is rotated, and ψ is a Yaw angle .

Accordingly, the attitude information of the drones 400 on which the image sensor 410 is mounted according to an embodiment of the present invention can be obtained by using the data information collected from the accelerometer sensor and the gyro sensor to calculate pitch, yaw and roll values Can be obtained.

The accelerometer sensor and the gyro sensor according to an exemplary embodiment of the present invention may be used as a low-cost sensor of a microelectromechanical systems (MEMS) type, and may include an SPI communication interface among synchronous communication methods, but the present invention is not limited thereto .

Referring to FIG. 4B, the drone 400, which is an unmanned aerial vehicle according to an exemplary embodiment of the present invention, can take off and can take off from the accelerator sensor and the gyro sensor included in the take- The attitude information of the drones 400 can be obtained. 3, the exposure angle measuring unit included in the drone 400 in which the attitude information is acquired is calculated from the attitude information of the drone 400 on which the image sensor 410 is mounted, the angle (Roll angle) of? (Pitch angle) and the angle of? (Yaw angle).

4B illustrates a case where only the angle (pitch angle) of the drones 400 on which the image sensor 410 is mounted is calculated and only the angle (pitch angle) of the angle? Is used For example, although the above-described exemplary embodiments are described for illustrative purposes only, the present invention is not limited thereto. In order to obtain a more accurate exposure time, the exposure angle measuring unit may include a drones 400 (Roll angle) of the angle? And the angle (Yaw angle) of? From the attitude information of the angle?

The angle of rotation with respect to the Y-axis direction according to an embodiment of the present invention may be determined by gravity or weight applied to the drones 400 on which the image sensor 410 is mounted, . ≪ / RTI >

The predetermined reference direction according to an embodiment of the present invention is a relationship in which the X-axis, the Y-axis, and the Z-axis direction are all vertical, the Z-axis direction is perpendicular to the ground, and the gravity applied to the drones 400 (Pitch angle) with respect to the Y-axis direction of the drones 400 on which the image sensor 410 is mounted. That is, the direction of the light incident on the image sensor 410 at zero degree of rotation angle (Pitch angle) indicates a preset reference direction. If the dragon (400) is flying in the sky with the same gravity and lift, it is indicated as horizontal flight.

When the image sensor 410 mounted on the drone 400 according to an embodiment of the present invention looks at the solution 430 at an angle of? 1, the light incident on the image sensor 410 can not be large. Accordingly, when the image sensor 410 views the solution 430 located in the sky at an angle of? 1, the exposure time for exposure of the image sensor 410 to light must be reduced, and when the image sensor 410 detects the angle? It is necessary to increase the exposure time unlike the case of looking at the solution 430 because the image sensor 410 receives the light reflected from the tank.

The method of measuring the exposure angle and the exposure time measurement unit of FIGS. 2 to 3 have been described so that the representative value calculated by the image processing unit is a preset data value, so that a description thereof will be omitted in the present embodiment.

Accordingly, in an embodiment of the present invention, the amount of light incident on the image sensor 410 varies depending on the weather, so that when the posture of the drones 400 on which the image sensor 410 is mounted is suddenly changed, It is possible to perform the test process by taking off the drone 400 so that the proper exposure time can be immediately applied according to the previously secured exposure angle without calculating the exposure time. Specifically, first, a test process is performed in which the proper exposure time for the image sensor 410 to be exposed to light according to the Θ angle (pitch angle) is obtained by changing the Θ angle (pitch angle) after taking off the drone 400, Can be performed.

Table 1 below shows the exposure time obtained according to the exposure angle at which the image sensor 410 is exposed to light.

Pitch angle Exposure time 20 degrees 18ms 21 degrees 18.2 ms 22 degrees 18.5 ms 23 degrees 19ms 24 degrees 19.1ms 25 degrees 19.5ms 26 degrees 19.7ms

The control unit may control the exposure time acquisition unit to store the exposure time according to the acquired pitch angle by changing the pitch angle of the drone 400 on which the image sensor 410 is mounted as shown in Table 1, Therefore, when the posture of the drone 400 on which the image sensor 410 is mounted is changed suddenly as described later, the controller directly applies the exposure time according to the pitch angle stored in the memory to the image sensor 410, It is possible to directly control the time during which the light source 410 is exposed to the external light without performing the arithmetic processing by the image processing unit.

According to another embodiment of the present invention, the controller included in the drone 400 may control the exposure angle measuring unit to measure the exposure angle of the image sensor unit in real time.

The drones 400 according to an embodiment of the present invention may include a low-cost sensor of a microelectromechanical systems (MEMS) type, an accelerometer sensor and a gyro sensor, and may take an interface by SPI communication among synchronous communication methods But is not limited thereto. Accordingly, it is possible to acquire the attitude information of the drones 400 using the attitude information sensor in real time to measure the exposure angle, and to obtain the exposure time according to the exposure angle.

In general, when the gyro sensor or accelerometer sensor described above is used, the data of the attitude information of the drones 400 on which the image sensor 410 is mounted can be updated at a speed of at least 1 KHz or more in real time. Also, the update rate of image data received through the CMOS sensor among commonly used image sensors can be about 30 Hz. Accordingly, the update rate of the attitude information of the drones 400 acquired through the attitude information sensor is much faster than the speed of updating the image data determined through the image sensor unit. Therefore, by using the accelerometer sensor and the gyro sensor included in the drone 400 according to the embodiment of the present invention, the attitude information of the dron 400 on which the image sensor 410 is mounted can be obtained faster than the image data update, The user can quickly recognize where the image sensor 410 mounted on the camera 400 is looking.

Accordingly, the exposure time obtaining unit according to an embodiment of the present invention obtains in real time the exposure time of the image sensor 410 according to the pitch angle of the drone 400 on which the image sensor 410 is mounted, It is possible to compare the exposure time according to the stored pitch angle with the exposure time obtained according to the Pitch angle stored in advance in the memory to update the changed exposure time among the obtained exposure time according to the memory stored Pitch angle in real time.

In addition, according to another embodiment of the present invention, when the drones 400 are moved and the posture information of the drones 400 on which the image sensor 410 is loaded is suddenly changed, It is possible to control the current exposure angle to be measured based on the attitude information of the drones 400 on which the image sensor 410 having been changed is mounted. The control unit can determine the exposure angle corresponding to the measured exposure angle while changing the posture information of the drones 400 on which the image sensor 410 is mounted among the exposure angles stored in the memory. In addition, the control unit can load an exposure time corresponding to the exposure angle determined by the above-described method among the exposure time according to the exposure angle stored in the memory, and can prevent the image sensor unit 410 from being exposed to light .

Specifically, according to another embodiment of the present invention, the drone 400 on which the image sensor 410 is mounted is stopped at a specific position in the air, and the exposure time of the image sensor 410 according to the exposure angle is determined by the above- And the drone 400 stopped at a specific position suddenly moves, so that the exposure time of the image sensor 410 according to the exposure angle can be obtained by the above-described method. Specifically, when the drone 400 is suddenly moved, the exposure angle measuring unit included in the drone 400 may measure a pitch angle change of the drone 400 that changes with time. For example, if the image sensor 410 mounted on the drone 400 is looking at the light incident from the outside at a Θ angle (pitch angle) of 30 degrees and the drone 400 is rotated at 10 deg / (Pitch angle), which is an exposure angle of the image sensor 410 mounted on the drone 400 and viewed from the outside, may be changed from 30 degrees to 50 degrees, If the stored exposure angle is 50 degrees, the exposure time can be immediately loaded, and the image sensor 410 can be controlled to be exposed to light during the loaded exposure time.

Therefore, when the conventional controller changes the posture of the drones 400 on which the image sensor 410 is mounted, the image sensor 410 is exposed to the light again to calculate a representative value from the pixels included in the image data The exposure time corresponding to the exposure angle stored in the existing memory is immediately applied to the image sensor 410 so as not to expose the image sensor 410 to external light during the obtained exposure time, The brightness of the data can be effectively controlled. Therefore, according to the embodiment of the present invention, it is possible to adjust the image so that the image can be seen clearly in real time.

It is to be understood that the present invention is not limited to these embodiments, and all elements constituting the embodiment of the present invention described above are described as being combined or operated in one operation. That is, within the scope of the present invention, all of the components may be selectively coupled to one or more of them. In addition, although all of the components may be implemented as one independent hardware, some or all of the components may be selectively combined to perform a part or all of the functions in one or a plurality of hardware. As shown in FIG. In addition, such a computer program may be stored in a computer readable medium such as a USB memory, a CD disk, a flash memory, etc., and read and executed by a computer to implement an embodiment of the present invention. As the recording medium of the computer program, a magnetic recording medium, an optical recording medium, or the like can be included.

It will be apparent to those skilled in the art that various modifications, substitutions and substitutions are possible, without departing from the scope and spirit of the invention as disclosed in the accompanying claims. will be. Therefore, the embodiments disclosed in the present invention and the accompanying drawings are intended to illustrate and not to limit the technical spirit of the present invention, and the scope of the technical idea of the present invention is not limited by these embodiments and the accompanying drawings . The scope of protection of the present invention should be construed according to the following claims, and all technical ideas within the scope of equivalents should be construed as falling within the scope of the present invention.

200: Exposure time securing device
210: image sensor unit
220:
230: Exposure angle measuring unit
240: exposure time obtaining unit
250: Memory
260:

Claims (12)

An apparatus for securing an exposure time of an image sensor,
An image sensor unit for acquiring image data using the image sensor;
An image processor for calculating a representative value from the pixels included in the image data obtained from the image sensor;
An exposure angle measuring unit for measuring an exposure angle, which is an angle between a predetermined reference direction and a direction of light incident to the image sensor unit, based on attitude information of the exposure time securing apparatus sensed using the attitude information sensor; And
And an exposure time acquiring unit for acquiring an exposure time, which is a time when the image sensor unit is exposed to the light so that the calculated representative value becomes a predetermined data value, according to the measured exposure angle. The method according to claim 1,
And a memory for storing the exposure time acquired according to the exposure angle. 3. The method of claim 2,
Wherein the control unit controls the exposure angle measuring unit to measure the exposure angle in real time and controls the exposure time acquiring unit to acquire an exposure time according to the exposure angle measured in real time so that the exposure time acquired according to the exposure angle stored in the memory And a controller for comparing the obtained exposure time with the exposure time measured in real time to update the exposure time stored in the memory. 3. The method of claim 2,
When the posture information is changed according to the movement of the exposure time securing device,
The exposure angle measuring unit controls to measure the current exposure angle based on the posture information of the exposure time securing apparatus in which the posture information is changed and determines an exposure angle corresponding to the currently measured exposure angle among the exposure angles stored in the memory And a control unit for loading an exposure time corresponding to the determined exposure angle into the memory and controlling exposure of the image sensor unit to the light during the loaded exposure time. The method according to claim 1,
Wherein the representative value is an average value of pixel values of pixels included in the image data acquired from the image sensor unit and operating according to the incident light. The method according to claim 1,
Wherein the predetermined data value is an intermediate value indicating a number of the pixels included in the image data in the middle of the number of colors. The method according to claim 1,
Wherein the exposure-
A plurality of posture information sensor units for collecting measurement data on posture of the exposure time securing apparatus through at least two posture information sensors;
An attitude information sensor fusion unit for acquiring state variables for the attitude of the exposure time securing device by fusing and processing the collected measurement data;
A filter unit for estimating state information on the attitude of the exposure time securing apparatus by applying a filter for estimating the state of the nonlinear dynamic system to the state variable; And
And an attitude information calculation unit for calculating attitude information according to a motion of the exposure time securing apparatus in real time based on the estimated state information. 8. The method of claim 7,
Wherein the plurality of posture information sensor units comprise:
A first posture information sensor unit for collecting acceleration values (X, Y, Z) of the X-axis, Y-axis and Z-axis of the exposure time securing device from one of the three-axis accelerometer sensors; And
And a second attitude information sensor unit for collecting angular velocity values (p, q, r) of the exposure time securing device from two biaxial gyro sensors. 9. The method of claim 8,
Wherein the posture information sensor fusion unit comprises:
Wherein the Euler angles of the Euler angles are obtained by integrating the change rates of the Euler angles using the relational expression between the angular velocity of the exposure time securing device and the Euler angles, The above Euler angles are obtained as the above-mentioned state variables composed of the rotation angle pie (?) With respect to the X-axis direction, the rotation angle theta (?) With respect to the Y-axis direction and the rotation angle? Wherein the exposure time securing device comprises: 10. The method of claim 9,
Wherein the predetermined reference direction is a relationship in which the X-axis, the Y-axis, and the Z-axis are all vertical, the Z-axis is perpendicular to the ground, and the rotation- ([theta]) angle (pitch angle) is 0 degree,
Wherein the exposure angle is the rotation angle theta (&thetas;). 9. The method of claim 8,
The filter unit includes:
Applying an extended Kalman filter to the state variable to measure a predicted value and a predicted error covariance for the state information of the exposure time securing apparatus and to calculate a Kalman gain and to calculate the exposure time using the Kalman gain and the predicted value Estimating the state information of the ensuring device, and obtaining the error covariance using the Kalman gain and the prediction error covariance. 12. The method of claim 11,
The filter unit includes:
(?) And theta (?) Obtained from the three-axis accelerometer sensor among the measurement data collected by the plurality of attitude information sensor units and updating the pie (?) And theta And estimating state information of the exposure time securing apparatus by applying a state variable of the time securing apparatus and applying the extended Kalman filter.

Images