KR101690887B1 - High-speed periodic signal reconstruction with compensation for rolling shutter effect using an off-the-shelf camera - Google Patents

Abstract

The system for restoring fast cyclic operation according to an embodiment is a system for restoring fast cyclic operation in one frame by randomly delaying the exposure time of frames obtained through arbitrary on / off switching from a single slow camera. An arbitrary delay unit for capturing a periodic signal in the same period; And a decompression unit for decompressing the collected information to a high-speed periodic signal by sub-Nyquist sampling, wherein the high-speed periodic operation is a high- motion, the rolling shutter effect caused by the read-out operation of the CMOS image sensor can be corrected by time-shifting the Fourier basis functions.

Description

BACKGROUND OF THE INVENTION 1. Field of the Invention [0001] The present invention relates to a high-speed periodic signal restoration and rolling shutter effect compensation method using a general camera,

The following description relates to a signal restoration technique, and relates to a super high-speed periodic signal restoration method and a rolling shutter effect compensation method using a general camera.

According to Nyquist sampling theory, the sampling rate to avoid loss of information should be at least twice the bandwidth of the signal. High-speed motions (such as hand blenders or electric fans that rotate more than 60 times per second) that can be encountered in real life can not be captured by conventional low-frame rate cameras (for example, 30 frames (fps)). In other words, a high-speed camera capable of covering a high sampling rate (at least 120 fps) is required to capture such fast motion. However, high-speed cameras are known to be relatively expensive, and short aperture times cause light inefficiency

Recently, in the area of image processing, there have been various studies on restoration of high-speed operation through a low-frame rate camera. For example, we proposed a high-speed image reconstruction method using several low-frame rate cameras. It should be noted that these methods require multiple cameras in order to obtain a suitable reconstruction. Conversely, Mergell et al. Attempted to capture high-speed signals through intentional aliasing caused by bandpass sampling, assuming that the operation in the scene is periodic. However, this method can not solve the light inefficiency problem because the operation is only exposed for a short time.

More recently, coded strobing pictures have been proposed as a method for sampling high-speed periodic signals based on random coded exposure. Since the randomly coded exposure technique does not affect the exposure time of the frame, it can overcome the light inefficiency problem by effectively restoring high-speed images. Nevertheless, the method requires a ferro-electric shutter which can not be used in existing products to control exposure with a high temporal resolution.

Furthermore, in the above method, a CCD image sensor is used for obtaining a high quality image because it is sensitive to light and strong against noise. However, in camera systems, CMOS image sensors with lower cost and lower power are used more widely. Unlike CCD image sensors, CMOS image sensors have different exposure times because of the sequential read-out characteristics of most CMOS image sensor arrays. Therefore, a rolling shutter effect, which causes geometric distortion of the captured image, necessarily occurs. The rolling shutter effect can be ignored if the motion is slow, but the rolling shutter effect will increase rapidly if the motion is very fast. Accordingly, when restoring high-speed motion with a low-frame rate camera composed of a CMOS image sensor, the rolling shutter effect should be considered.

A problem to be solved by the present invention is to use a random delay of camera exposure to recover a high-speed periodic signal by using a single low-speed camera.

A problem to be solved by the present invention is to provide a method for attenuating a rolling shutter effect caused by a characteristic of a CMOS imaging sensor when restoring high-speed periodic signals.

According to one embodiment, a method for recovering a fast cyclic operation may include the steps of (a) delaying the exposure time of frames obtained through arbitrary on / off switching from a single slow camera to a different time range Capturing a periodic signal within the same period; And a step of performing information collection on the captured signal and restoring the collected information into a high-speed periodic signal by sub-Nyquist sampling.

According to one aspect, a method for recovering the fast cyclic operation comprises:

The fundamental frequency of

And if the frame is exposed for E seconds,

The intensity observed during the exposure for an interval of time is calculated as: < EMI ID =

The

, And the '*' sign may represent a convolution operator.

Equation 5:

According to another aspect of the present invention, in the method for restoring the high-speed cyclic operation, the Fourier transform of Equation (5) is expressed by Equation (6) according to the characteristic of the Fourier transform, sinc (k) is a normalized sinc function,

Lt; / RTI >

Equation (6)

According to another aspect, a method of recovering the fast cyclic operation comprises:

Is expressed by Equation (7) by the Fourier series,

Represents a magnitude spectrum,

Lt; / RTI > represents the phase spectrum,

Lt; / RTI >

Equation (7)

According to another aspect, the method for restoring the high-speed cyclic operation includes:

The respective frequency components of

≪ RTI ID = 0.0 >

, And the signal < RTI ID = 0.0 >

Is formulated as shown in equation (8)

, And

Quot;

May be a periodic signal having the same fundamental frequency as that of the periodic signal.

Equation (8)

According to one embodiment, a method of attenuating a rolling shutter effect is a method of reducing read-out of a CMOS image sensor through time-shifting of Fourier basis functions in the course of restoring high-speed periodic motion. The rolling shutter effect generated by the operation can be corrected.

According to one aspect, the rolling shutter effect attenuation method includes:

To obtain the signal of the r-th row indicated by

The signal of the r < th > row at the time t indicated by

Wow

Can be formulated as a time shift, as shown in equation (9).

Equation (9)

According to another aspect of the present invention, the rolling shutter effect attenuation method is characterized in that the Fourier basis matrix B is composed of N orthogonal Fourier bases, and B is an N orthogonal Fourier bases function

Lt; / RTI >

, And

The

Lt; / RTI >

Represents the basis coefficient vector of the r-th row, and using Equation (9)

Can be expressed by Equation (10).

Equation (10)

According to another aspect of the present invention, there is provided a method for attenuating a rolling shutter effect,

Is the number of rows in one frame, and Equation (10)

Can be reconstructed using a time-shifted Fourier basis matrix, and a structured sparse reconstruction

And then,

The

Wow

By the vector product of < / RTI >

According to one embodiment, a system for restoring high-speed cyclic operation may be configured to adjust the exposure time of the frames obtained by switching arbitrary on / off from a single low-speed camera to a different time range by arbitrarily delaying the exposure time of the frames. A random delay unit for capturing a periodic signal within the same period; And a decompression unit for decompressing the collected information to a high-speed periodic signal by sub-Nyquist sampling, wherein the high-speed periodic operation is a high- motion, the rolling shutter effect caused by the read-out operation of the CMOS image sensor can be corrected by time-shifting the Fourier basis functions.

According to embodiments of the present invention, any delay between collection times of frames obtained through random switching can be used to restore fast cycle operation.

According to embodiments of the present invention, rolling shutter effect generated by the read-out operation of the CMOS image sensor can be ensured through time-shifting of Fourier basis functions in the restoration process of the high-speed cyclic operation. This allows geometric distortion due to the rolling shutter effect to be avoided when restoring high-speed cyclic operation.

Figure 1 illustrates an analysis of a proposed high-speed periodic signal restoration algorithm, according to one embodiment.
2 is a timing diagram of a CMOS sensor array according to an exemplary embodiment of the present invention.
3 is a block diagram illustrating a configuration of a fast cyclic operation restoration system according to an embodiment.
FIG. 4 is a flowchart illustrating a fast cyclic operation restoration method of the fast cyclic operation restoration system according to an embodiment.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings.

Coded strobing will be described to recover a high-speed periodic signal from a low-frame rate video. An image system combining a low-frame rate camera and a high-speed shutter can be used for this purpose. In the coded strobing imaging system, the problem of fast signal restoration can be formulated as a structured sparse reconstruction based on the camera observation model and the signal model.

(A) Camera observation model and signal model

Cycle

In period signal

In the given state (that is,

), sampling rate

Is higher than the Nyquist sampling rate, the signal can be accurately represented and reconstructed (i. E., Temporal resolution

The

being). If the total number of samples is N, then the sample is an N-dimensional vector

. ≪ / RTI > Instead of amplitude integration in the entire frame interval, an imaging system that performs amplitude modulation of incoming light values is used to generate an M-dimensional vector,

Can be obtained. Also, the number of measurements is usually expressed as x

Of the sample. The relationship between the sample x and the obtained intensity value y can be expressed as:

Equation 1:

here,

Is a court matrix representing the integration of randomly coded exposure and frame intervals,

Represents the observed noise. If the periodic signal

, The sample x can be expressed as a linear combination of Fourier basis as follows.

Equation 2:

Where B and s represent the Fourier basis matrix and the basis coefficient, respectively. By combining Equations (1) and (2), the observed intensity value y can be modeled by the following equation.

Equation (3)

In other words, since the observed intensity value y and the matrix A are known, a high-speed periodic signal can be restored by calculating the basis coefficient vector s when Equation (3) is solved.

(B) Restoration of high-speed periodic motion through structured sparse restoration

The high-speed motion reconstruction problem can be formulated into Equation 3, which is used to obtain the basis coefficient vector s. In addition to this, prior knowledge of the signal is essential to find a robust solution to s in equation (3). If the signal is periodic, the signal consists of a fundamental frequency component and a harmonic component. However, in reality, accurate periodic signals are difficult to exist due to noise, and quasi-periodic signals are generally observed. If it is a quasi-periodic signal, the frequency components are concentrated around the fundamental frequency and harmonics. In other words, quasi-periodic signals are rarely seen in the frequency domain (for example, the basis coefficient vector s is sparse). This knowledge can be used to apply sparsity to s to solve equation (3).

To represent a set of basic elements that a sparse vector s can have,

.

Has a fundamental frequency of

ego,

In the range of

All sparse vectors s are included in the above subset.

To calculate the sparse vector s, the so-called "structured sparse restoration" problem is formulated as follows.

Equation 4:

here,

Is a new vector of non-zero entries of s entries. The s value in Equation (4) is determined through the CoSaMP algorithm. Finally, the signal x (e.g., high-speed motion) can be simply calculated by equation (2).

However, even if the above method shows an effective high-speed motion reconstruction result, a ferro-electric shutter having a high-frame rate that can not be used in conventional cameras must be additionally provided. Also, the frame work based on the CCD image sensor does not take into account the rolling shutter effect encountered in a commercial CMOS image sensor.

According to one embodiment, a method of restoring a fast period / semi periodic signal without additional hardware by introducing a random exposure delay is proposed. It also explains the effect of introduced random delays. A method for correcting a rolling shutter effect caused by a read-out operation of a CMOS image sensor will be described.

(C) Restoration of high-speed periodic motion based on random delay

3, the high-speed cyclic operation system 300 may include a random delay unit 310 and a decompression unit 320, and steps 410 and 420 of FIG. 4 may include a random delay unit 310 And a restoration unit 320. [0034]

When an existing camera captures a high-speed periodic motion at a sub-Nyquist sampling rate, information loss is inevitable because of aliasing. In order to solve this problem, a high-speed signal reconstruction method based on random delay is proposed. Here, a random delay can be defined as an intentionally randomly generated delay between arbitrarily selected consecutive frames. The random delay unit 310 at step 410 has to randomly turn on / off the camera in order to apply a random delay to the existing camera system. In step 410, the random delay unit 310 may capture the different time ranges of a period by arbitrarily delaying the exposure time. In step 420, the decompression unit 420 continues to collect additional information on the periodic signal. Accordingly, a high-speed periodic signal is restored by sub-Nyquist sampling. . In terms of light inefficiency, the average exposure time (E) of a frame is the average exposure time of a coded strobing approach, although the proposed algorithm does not expose light during random delays. Which is several times longer than the random delay of a high-speed camera.

Figure 1 illustrates an analysis of a proposed high-speed periodic signal restoration algorithm, according to one embodiment. The left graph shows the time domain of the signal and the right graph shows the frequency domain of the signal. Periodic signal

The fundamental frequency of

(Fig. 1 (a)). If the frame is exposed for E seconds (Fig. 1 (c)),

The intensity observed during exposure for an interval of time can be calculated as

Equation 5:

here

The

And a '*' denotes a convolution operator. Depending on the nature of the Fourier transform, the Fourier transform (denoted by 'F') in equation (5) can be expressed as:

Equation (6)

Where sinc (k) is the normalized sinc function,

to be. signal

Is periodic, it can be expressed as follows by the Fourier series.

Equation 7

here,

Represents a magnitude spectrum,

Lt; / RTI > represents the phase spectrum,

to be. As shown in Equation (6)

The respective frequency components of

≪ RTI ID = 0.0 >

. Accordingly,

Can be formulated as follows.

Equation (8)

here

to be. finally,

The

(Fig. 1 (e) and Fig. 1 (f)). In other words,

If you know

Can be restored.

In this way, the object of the problem is that the periodic signal

Observation signal sampled from finding

.

Should be sampled higher than the Nyquist sampling rate to capture information without loss of information. signal

Can be recovered by the periodicity and the randomly delayed exposure technique, even if it is captured at the sub-Nyquist sampling rate.

For example, as shown in FIG. 1, the frame rate

The operating frequency is

Lt; / RTI &

The bandwidth of

.

, The Nyquist sampling theory is not satisfied. In order to meet this theory,

A shorter sampling interval is required. therefore

Six different samples are needed in one cycle.

However,

(1) can be regarded as the same value (= (g) in FIG. 1), since a certain point in one period can correspond to a sampled point in another period. When sampling three frames,

So one cycle is repeated four times. therefore

Three points are obtained at different points in the operation cycle of the light source (the red dot in (e) and (g) in FIG. 1).

Frame

(In other words,

Set of

, Three additional values can be observed (blue dot in (e) and (g) in FIG. 1). The more random delays, the more additional information is obtained and, as a result, the high-speed periodic signal is restored through sub-Nyquist sampling.

(D) Rolling shutter effect correction

As described above, the CMOS image sensor inherently has a rolling shutter effect caused by a read-out operation. Specifically, this effect occurs due to line-delays in which rows are exposed at different times, as shown in FIG. For example, if you want to get the information in the red box in Figure 2 (i.e.,

(I.e., the signal of the r-th row indicated by " 1 "),

The signal of the r-th row at the time t indicated by " r "

Wow

Is formulated as a time shift as follows.

Equation (9)

Since the Fourier basis matrix B is composed of N orthogonal Fourier bases, the basis matrix B is the N orthogonal Fourier bases function

Can be expressed as

to be. With the signal model of Equation (2)

The

Lt; / RTI >

Represents the basis coefficient vector of the r-th row. Thus, using equation (9)

Can be expressed as follows.

Equation (10)

here,

Is the number of rows in a frame. Equation (10)

Can be reconstructed using a time-shifted Fourier basis matrix. Through a structured sparse restore

Lt; / RTI >

The

Wow

By the vector product of < RTI ID = 0.0 >

The apparatus described above may be implemented as a hardware component, a software component, and / or a combination of hardware components and software components. For example, the apparatus and components described in the embodiments may be implemented within a computer system, such as, for example, a processor, a controller, an arithmetic logic unit (ALU), a digital signal processor, a microcomputer, a field programmable gate array (FPGA) , A programmable logic unit (PLU), a microprocessor, or any other device capable of executing and responding to instructions. The processing device may execute an operating system (OS) and one or more software applications running on the operating system. The processing device may also access, store, manipulate, process, and generate data in response to execution of the software. For ease of understanding, the processing apparatus may be described as being used singly, but those skilled in the art will recognize that the processing apparatus may have a plurality of processing elements and / As shown in FIG. For example, the processing unit may comprise a plurality of processors or one processor and one controller. Other processing configurations are also possible, such as a parallel processor.

The software may include a computer program, code, instructions, or a combination of one or more of the foregoing, and may be configured to configure the processing device to operate as desired or to process it collectively or collectively Device can be commanded. The software and / or data may be in the form of any type of machine, component, physical device, virtual equipment, computer storage media, or device , Or may be permanently or temporarily embodied in a transmitted signal wave. The software may be distributed over a networked computer system and stored or executed in a distributed manner. The software and data may be stored on one or more computer readable recording media.

The method according to an embodiment may be implemented in the form of a program command that can be executed through various computer means and recorded in a computer-readable medium. The computer-readable medium may include program instructions, data files, data structures, and the like, alone or in combination. The program instructions to be recorded on the medium may be those specially designed and configured for the embodiments or may be available to those skilled in the art of computer software. Examples of computer-readable media include magnetic media such as hard disks, floppy disks and magnetic tape; optical media such as CD-ROMs and DVDs; magnetic media such as floppy disks; Magneto-optical media, and hardware devices specifically configured to store and execute program instructions such as ROM, RAM, flash memory, and the like. Examples of program instructions include machine language code such as those produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter or the like. The hardware devices described above may be configured to operate as one or more software modules to perform the operations of the embodiments, and vice versa.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments. For example, it is to be understood that the techniques described may be performed in a different order than the described methods, and / or that components of the described systems, structures, devices, circuits, Lt; / RTI > or equivalents, even if it is replaced or replaced.

Therefore, other implementations, other embodiments, and equivalents to the claims are also within the scope of the following claims.

Claims (10)

A method for restoring fast cycle operation,
Capturing a periodic signal in the same period in different time ranges by arbitrarily delaying the exposure time of the frames obtained through the arbitrary switching on / off from a single low-speed camera; And
Performing information collection on the captured signal, and restoring the collected information into a high-speed periodic signal by sub-Nyquist sampling
Wherein the high speed cyclic operation restoration method comprises: The method according to claim 1,
Periodic signal

The fundamental frequency of

And if the frame is exposed for E seconds,

The intensity observed during the exposure for an interval of time is calculated as: < EMI ID =

The

Represents a rectangular function of 1 in the range of " a ", and " * " represents a convolution operator
Wherein the high-speed cyclic operation restoration method comprises:
Equation 5:

3. The method of claim 2,
The Fourier transform of Equation (5) is expressed by Equation (6) by the characteristic of the Fourier transform,
Since sinc (k) is a normalized sinc function,

sign
Wherein the high-speed cyclic operation restoration method comprises:
Equation (6)

The method of claim 3,
Periodic signal

Is expressed by Equation (7) by the Fourier series,

Represents a magnitude spectrum,

Lt; / RTI > represents the phase spectrum,

sign
Wherein the high-speed cyclic operation restoration method comprises:
Equation (7)

The method of claim 3,
In Equation (6)

The respective frequency components of

≪ RTI ID = 0.0 >

, And the signal < RTI ID = 0.0 >

Is formulated as shown in equation (8)

, And

Quot;

Which is a periodic signal having the same fundamental frequency as that of
Wherein the high-speed cyclic operation restoration method comprises:
Equation (8)

A method of attenuating a rolling shutter effect,
Correction of rolling shutter effect caused by read-out operation of CMOS image sensor through time shift of Fourier basis functions in restoring high-speed periodic motion
Wherein the rolling shutter effect is attenuated. The method according to claim 6,

To obtain the signal of the r-th row indicated by

The signal of the r < th > row at the time t indicated by

Wow

Is formulated as a time shift, as shown in equation (9)
R is the number of rows in one frame, and d is the line-delay between the rows.
Equation (9)

8. The method of claim 7,
The Fourier basis matrix B is composed of N orthogonal Fourier bases, and B is the N orthogonal Fourier bases function

Lt; / RTI >

, ≪ / RTI >
remind

The

Lt; / RTI >

Represents the basis coefficient vector of the r-th row, and using Equation (9)

(10)
Wherein the rolling shutter effect is attenuated.
Equation (10)

9. The method of claim 8,
remind

Is the number of rows in one frame, and Equation (10)

Lt; / RTI > can be recovered using a time-shifted Fourier basis matrix,
Through structured sparse restoration

And then,

The

Wow

Calculated by the vector product of
Wherein the rolling shutter effect is attenuated. A system for restoring high-speed cyclic operation,
An arbitrary delay unit for capturing a periodic signal in the same period in different time ranges by arbitrarily delaying the exposure time of frames obtained through switching of arbitrary on / off from a single low-speed camera; And
A restoration unit for performing information collection on the captured signal and restoring the collected information into a high-speed periodic signal by sub-Nyquist sampling;
Lt; / RTI >
In the process of restoring the high-speed periodic motion, the rolling shutter effect generated by the read-out operation of the CMOS image sensor is corrected through time-shifting of the Fourier basis functions
Wherein the high-speed periodic operation is performed by a computer.

Images