KR102517323B1 - Up-sampling system and method using multiple images - Google Patents

Abstract

An upsampling system and method using multiple images are provided. The up-sampling system using the multiple images may include: a video signal acquisition unit configured to acquire a plurality of video signals by capturing a plurality of images in different time zones; a video signal pre-processing unit configured to generate a plurality of classified normalized signals by performing pre-processing of the plurality of video signals; a delay information processor configured to obtain an optimal delay value of each of the normalized signals using the plurality of normalized signals and partial delay signals; and a video signal post-processing unit configured to output an up-sampling signal obtained by merging the plurality of normalization signals using the optimal delay values.

Description

Up-sampling system and method using multiple images {Up-sampling system and method using multiple images}

The present invention relates to an upsampling system and method using multiple images, and more particularly, multiple images in which upsampling is performed using a plurality of images captured using a low frame camera without using a vibration sensor or a high frame camera. It relates to a used upsampling system and method.

In general, in order to detect an abnormality in a machine, a vibration sensor is attached to measure the vibration, and when abnormal vibration occurs, it is used to determine that an abnormality has occurred in the machine.

Attaching the vibration sensor has the advantage of being able to easily determine the normal state and the abnormal state by using the size and shape of the vibration, but at the same time, it can be installed in a machine that is larger than the size that the vibration sensor can be attached to, However, there is a disadvantage that only acceleration is measured and measurement is possible only at one location.

In addition, since a non-designed mass is added to the machine, a significant interaction may occur in the operation of the machine, and there is a disadvantage that vibration of a standard size or more cannot be detected.

Therefore, in order to overcome these disadvantages, a system for detecting an abnormality of an object using an image capturing sensor such as a camera system is being used. However, when such a camera system is used, there is a problem in that it not only has a low shooting frame rate, but also increases the size of a signal compared to a conventional vibration sensor because it is image data.

In addition, since there is a problem in that the resolution decreases in inverse proportion when the capturing frame is increased, even if an abnormality is detected by increasing the capturing frame, a problem may occur in which the abnormal state cannot be analyzed due to the low resolution.

Korean Patent Registration No. 10-1632514

In order to solve the problems of the prior art as described above, an embodiment of the present invention is to obtain an image signal having an effect similar to that of an image taken by a camera having a high shooting frame using a camera having a low shooting frame. It is intended to provide an upsampling system and method using multiple images.

According to one aspect of the present invention for solving the above problems, an upsampling system using multiple images is provided. The up-sampling system using the multiple images may include: a video signal acquisition unit configured to acquire a plurality of video signals by capturing a plurality of images in different time zones; a video signal pre-processing unit configured to generate a plurality of classified normalized signals by performing pre-processing of the plurality of video signals; a delay information processor configured to obtain an optimal delay value of each of the normalized signals using the plurality of normalized signals and partial delay signals; and a video signal post-processing unit configured to output an up-sampling signal obtained by merging the plurality of normalization signals using the optimal delay values.

The image signal obtaining unit obtains a positional change of the imaging target as the video signal when the imaging target exists in the imaging target, and sets a characteristic point of the imaging target as the imaging target when the imaging target does not exist, thereby positioning the imaging target. It may be configured to obtain the change as the image signal or to obtain the change in position of a location where the change in the motion of the object to be photographed is the greatest among the specific positions of the object to be photographed as the image signal.

The feature point of the photographing target may be any one of feature points obtained through any one of the SIFT algorithm, the Harris algorithm, and the FAST algorithm.

The video signal pre-processing unit may include: a video signal normalization module configured to normalize the plurality of video signals using a predetermined criterion to generate the plurality of normalized signals; and a normalization signal classification module which selects one of the plurality of normalization signals as a reference normalization signal and selects the remaining signals as acquired normalization signals, wherein the video signal normalization module includes a Chebyshev IIR filter. After filtering the image signal by using, it may be formed to perform normalization.

The delay information processing unit may include a partial delay signal acquisition module configured to obtain a partial delay signal of any one of the normalized signals, wherein the partial delay signal is obtained using an arbitrary integer delay N; and an optimal delay value acquisition module configured to obtain the optimal delay value using the partial delay signal and the normalization signal.

The partially delayed signal obtaining module may be configured to acquire the partially delayed signal of the reference normalized signal among the normalized signals.

The partially delayed signal obtaining module may acquire the partially delayed signal using Equation 1 below by applying a Farrow structure to Lagrange interpolation.

[Equation 1]

: 패로우 구조 필터,

: 패로우 계수)(where x(n): normalized signal, x(n)': partial delay signal of normalized signal,

: Farrow structure filter,

: Farrow coefficient)

The optimal delay value acquisition module acquires a plurality of delay values for a normalized signal using any one of the error sum-of-squares formulas represented by Equation 2 below, and obtains a minimum value of the obtained plurality of delay values as the optimal A plurality of delay values for the arbitrary normalized signal may be obtained using a plurality of partial delay signals generated by changing the arbitrary integer delay N.

[Equation 2]

(Where, X _ij : j-th data of the i-th acquired normalized signal, X _ij ': j-th data of a partial delay signal of the i-th acquired normalized signal, Y _j : i-th data of the reference normalized signal, Y _ij ': reference The j-th data of the i-th partial delayed signal made of the normalized signal, L: signal length)

The video signal post-processing unit may include an upsampling signal generation module configured to generate the upsampling signal by merging the reference normalization signal and the acquisition normalization signal using the optimal delay value and performing interpolation; and an up-sampling signal filtering module configured to perform anti-aliasing by filtering the up-sampling signal.

The upsampling signal filtering module may use a Chebyshev filter to filter the upsampling signal.

According to one aspect of the present invention, an upsampling method using multiple images is provided. The upsampling method using the multiple images may include acquiring a plurality of image signals by capturing a plurality of images in different time zones using an image signal acquisition unit; generating a plurality of classified normalized signals by performing pre-processing of the plurality of video signals using a video signal pre-processing unit; obtaining an optimal delay value using the plurality of normalized signals and partial delay signals through a delay information processing unit; and outputting an up-sampling signal obtained by merging the plurality of normalized signals using the optimal delay values through a video signal post-processing unit.

The acquiring of the plurality of image signals may include acquiring a positional change of the imaging target as the video signal when the imaging target exists in the imaging target, and capturing feature points of the imaging target if the imaging target does not exist. It may be set as a target to obtain a positional change as the image signal, or a positional change of a position where the movement change of the photographing target is greatest among specific positions of the photographing target may be acquired as the image signal.

The feature point of the photographing target may be any one of feature points obtained through any one of the SIFT algorithm, the Harris algorithm, and the FAST algorithm.

The generating of the plurality of classified normalized signals may include: generating the plurality of normalized signals by normalizing each of the plurality of video signals using a predetermined criterion; and selecting one of the plurality of normalization signals as a reference normalization signal and selecting the remaining signals as acquisition normalization signals. After filtering the video signal by doing so, it may be formed to perform normalization.

The obtaining of the optimal delay value may include: obtaining a partial delay signal of any one of the normalized signals and obtaining the partial delay signal using an arbitrary integer delay N; and obtaining the optimal delay value using the partial delay signal and the normalization signal.

The obtaining of the partially delayed signal may include acquiring the partially delayed signal of the reference normalized signal among the normalized signals.

In the obtaining of the partially delayed signal, the partial delayed signal may be obtained using Equation 3 obtained by applying a Farrow structure to the Lagrange interpolation method.

[Equation 3]

: 패로우 구조 필터,

: 패로우 계수)(where x(n): normalized signal, x(n)': partial delay signal of normalized signal,

: Farrow structure filter,

: Farrow coefficient)

The obtaining of the optimal delay value may include obtaining a plurality of delay values for a normalized signal using any one of the error sum-squared formulas represented by Equation 4 below, and obtaining the minimum value of the plurality of delay values obtained It is formed to obtain the optimal delay value, and a plurality of delay values for the arbitrary normalized signal may be obtained using a plurality of partial delay signals generated by changing the arbitrary integer delay N.

[Equation 4]

(Where, X _ij : j-th data of the i-th acquired normalized signal, X _ij ': j-th data of a partial delay signal of the i-th acquired normalized signal, Y _i : i-th data of the reference normalized signal, Y _ij ': reference The j-th data of the i-th partial delayed signal made of the normalized signal, L: signal length)

The outputting of the upsampling signal may include generating the upsampling signal by merging the reference normalization signal and the acquired normalization signal using the optimal delay value and performing interpolation; and performing anti-aliasing by filtering the up-sampling signal.

In the performing of the anti-aliasing, a Chebyshev filter may be used to filter the upsampling signal.

An upsampling system and method using multiple images according to an embodiment of the present invention has an effect of obtaining a result similar to that of a high frame rate camera using a low frame rate camera.

1 is a block diagram illustrating an upsampling system using multiple images according to an embodiment of the present invention.
FIG. 2 is a block diagram illustrating a video signal pre-processing unit of FIG. 1 .
FIG. 3 is a block diagram illustrating a delay information processing unit of FIG. 1 .
FIG. 4 is a block diagram illustrating a video signal post-processing unit of FIG. 1 .
5 is a flowchart illustrating an upsampling method using multiple images according to an embodiment of the present invention.
6 is a flowchart illustrating step S520 of FIG. 5 .
FIG. 7 is a flowchart illustrating step S530 of FIG. 5 .
8 is a flowchart illustrating step S540 of FIG. 5 .

Hereinafter, with reference to the accompanying drawings, embodiments of the present invention will be described in detail so that those skilled in the art can easily carry out the present invention. This invention may be embodied in many different forms and is not limited to the embodiments set forth herein. In order to clearly describe the present invention in the drawings, parts irrelevant to the description are omitted, and the same reference numerals are assigned to the same or similar components throughout the specification.

1 is a block diagram showing an upsampling system using multiple images according to an embodiment of the present invention, FIG. 2 is a block diagram showing a video signal pre-processing unit of FIG. 1, and FIG. 3 is a block diagram showing a delay information processing unit of FIG. FIG. 4 is a block diagram illustrating the image signal post-processing unit of FIG. 1 .

Hereinafter, an upsampling system using multiple images according to an embodiment of the present invention will be described in detail with reference to FIGS. 1 to 4 . The upsampling system 100 using multiple images according to an embodiment of the present invention acquires a plurality of captured images, obtains an optimal delay value of the obtained captured image signal, and then uses the obtained optimal delay value It is formed to generate an upsampling signal by merging captured image signals. To this end, the upsampling system 100 using multiple images of the present invention, as shown in FIG. It may be formed to include the processing unit 140 .

The image signal acquisition unit 110 may be configured to acquire a plurality of photographed images of the object by photographing the object. The image signal acquisition unit 110 may capture a photographing target attached to an object to acquire vibration or motion of the object set by the user, or, when the photographing target does not exist in the object, capture vibration or motion of a specific location of the object. You can also take a picture of that specific location to acquire it. Here, as an example of the present invention, the specific position of the object may be the position with the greatest change in motion when the object is moving, and as another example, SIFT algorithm, Harris algorithm, or FAST algorithm, etc. Using algorithms for extracting feature points It may be any one of a plurality of feature points obtained by doing.

To this end, the image signal acquisition unit 110 may be formed to capture an image using a photographing camera and convert the captured image into an image signal, where the image signal may be a signal representing the motion of an object as described above. there is. In addition, the image signal acquisition unit 110 may be formed to repeatedly photograph at predetermined intervals using a single photographing camera to acquire a plurality of images, and converts each of the plurality of photographed images into image signals. A plurality of video signals may be acquired.

The plurality of image signals acquired by the image signal acquisition unit 110 are transferred to the image signal pre-processing unit 120 . The video signal pre-processing unit 120 is configured to pre-process the video signal prior to processing delay information of a plurality of video signals. Referring to FIG. 2 , the image signal pre-processing unit 120 according to an embodiment of the present invention may include an image signal normalization module 121 and a normalized signal classification module 122 .

The video signal normalization module 121 is configured to normalize each of the plurality of video signals acquired by the video signal acquisition unit 110 . In order to perform upsampling by merging a plurality of video signals into one, the intensity of each video signal must have the same standard size. When they have different sizes, since unwanted errors may occur due to video signals having higher intensities, the video signal normalization module 121 normalizes a plurality of video signals based on a predetermined standard to normalize the plurality of video signals. signal can be generated.

Also, the video signal normalization module 121 may first remove noise from each video signal before normalizing the video signal according to a user's setting. The image signal normalization module 121 may obtain a clearer normalized signal by removing noise of the obtained image signals.

The image signal normalization module 121 according to an embodiment of the present invention may use a Chebysev IIR band pass filter as an example for removing such noise, and normalizes the noise removal signal passing through the filter. Thus, a normalized signal may be obtained.

Next, the normalization signal classification module 122 is configured to classify the plurality of normalization signals into a reference signal and an acquisition signal. The normalization signal classification module 122 classifies any one of a plurality of normalization signals obtained from the video signal normalization module 121 as a reference signal in order to perform upsampling of the normalization signal, and classifies the obtained plurality of normalized signals. Among the normalized signals, signals other than the reference normalized signal may be classified as acquired normalized signals. In other words, the plurality of normalization signals may be classified into one reference normalization signal and a plurality of acquisition normalization signals through the normalization signal classification module 122 . At this time, the length of each signal of the reference normalization signal and the acquisition normalization signal may be preferably expressed as L.

The reference normalization signal and the acquisition normalization signal preprocessed by the video signal pre-processing unit 120 are transferred to the delay information processing unit 130 . The delay information processing unit 130 is configured to obtain an optimal delay value by receiving the reference normalization signal and the acquisition normalization signal. To this end, the delay information processor 130 may include a partial delay signal acquisition module 131 and an optimal delay value acquisition module 132 as shown in FIG. 3 .

Hereinafter, for convenience of description of an embodiment of the present invention, an optimal delay value is obtained using a partial delay signal of a reference normalized signal. In an embodiment of the present invention, when merging a plurality of video signals for upsampling, the acquisition normalized signal may be merged based on the reference normalized signal, but the reference normalized signal may also be merged using the acquired normalized signal. Since this is only a difference between using a delay value having an absolute standard (using a reference signal) and using a delay value having a relative standard (using an acquired normalization signal), both can be included in the present invention. In other words, the following The partial delay signal in the contents of is obtained from the acquisition normalized signal, and may be compared with the reference normalized signal.

The partially delayed signal acquisition module 131 is configured to acquire a partially delayed signal Y', which is a signal obtained by applying partial delay to a reference normalized signal for upsampling.

The video signal obtained in the present invention is expressed as a discrete time system because both input and output signals are defined only at a certain point in time (discrete time). Can be expressed by Equation 1 below.

Meanwhile, an impulse response to a delay element in such a discrete-time system is expressed by Equation 2 below.

(If d=0, h _id (n)=0 for all natural numbers n, and if d∈(0,1), then h _id (n)≠0 for all natural numbers n)

In the case of using a finite-length filter, an accurate value for the impulse response h _id (n) cannot be obtained. Therefore, the partially delayed signal acquisition module 131 of the present invention is configured to obtain an impulse response of finite length by using an FIR filter instead of an IIR filter to estimate the impulse response.

The partially delayed signal acquisition module 131 of the present invention may use the Lagrange interpolation method to obtain a finite impulse response. Equation 3 can be used.

(Where, △: delay control variable, C _k : Farrow coefficient)

Therefore, the partially delayed signal acquisition module 131 according to an embodiment of the present invention may obtain the partially delayed signal Y' of the reference normalized signal using Equation 4 using Equations 1 to 3 above.

: 패로우 구조 필터,

: 패로우 계수)(Where, Y: reference normalized signal, Y': partial delayed signal of reference normalized signal,

: Farrow structure filter,

: Farrow coefficient)

Meanwhile, the optimal delay value acquisition module 132 is configured to obtain an optimal delay value using the partial delay signal received from the partial delay signal acquisition module 131 . The optimal delay value means a value at which the sum of square errors is minimized (Minimized Sum of square errors). Among the obtained plurality of delay values, the minimum value may be selected as the optimal delay value.

Accordingly, the optimal delay value acquisition module 132 may obtain a delay value between the acquired normalized signal and the partial delay signal using Equation 5 below.

(Where, X _ij : j-th data of the i-th acquisition normalized signal, Y _ij ': j-th data of the i-th partial delayed signal made of the reference normalized signal, L: signal length)

The optimal delay value acquisition module 132 first selects one of a plurality of acquired normalization signals and determines an arbitrary integer delay N in order to obtain a delay value using Equation 5 above. Thereafter, the optimal delay value acquisition module 132 acquires the partial delay signal of the reference normalized signal corresponding to the selected acquisition normalized signal using an arbitrary integer delay N using the partial delay signal acquisition module 131, and obtains the selected normalized acquisition normalization signal. A delay value may be obtained by applying the partial delay signal of the signal and the corresponding reference normalized signal to Equation 5 above.

Next, the optimal delay value acquisition module 132 acquires a plurality of delay values using partial delay signals obtained by changing an arbitrary integer delay N to obtain various delay values for the selected acquisition normalized signal.

Finally, the optimal delay value acquisition module 132 may select a delay value having the smallest difference from the selected acquired normalized signal among a plurality of delay values obtained using an arbitrary integer delay N as the optimal delay value.

The optimal delay value acquisition module 132 obtains an optimal delay value for each of a plurality of acquired normalized signals. Since the optimal delay value may be different for each acquired normalized signal, in an embodiment of the present invention, the optimal delay value obtaining module 132 obtains an optimal delay value for each acquired normalized signal to obtain a plurality of optimal delay values. can do.

The optimal delay value obtained by the delay information processing unit 130 is transmitted to the image signal post-processing unit 140 . The video signal post-processing unit 140 is configured to generate an upsampling signal using an optimal delay value. To this end, the image signal post-processing unit 140 may include an upsampling signal generation module 141 and an upsampling signal filtering module 142.

The upsampling signal generating module 141 is configured to generate an upsampling signal using an optimal delay value. If the optimal delay value is used, the upsampling signal generation module 141 can obtain an integer delay value N and a partial delay value d of the acquired normalized signal that generated the corresponding optimal delay value, and using these, the acquired normalized signal and reference One upsampling signal may be generated by merging the normalized signals and performing interpolation between time delays of the merged signals.

The upsampling signal filtering module 142 is configured to filter the upsampling signal generated by the upsampling signal generation module 141 . Aliasing may occur in the process of generating the upsampling signal, and since aliasing is a kind of noise in the process of analyzing the upsampling signal, the upsampling signal filtering module 142 of the present invention uses a preset filter to Signal filtering represented by anti-aliasing can be performed. At this time, as an example, the upsampling signal filtering module 142 may use a Chebysev band stop filter to filter the upsampling signal.

Meanwhile, FIGS. 5 to 8 illustrate an upsampling method using multiple images according to an embodiment of the present invention. Hereinafter, for convenience of explanation, the present invention will be described using the system shown in FIGS. 1 to 4, but it is clear that the present invention is not limited thereto and can be used in various devices or systems capable of performing similar operations or processes. do.

5 is a flowchart illustrating an upsampling method using multiple images according to an embodiment of the present invention, FIG. 6 is a flowchart illustrating step S520 of FIG. 5, FIG. 7 is a flowchart illustrating step S530 of FIG. 5, and FIG. 8 is a flowchart illustrating step S540 of FIG. 5 .

Referring to FIG. 5 , the upsampling method 500 using multiple images according to an embodiment of the present invention acquires a plurality of captured images, obtains an optimal delay value of the obtained captured image signal, and then obtains An upsampling signal is generated by merging photographed image signals using an optimal delay value. To this end, the upsampling method 500 using multiple images of the present invention includes obtaining a plurality of image signals (S510), generating a plurality of classified normalized signals (S520), and obtaining an optimal delay value (S520). S530) and outputting an upsampling signal (S540).

Acquiring a plurality of image signals ( S510 ) may include photographing an object using an image signal acquisition unit and acquiring a plurality of photographed images of the object. In step S510, a shooting target attached to the object may be photographed to obtain the vibration or motion of the object set by the user, or if the photographing target does not exist in the object, the specific position of the object may be captured to obtain the vibration or motion of the specific position of the object. can also be filmed. Here, as an example of the present invention, the specific position of the object may be the position with the greatest change in motion when the object is moving, and as another example, SIFT algorithm, Harris algorithm, or FAST algorithm, etc. Using algorithms for extracting feature points It may be any one of a plurality of feature points obtained by doing.

To this end, in step S510, an image may be captured using a photographing camera and the captured image may be converted into an image signal, where the image signal may be a signal representing motion of an object as described above. In addition, in step S510, in order to obtain a plurality of images, preferably, one photographing camera may be used to repeatedly photograph at predetermined intervals, and each of the plurality of photographed images may be converted into image signals to obtain a plurality of image signals. can be obtained

In step S510, the acquired plurality of image signals are transferred to step S520. The step of generating a plurality of classified normalized signals (S520) is performed to pre-process the video signal using the video signal pre-processing unit prior to processing the delay information of the plurality of video signals. Referring to FIG. 6, generating a plurality of classified normalization signals (S520) according to an embodiment of the present invention includes generating normalization signals (S521) and performing classification (S522). can

In the step of generating a normalization signal (S521), the plurality of image signals obtained in step S510 are normalized respectively. In order to perform upsampling by merging a plurality of video signals into one, the intensity of each video signal must have the same standard size. Since an undesirable error may occur due to an image signal having a greater intensity when they have different sizes, in step S521, a plurality of normalized signals may be generated by normalizing the plurality of video signals based on a predetermined standard. there is.

Also, in step S521, prior to normalizing the video signals according to user settings, noise removal of each video signal may be first performed. By removing the noise of the obtained image signals, a clearer normalized signal may be obtained in step S521.

Step S521 according to an embodiment of the present invention may use a Chebysev IIR band pass filter as an example for removing such noise, normalize the noise removal signal that has passed through the filter to obtain a normalized signal You may.

Next, the step of performing classification (S522) is formed to classify a plurality of normalized signals into a reference signal and an acquisition signal. Step S522 classifies any one of the plurality of normalized signals obtained in step S521 as a reference normalized signal to perform upsampling of the normalized signal, and excluding the reference normalized signal from among the plurality of obtained normalized signals. A signal can be classified as a collected signal. In other words, the plurality of normalization signals may be classified into one reference normalization signal and a plurality of acquisition normalization signals through step S521. At this time, the length of each signal of the reference normalization signal and the acquisition normalization signal may be preferably expressed as L.

The reference normalized signal and the acquired normalized signal preprocessed in step S520 are transferred to the step of obtaining an optimal delay value (S530). In step S530 of obtaining an optimal delay value, an optimal delay value is obtained by receiving a reference normalization signal and an acquisition normalization signal using a delay information processing unit. To this end, step S530 includes obtaining a partial delay signal (step S531) and obtaining an optimal delay value (step S532) as shown in FIG.

Hereinafter, for convenience of description of an embodiment of the present invention, an optimal delay value is obtained using a partial delay signal of a reference normalized signal. In an embodiment of the present invention, when merging a plurality of video signals for upsampling, the acquisition normalized signal may be merged based on the reference normalized signal, but the reference normalized signal may also be merged using the acquired normalized signal. Since this is only a difference between using a delay value having an absolute standard (using a reference signal) and using a delay value having a relative standard (using an acquired normalization signal), both can be included in the present invention. In other words, the following The partial delay signal in the contents of is obtained from the acquisition normalized signal, and may be compared with the reference normalized signal.

In step S531 of obtaining a partially delayed signal, a partially delayed signal Y', which is a signal obtained by applying a partial delay to a reference normalized signal for upsampling, is configured to be obtained.

The video signal obtained in the present invention is expressed as a discrete time system because both input and output signals are defined only at a certain point in time (discrete time). Can be expressed by Equation 6 below.

Meanwhile, the impulse response to the delay element in this discrete-time system is expressed by Equation 7 below.

(If d=0, h _id (n)=0 for all natural numbers n, and if d∈(0,1), then h _id (n)≠0 for all natural numbers n)

In the case of using a finite-length filter, an accurate value for the impulse response h _id (n) cannot be obtained. Therefore, in step S531 of the present invention, an impulse response of a finite length is obtained by using an FIR filter instead of an IIR filter to estimate the impulse response.

Step S531 of the present invention may use Lagrange interpolation to obtain a finite impulse response, and in particular, Equation 8, in which a Farrow structure is applied to Lagrange interpolation, can be used to obtain a partial delay signal with higher accuracy. there is.

(Where, △: delay control variable, C _k : Farrow coefficient)

Therefore, in step S531 according to an embodiment of the present invention, the partial delay signal Y' of the reference normalized signal may be obtained using Equation 9 using Equations 6 to 8 above.

: 패로우 구조 필터,

: 패로우 계수)(Where, Y: reference normalized signal, Y': partial delayed signal of reference normalized signal,

: Farrow structure filter,

: Farrow coefficient)

Meanwhile, in step S532 of obtaining an optimal delay value, an optimal delay value is obtained by using the partial delay signal transmitted from step S531. The optimal delay value means a value at which the sum of square errors is minimized (Minimized Sum of square errors). Among the obtained plurality of delay values, the minimum value may be selected as the optimal delay value.

Accordingly, in step S532, a delay value between the acquisition normalized signal and the partial delay signal may be obtained using Equation 10 below.

(Where, X _ij : j-th data of the i-th acquisition normalized signal, Y _ij ': j-th data of the i-th partial delayed signal made of the reference normalized signal, L: signal length)

Step S532 first selects one of a plurality of acquired normalization signals and determines an arbitrary integer delay N in order to obtain a delay value using Equation 10 above. Thereafter, in step S532, a partial delay signal of the reference normalized signal corresponding to the acquired normalized signal selected in step S531 is obtained using an arbitrary integer delay N, and the selected acquired normalized signal and the partial delayed signal of the reference normalized signal corresponding thereto are described above. A delay value can be obtained by applying Equation 10.

Next, in step S532, a plurality of delay values are obtained using partial delay signals obtained by changing an arbitrary integer delay N to obtain various delay values for the selected acquired normalized signal.

Finally, in step S532, a delay value having the smallest difference from the selected acquired normalized signal among a plurality of delay values acquired using an arbitrary integer delay N may be selected as an optimal delay value.

Step S532 obtains an optimal delay value for each of a plurality of acquired normalization signals. Since the optimal delay value may be different for each acquired normalized signal, in step S532, a plurality of optimal delay values may be obtained by obtaining an optimal delay value for each acquired normalized signal in step S532.

The optimal delay value obtained in step S530 is transmitted to the step of outputting an upsampling signal (S540). In the step of outputting the upsampling signal (S540), the upsampling signal is generated using the optimal delay value through the video signal post-processing unit. To this end, step S540 may include generating an upsampling signal (step S541) and performing anti-aliasing (step S542).

In the step of generating an upsampling signal (S541), the upsampling signal is generated using an optimal delay value. If the optimal delay value is used, step S541 may obtain an integer delay value N and a partial delay value d of the acquired normalized signal that generated the corresponding optimal delay value, and merge the acquired normalized signal and the reference normalized signal using this, One upsampling signal may be generated by performing interpolation between time delays of the merged signals.

The step of performing anti-aliasing (S542) is configured to filter the upsampling signal generated in step S541. Aliasing may occur in the process of generating the upsampling signal, and since aliasing is a kind of noise in the process of analyzing the upsampling signal, step S542 of the present invention is expressed as anti-aliasing using a preset filter. Signal filtering can be performed. In this case, as an example, in step S542, a Chebysev band stop filter may be used to filter the upsampling signal.

Meanwhile, in the system or method of FIGS. 1 to 8 , the captured image is subjected to distortion correction for correcting the distortion effect according to the capturing position in the image signal pre-processing unit 120 or generating a plurality of classified normalization signals (S520). Each may be further performed in, and any one of various known image distortion correction methods may be used for distortion correction of the captured image.

9 shows a simulation using a system and method according to an embodiment of the present invention and its results. In the simulation of FIG. 9, a marker was placed on top of the air purifier, and vibration generated in the marker due to the operation of the air purifier was measured. At this time, in order to measure the vibration of the marker, an accelerometer was placed on the marker to directly measure the vibration value of the marker. In order to obtain the upsampling result of the present invention, a camera was placed 55 cm away from the marker and an image was taken with the camera. did In addition, the specifications of the experimental devices used in this simulation are shown in Table 1 below.

equipment product name Performance High-speed camera Optronis
CR450x3 200 x 320 @ 1000 fps
200 x 320 @ 2000 fps Lamp Multiled
PT-V8-15 24 HP LEDs7,700 lumen total Attached sensor PCB
Accelerometer Fs=4,266 Hz DAQ NI NI9232 air cleaner Mi Purifier Pro Normal mode

Matrix Parameter Properties Intrinsic[I] f _x , f _y 1.134e4, 1.149e4 c _x ,c _y 135.95, 129.64 s 0 Extrinsic[E] r 3x3x15 t 1x3x15 Radial Distortion k ₁ , k ₂ [3.215e+02, -1.028e+06]

9A is a graph showing vibration (left) obtained from an image of an object captured at 1000 fps in this simulation experiment and vibration (right) measured using an accelerometer of 4,266 Hz. Referring to FIG. 9A , it can be seen that the same peaks appear at 28.3 Hz, 87.7 Hz, 141.7 Hz, 226.8 Hz, 255.1 Hz, 283 Hz, and 425.2 Hz in the range of 0 to 500 Hz, respectively. In particular, the peak at 255.1 Hz was identified as the highest peak in both measurement results. However, in this case, the peak at a frequency of 500 Hz or higher cannot be measured due to the limit of 1000 fps measured in this experiment.

In order to overcome this limitation, in this simulation, as shown in FIG. 9B, vibration (left) obtained from a signal obtained by upsampling an image taken by a 1000 fps camera to 2000 fps using the present invention, using a 2000 fps camera A graph of the obtained vibration (middle) and the obtained vibration (right) from the accelerometer sensor were obtained.

Referring to FIG. 9B, it can be seen that peaks appear at frequencies of 28.3 Hz, 87.7 Hz, 141.7 Hz, 226.8 Hz, 255.1 Hz, 283 Hz, and 425.2 Hz in all three graphs. In addition, comparing the 2000 fps upsampling graph (left) and the 2000 fps graph (middle), it can be seen that noise appears lower in the upsampling graph, because less noise is obtained from images with low fps.

That is, referring to FIG. 9B , rather than acquiring vibration of an object by capturing an image using a camera having a high fps, by upsampling an image of a camera captured at a low fps using the present invention, vibration of an object is obtained. It can be seen that acquiring is more effective.

Although one embodiment of the present invention has been described above, the spirit of the present invention is not limited to the embodiments presented herein, and those skilled in the art who understand the spirit of the present invention may add elements within the scope of the same spirit. However, other embodiments can be easily proposed by means of changes, deletions, additions, etc., but these will also fall within the scope of the present invention.

100: Upsampling system using multiple images
110: video signal acquisition unit 120: video signal pre-processing unit
121: video signal normalization module 122: normalization signal classification module
130: delay information processing unit
131: partial delay signal acquisition module 132: optimum delay value acquisition module
140: video signal post-processing unit 141: upsampling signal generation module
142: upsampling signal filtering module

Claims (20)

an image signal acquisition unit configured to acquire a plurality of image signals by capturing a plurality of images of different time zones;
a video signal pre-processing unit configured to generate a plurality of classified normalized signals by performing pre-processing of the plurality of video signals;
a delay information processor configured to obtain an optimal delay value of each of the normalized signals using the plurality of normalized signals and partial delay signals; and
A video signal post-processing unit configured to output an upsampling signal obtained by merging the plurality of normalization signals using the optimal delay values;
The video signal acquisition unit,
When there is a shooting target in the shooting target, the location change of the shooting target is acquired as the video signal. Acquiring, or acquiring the positional change of a position where the change in motion of the photographing target is the greatest among specific positions of the photographing target as the image signal,
The video signal pre-processing unit,
a video signal normalization module configured to generate the plurality of normalized signals by normalizing each of the plurality of video signals using a predetermined criterion; and
A normalization signal classification module for performing classification in which one of the plurality of normalization signals is selected as a reference normalization signal and the remaining signals are selected as acquired normalization signals;
The video signal normalization module filters the video signal using a Chebyshev IIR filter and then performs normalization. delete According to claim 1,
The upsampling system using multiple images, wherein the feature point of the photographing target is any one of feature points obtained through any one of the SIFT algorithm, the Harris algorithm, and the FAST algorithm. delete According to claim 1,
The delay information processing unit,
a partial delay signal acquisition module configured to obtain a partial delay signal of any one of the normalization signals, wherein the partial delay signal is obtained using an arbitrary integer delay N; and
An optimal delay value acquisition module configured to obtain the optimal delay value by using the partial delay signal and the normalization signal. According to claim 5,
The partially delayed signal acquisition module,
An upsampling system using multiple images configured to obtain the partially delayed signal of the reference normalized signal among the normalized signals. According to claim 5,
The partially delayed signal acquisition module acquires the partially delayed signal using Equation 1 to which a Farrow structure is applied to Lagrange interpolation.
[Equation 1]

(where x(n): normalized signal, x(n)': partial delay signal of normalized signal,

: Farrow structure filter,

: Farrow coefficient) According to claim 7,
The optimal delay value obtaining module,
A plurality of delay values for an arbitrary normalized signal are obtained using any one of the error sum squared formulas represented by Equation 2 below, and a minimum value among the obtained plurality of delay values is obtained as the optimal delay value,
A plurality of delay values for the arbitrary normalized signal are obtained using a plurality of partial delay signals generated by changing the arbitrary integer delay N.
[Equation 2]

(Where, X _ij : j-th data of the i-th acquired normalized signal, X _ij ': j-th data of a partial delay signal of the i-th acquired normalized signal, Y _j : i-th data of the reference normalized signal, Y _ij ': reference The j-th data of the i-th partial delayed signal made of the normalized signal, L: signal length) According to claim 8,
The video signal post-processing unit,
an upsampling signal generation module, configured to generate the upsampling signal by merging the reference normalization signal and the acquired normalization signal using the optimal delay value and performing interpolation; and
An up-sampling signal filtering module configured to perform anti-aliasing by filtering the up-sampling signal; According to claim 9,
The upsampling signal filtering module uses a Chebyshev filter to filter the upsampling signal. acquiring a plurality of image signals by photographing a plurality of images of different time zones using an image signal acquisition unit;
generating a plurality of classified normalized signals by performing pre-processing of the plurality of video signals using a video signal pre-processing unit;
obtaining an optimal delay value using the plurality of normalized signals and partial delay signals through a delay information processing unit; and
Outputting an upsampling signal obtained by merging the plurality of normalized signals using the optimal delay values through a video signal post-processing unit;
Acquiring the plurality of video signals,
When there is a shooting target in the shooting target, the location change of the shooting target is obtained as the video signal, and when the shooting target does not exist, a feature point of the shooting target is set as the shooting target, and the position change is converted into the video signal. Acquiring, or acquiring the positional change of a position where the change in motion of the photographing target is the greatest among specific positions of the photographing target as the image signal,
Generating the plurality of classified normalization signals,
generating the plurality of normalized signals by normalizing each of the plurality of video signals using a preset standard; and
Selecting one of the plurality of normalization signals as a reference normalization signal and performing classification to select the remaining signals as acquisition normalization signals;
Generating the normalization signal,
An upsampling method using multiple images formed to perform normalization after filtering the image signal using a Chebyshev IIR filter. delete According to claim 11,
The feature point of the photographing target is any one of feature points obtained through any one of the SIFT algorithm, the Harris algorithm, and the FAST algorithm. delete According to claim 11,
The step of obtaining the optimal delay value,
obtaining a partial delay signal of any one of the normalized signals, and acquiring the partial delay signal using an arbitrary integer delay N; and
Acquiring the optimal delay value using the partial delay signal and the normalization signal; upsampling method using multiple images to include. According to claim 15,
The obtaining of the partially delayed signal is performed to obtain the partially delayed signal of the reference normalized signal among the normalized signals. According to claim 15,
The step of obtaining the partially delayed signal is an upsampling method using multiple images in which the partially delayed signal is obtained using Equation 3 in which a Farrow structure is applied to Lagrange interpolation.
[Equation 3]

(where x(n): normalized signal, x(n)': partial delay signal of normalized signal,

: Farrow structure filter,

: Farrow coefficient) According to claim 17,
The step of obtaining the optimal delay value,
A plurality of delay values for an arbitrary normalized signal are obtained using any one of the error sum squared formulas represented by Equation 4 below, and a minimum value among the obtained plurality of delay values is obtained as the optimal delay value,
A plurality of delay values for the random normalization signal are obtained using a plurality of partial delay signals generated by changing the random integer delay N.
[Equation 4]

(Where, X _ij : j-th data of the i-th acquired normalized signal, X _ij ': j-th data of a partial delay signal of the i-th acquired normalized signal, Y _i : i-th data of the reference normalized signal, Y _ij ': reference The j-th data of the i-th partial delayed signal made of the normalized signal, L: signal length) According to claim 18,
The step of outputting the upsampling signal,
generating the upsampling signal by merging the reference normalization signal and the acquisition normalization signal using the optimal delay value and performing interpolation; and
Filtering the up-sampling signal to perform anti-aliasing; According to claim 19,
The performing of the anti-aliasing may include using a Chebyshev filter to filter the up-sampling signal.

Images