KR20120022651A - Signal processing method and apparatus based on multiple texture using video sensor excitation signal - Google Patents

Abstract

PURPOSE: A signal processing method and apparatus based on multiple texture using a video sensor excitation signal are provided to express various signal properties through time-space variable of a texture signal. CONSTITUTION: A signal processing apparatus detects a plurality of feature points from a selected seed signal(S11,S12). The signal processing apparatus defines the texture signals(S14). The signal processing apparatus performs an approximation of texture signals which has similar variables for time-space location conversion(S15). The signal processing apparatus compresses video sensor filtering signal, a variable of texture composition filter, and a plurality of variables for time-space location conversion(S17).

Description

Signal processing method and apparatus based on multiple texture using video sensor excitation signal

The present invention relates to a multi-texture based signal processing method and apparatus using an image sensor excitation signal. More specifically, the present invention relates to a multi-texture based signal processing method and apparatus using an image sensor excitation signal capable of compressing and expressing an image and a sensor signal at a low data rate.

In general, the signal processing and compression method of the video signal and the sensor signal is not an integrated method but a method of compressing and expressing the image and sensor signals separately. The video signal processing and compression method is based on the method of transform domain processing of motion estimation processed signal between video frames by using discrete cosine transform, and the sensor signal processing and compression method is its own signal or physical signal characteristics of the signal. It is based on the method of adjusting the compression bit rate according to the importance and importance. However, the general signal compression method has a problem in that it is difficult to express various real signal characteristics due to inaccurate estimation and modeling of image and sensor signal characteristics. In addition, the error value of the signal expression model estimation error is increased in the frames with a lot of image motion or the sensor signal with a lot of noise. In order to solve this problem, video signal compression standards represented by MPEG1 / 2/4 and H.261 / 3/4 have been proposed, but still the picture quality and sound quality deterioration is severe in the compression of 1/1000 low-rate image sensor signal. In this situation, a specific compression method has not been proposed for the sensor signal.

An object of the present invention is to provide a compressed signal compressed by integrating a video signal and a sensor signal.

An object of the present invention is to express various signal characteristics through a plurality of texture signals and the space-time position transformation variables of the texture signals.

In addition, an object of the present invention is to provide a compressed signal significantly reduced in size compared to the original size by compressing the original signal with only a plurality of texture signals and a plurality of space-time position conversion variables corresponding thereto. In addition, the present invention defines each of the plurality of texture signals as the sum of the image sensor excitation signal represented by the Gaussian function and the texture block which is the output of the texture synthesis filter, thereby providing a compressed signal that is significantly smaller in size than the original size. It aims to do it.

In addition, the present invention aims to further reduce the size of the compressed signal by approximating a plurality of predetermined texture signals using the similarity of the spatiotemporal position transformation variables.

In addition, an object of the present invention is to process an image with an optimum image quality at a low data rate.

In the multi-texture based signal processing method according to the present invention for achieving the above object, a signal including an input image signal and a sensor signal is classified into a signal component unit signal, and one frame among a plurality of frames of the signal component unit signal. Selecting as a seed signal; Detecting a plurality of feature points in the seed signal; Calculating space-time position transformation variables for each of the feature points by tracking the plurality of feature points in the plurality of frames of the signal configuration unit signal; Defining a plurality of texture signals using feature points corresponding to the spatiotemporal position transformation variables; And defining each of the plurality of texture signals as a sum of a plurality of texture blocks that are outputs of a texture synthesis filter that receives an image sensor excitation signal.

In this case, the image sensor excitation signal may be represented by a two-dimensional Gaussian function.

The method may further include compressing the image sensor excitation signal of each of the plurality of texture blocks defining the plurality of texture signals, the variable of the texture synthesis filter, and the spatiotemporal position conversion variable corresponding to each of the plurality of texture signals. Include.

At this time, the compressing may include compressing the image sensor excitation signal, the variable of the texture synthesis filter, and the spatiotemporal position conversion variable by a bitstream compression method.

At this time, in the plurality of texture signals, texture signals having spatio-temporal position transformation variables having similarities calculated by obtaining correlation characteristics of texture signal signals having values within a preset threshold are converted into one texture signal. And further approximating.

In this case, the detecting of the plurality of feature points detects, as the feature points, points having a change amount greater than or equal to a predetermined numerical value in the plurality of frames.

Decompressing the compressed image sensor excitation signal, the variable of the texture synthesis filter, and the space-time position conversion variable corresponding to each of the texture signals; Generating the plurality of texture blocks using the image sensor excitation signal and the variables of the texture synthesis filter, and generating the plurality of texture signals by adding the texture blocks; Matching the space signal with the space-time displacement variable corresponding to the texture signal; Generating a visual sensor texture using the texture signal and the spatiotemporal position transformation variable; And combining the visual sensor textures generated in correspondence to the respective texture signals.

In this case, the method may further include filtering and correcting an artifact at the combined boundary of the visual sensor textures.

In this case, the method may further include decomposing the reconstruction signal into a reconstruction image signal and a reconstruction sensor signal.

In addition, the multi-texture signal processing apparatus according to the present invention for achieving the above object is to classify the signal including the input image signal and the sensor signal as a signal component unit signal, and a plurality of frames of the signal component unit signal A seed signal selector configured to select one frame as a seed signal; A feature point detector for detecting a plurality of feature points from the seed signal; A variable calculator configured to track the plurality of feature points in the plurality of frames of the signal configuration unit signal and calculate a space-time position conversion variable for each feature point; A texture signal definition unit defining a plurality of texture signals using feature points corresponding to the spatiotemporal position conversion variables; And a texture block definition unit which defines each of the plurality of texture signals as a sum of a plurality of texture blocks that are outputs of a texture synthesis filter that receives an image sensor excitation signal.

In this case, the image sensor excitation signal may be represented by a two-dimensional Gaussian function.

At this time, the compression unit for compressing the image sensor excitation signal of each of the plurality of texture blocks defining the plurality of texture signals, the variable of the texture synthesis filter, and the spatiotemporal position conversion variable corresponding to each of the plurality of texture signals. Include.

In this case, the compression unit separately compresses the image sensor excitation signal, the texture synthesis filter variable, and the space-time position conversion variable by a bitstream compression method.

At this time, in the plurality of texture signals, texture signals having spatio-temporal position transformation variables having similarities calculated by obtaining correlation characteristics of texture signal signals having values within a preset threshold are converted into one texture signal. It further includes an approximation unit that approximates in total.

At this time, the feature point detector detects, as the feature point, a point having a change amount equal to or greater than a predetermined numerical value in the plurality of frames.

At this time, the decompression unit for decompressing the compressed image sensor excitation signal, the variable of the texture synthesis filter and the space-time position conversion variable corresponding to each of the texture signal; A texture signal generator configured to generate the plurality of texture blocks using the image sensor excitation signal and the variables of the texture synthesis filter, and generate the texture signal by adding the plurality of texture blocks; A matching unit matching the texture signal and the space-time position conversion variable corresponding to the texture signal; A visual sensor texture generation unit configured to generate a visual sensor texture using the texture signal and the spatiotemporal position transformation variable; And a visual sensor texture combiner configured to combine the visual sensor textures generated corresponding to the respective texture signals.

At this time, the visual sensor texture further comprises a correction unit for filtering and correcting (Artifact) at the boundary of the combination.

The apparatus may further include a decomposition unit for decomposing the restoration signal into a restoration image signal and a restoration sensor signal.

According to the present invention, it is possible to provide a compressed signal compressed by integrating a video signal and a sensor signal.

In addition, various signal characteristics may be expressed through the plurality of texture signals and the spatiotemporal position transformation variables of the texture signals.

In addition, the present invention can compress the original signal using only a plurality of texture signals and a plurality of space-time position transformation variables corresponding thereto, thereby providing a compressed signal having a significantly reduced size compared to the original size. In addition, the present invention defines each of the plurality of texture signals as the sum of the image sensor excitation signal represented by the Gaussian function and the texture block which is the output of the texture synthesis filter, thereby providing a compressed signal that is significantly smaller in size than the original size. It aims to do it.

In addition, the present invention can further reduce the size of the compressed signal by approximating a plurality of predetermined texture signals using similarities of the spatiotemporal position transformation variables.

In addition, the present invention can process an image with an optimum image quality at a low data rate. That is, the present invention can minimize image quality degradation at low data rates such as 1/500 bit rate.

1 is a flowchart illustrating an encoding method in a multiple texture based signal processing method according to the present invention.
2 is a diagram for describing a method of encoding in a multiple texture based signal processing method according to the present invention.
3 is an operation flowchart for explaining a decoding method in a multi-texture based signal processing method according to the present invention.
4 is a view for explaining a decoding method in a multi-texture based signal processing method according to the present invention.
5 is a block diagram showing the configuration of a multi-texture based signal processing apparatus according to the present invention.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. Here, the repeated description, well-known functions and configurations that may unnecessarily obscure the subject matter of the present invention, and detailed description of the configuration will be omitted. Embodiments of the present invention are provided to more completely describe the present invention to those skilled in the art. Accordingly, the shape and size of elements in the drawings may be exaggerated for clarity.

Hereinafter, an encoding method in a multiple texture-based signal processing method according to the present invention will be described.

1 is a flowchart illustrating an encoding method in a multiple texture based signal processing method according to the present invention. 2 is a diagram for describing a method of encoding in a multiple texture based signal processing method according to the present invention.

1 and 2, the encoding method of the multi-texture based signal processing method according to the present invention first receives a signal 200 composed of a plurality of frames (S10). The signal is a signal including an image signal 201 and a sensor signal 202. Herein, the sensor signal 202 means a signal input from a sensor, and the sensor may include all sensors that can be generally used, such as a temperature sensor, a chemical sensor, an environmental sensor, a wind sensor, and the like. Can be.

The received signal 200 is classified into a signal component unit signal, and one frame among the plurality of frames of the signal component unit signal is selected as the seed signal 210 (S11). The remaining frames other than the seed signal 210 of the signal component unit signal are defined as the remaining frame signal 220. That is, when the signal configuration unit signal is composed of k frames, one seed signal is selected, and the remaining k-1 frames are defined as the remaining frame signal 220. At this time, the signal configuration unit signal means a signal constituting the signal. For example, in the video signal, the shot unit signal is a signal component unit signal, and the shot unit signal corresponds to an image captured by one camera continuously.

A plurality of feature points are detected from the seed signal 210 selected in step S11 (S12). At this time, in a plurality of frames of the signal structure unit signal, a point having a change amount equal to or greater than a predetermined numerical value can be detected as a feature point. That is, if a specific point in the seed signal 210 and the remaining frame signal 220 shows a change over a predetermined value, the specific point may be detected as a feature point.

Then, the plurality of feature points are tracked in the plurality of frames of the signal component unit signal to calculate a space-time position conversion variable for each feature point (S13). That is, the spatiotemporal position transformation variable defining the change of the feature point in the seed signal 210 and the remaining frame signal 220 is calculated. The space-time position transformation variable may be in the form of a function indicating an amount of change of the position of the feature point over time.

The plurality of texture signals 211a, 212a, 213a, 214a, and Na are defined using feature points corresponding to the spatiotemporal position conversion variables 211b, 212b, 213b, 214b, and Nb calculated in step S13 (S14). ). In this case, the space-time position transformation variables 211b, 212b, 213b, 214b, and Nb may define one texture signal by connecting the same feature points to each other.

Then, in the plurality of texture signals, texture signals having similar spatiotemporal position transformation variables are combined and approximated as one texture signal (S15). In this case, the similarity between the spatiotemporal position transformation variables may be calculated by obtaining correlation characteristics of the texture signal signals. Similarity between the spatiotemporal position transformation variables may combine texture signals having values within a predetermined threshold into one texture signal. In FIG. 2, the first texture signal 211a and the second texture signal 212a are assumed to have a high similarity in space time position transformation variables, and correspondingly, the first space time position transformation variable 211b and the second space time position are combined. By combining the transform variables 212b, the first approximated texture signal 211a 'and the first approximated spatiotemporal position change variable 211b' are generated. Then, the third texture signal 213a and the fourth texture signal 214a are combined, and the third space-time position transformation variable 213b and the fourth space-time position transformation variable 214b are correspondingly combined to form the second texture. An approximated texture signal 213a 'and a second approximated spatiotemporal position change variable 213b' are generated.

Each of the plurality of texture signals 211a, 212a, 213a, 214a, and Na is defined as the sum of the plurality of texture blocks (S16). If step S15 is performed, each of the plurality of approximated texture signals 211 ', 213', and Na 'may be defined as the sum of the plurality of texture blocks. In this case, the texture block may be defined as an output of the texture synthesis filter that receives the image sensor excitation signal. The image sensor excitation signal may be represented by a two-dimensional Gaussian function. The image sensor excitation signal, that is, the Gaussian function, has the size variable G, the mean value variable m, and the dispersion value a as model variables. The texture synthesis filter has a transform domain filter coefficient as a model variable, as shown in Equation 1 below.

The variables of the image sensor excitation signal, that is, the values of G, m, a and the variable h of the texture synthesis filter are calculated to minimize the difference between the texture estimation value and the original texture signal value in the transform domain. The texture estimation signal R in the transform domain is expressed by Equation 2 below.

E and H represent the excitation signal vector and the texture synthesis filter coefficient vector in the transform domain, and '?' Represents the product of each component of the vector. The excitation signal vector E is approximated by a two-dimensional Gaussian function, and the texture synthesis filter H has a property that most of the variable values are zero and only some areas have variable values according to the texture conversion region characteristics. Accordingly, the signal processing method according to the present invention enables the compression at a very low bit rate using a variable length encoder or an Arithmetic encoder, and has the same structure as the image signal 201 and the sensor signal 202. This allows for efficient signal processing and compression.

The image sensor excitation signal of each of the plurality of texture blocks defining each of the plurality of texture signals 211a, 212a, 213a, 214a, and Na, a variable of the texture synthesis filter, and a plurality of spacetimes corresponding to each of the plurality of texture signals. The position conversion variables 211b, 212b, 213b, 214b, and Nb are compressed (S17). Further, in step S17, on the premise that the approximation step S15 of the texture signal is performed, the image sensor excitation signal of each of the plurality of texture blocks defining each of the plurality of approximation texture signals 211a ', 213a', and Na '. The variable of the texture synthesis filter and the plurality of approximated spatiotemporal position conversion variables 211b ', 213b', and Nb 'corresponding to the plurality of approximated texture signals may be compressed. In this case, the compression may be performed by a bitstream compression method.

Hereinafter, the encoding method in the multi-texture-based signal processing method according to the present invention will be described through equations.

에 대하여, 자기 상관 매트릭스(Autocorrelation matrix)

를 계산한다. 여기서,

는 {x,y}가

를 만족할 때의 포인트의 주변 윈도우 신호이다. 그리고, x와 y는 각각 x축 방향과 y축 방향의 픽셀 포인트이며,

는 통계적 기대 함수(Statistical expectation operator)로 정의된다. The feature point can be detected as follows. First, an input image consisting of k frames

With respect to the autocorrelation matrix

Calculate here,

Is { x , y }

The window signal around the point at which is satisfied. And x and y are pixel points in the x- and y- axis directions, respectively.

Is defined as a statistical expectation operator.

를 통하여 계산된 고유치들 즉,

및

로부터, 텍스쳐 포인트 매트릭스

를 다음의 수학식 3과 같이 구할 수 있다. At pixel point { x , y }

Eigenvalues computed through

And

Texture point matrix

Can be obtained as in Equation 3 below.

및

는 기 설정된 임계치에 해당한다. 상기 수학식 3에서 특정 픽셀 위치의

및

가 임계치

및

보다 큰 경우, 해당 특정 픽셀을 1로 정의한다. 그리고, 특정 픽셀 위치의

및

가 임계치

및

보다 작은 경우, 해당 특정 픽셀을 0으로 정의하여 텍스쳐 포인트 매트릭스를 구한다.
here,

And

Corresponds to a preset threshold. In Equation 3, a specific pixel position

And

Threshold

And

If it is larger, the specific pixel is defined as 1. And the specific pixel position

And

Threshold

And

If it is smaller, we define that particular pixel as 0 to get a texture point matrix.

In addition, the plurality of space-time position transformation variables defining each texture signal and texture signals thereof may be defined as in Equation 4 below.

는 다음의 수학식 5와 같이 정의될 수 있다.here,

May be defined as in Equation 5 below.

는 다음의 수학식 6과 같이, N개의 텍스쳐 신호의 합으로써 정의될 수 있다. And, the input image consisting of k frames

May be defined as the sum of the N texture signals, as shown in Equation 6 below.

In addition, the i- th partitioned texture signal in Equation 6 may be expressed by approximating Equation 7 below.

는 변환 함수를,

은 입력 영상의 l번째 프레임의 i번째 분할된 텍스쳐 신호를,

는 x축 방향과 y축 방향의 위치변환 벡터를,

는

에서의 개략적인 추정 오류 신호를 나타낸다. 그리고, 수학식 7에서, 프레임 넘버 k는 k+1부터 l+M의 범위에 속한다. 수학식 5는 Taylor expansion에 의하여 다음의 수학식 8과 같이 근사화될 수 있다.here,

Is a conversion function,

Is the i- th partitioned texture signal of the l- th frame of the input image,

Is the displacement vector in the x- and y-axis directions,

Is

Shows a rough estimation error signal in. And, in equation (7), the frame number k is in the range of k + 1 to l + M. Equation 5 may be approximated by Equation 8 by Taylor expansion.

및

각각은

의 x축 방향과 y축 방향의 경사도(Gradient value)의 합을 나타낸다. 그리고, 추정 오류 신호의 제곱합에 대한 정리는 다음의 수학식 9와 같이 나타낼 수 있다. here,

And

Each one

The sum of the gradient values in the x-axis direction and the y-axis direction of. The theorem for the sum of squares of the estimated error signals may be expressed as in Equation 9 below.

크기의 최소화를 가정하여,

의 값을 구할 수 있다. 즉, 다음의 수학식 10 및 수학식 11을 계산하여,

의 값을 구한다.Where the sum of squares of the estimated error signals

Assuming minimization of size,

Can be found. That is, the following equations (10) and (11) are calculated,

Find the value of.

가 항등변환(Identity transform)식임을 가정하면, 수학식 9와 수학식 10을 통해, 다음의 수학식 12를 얻을 수 있다.here,

Assuming that is an identity transform equation, the following equation 12 can be obtained through equations (9) and (10).

에 대한 다음의 수학식 13을 얻을 수 있다.And, if you solve the equation (12),

The following equation (13) can be obtained.

및 수학식 11을 사용하여 다음의 수학식 13과 같은 변환함수

를 얻을 수 있다.And, obtained through the equation (13)

And a conversion function such as Equation 13 below using Equation 11

Can be obtained.

의 변환함수

으로부터

를 얻을 수 있다. 그리고,

의 변환함수

으로부터

를 얻을 수 있다. 또한, 수학식 3 내지 수학식 14를 통해,

를 시드 신호

및 변환함수

로써 표현할 수 있다. 그리고,

간의 유사성을 계산하여 텍스쳐 신호의 근사화가 이루어질 수 있다.
To sum up Equation 10 and Equation 11,

Conversion function of

From

Can be obtained. And,

Conversion function of

From

Can be obtained. Further, through Equations 3 to 14,

Seeding signal

And conversion functions

Can be expressed as: And,

An approximation of the texture signal can be made by calculating the similarity between them.

Hereinafter, a decoding method of the multi-texture based signal processing method according to the present invention will be described.

3 is an operation flowchart for explaining a decoding method in a multi-texture based signal processing method according to the present invention. 4 is a view for explaining a decoding method in a multi-texture based signal processing method according to the present invention.

3 and 4, the decoding method of the multi-texture based signal processing method according to the present invention first receives a compressed signal (S30). In this case, the compressed signal may be an image sensor excitation signal of each of a plurality of texture blocks defining a plurality of texture signals, a variable of a texture synthesis filter, and a signal of a plurality of spatiotemporal position conversion variables corresponding to each of the texture signals. . Of course, the compressed signal may be a signal obtained by compressing an image sensor excitation signal, a texture synthesis filter variable, and a plurality of approximated spatiotemporal position conversion variables of each of the plurality of texture blocks that define the plurality of approximated texture signals. Also, in the compressed signal, the image sensor excitation signal, the variable of the texture synthesis filter, and the plurality of spatiotemporal position conversion variables may be compressed by a bitstream compression method.

Then, the compressed signal is decompressed (S31). That is, the compressed image sensor excitation signal, the texture synthesis filter variable, and the plurality of space-time position conversion variables corresponding to each texture signal are decompressed.

Then, a plurality of texture blocks are generated using the compressed image sensor excitation signal and the variable of the texture synthesis filter, and a texture signal is generated by adding the plurality of texture blocks (S32). The texture signal at this time may be an approximated texture signal in the encoding process.

In the generated plurality of texture signals and the plurality of space-time position transformation variables, the space-time position transformation variables corresponding to each texture signal and the corresponding texture signal are matched one-to-one (S33). Of course, each of the approximated texture signals and the approximated spatiotemporal position transformation variables corresponding to the corresponding approximated texture signals may be matched. In FIG. 4, the first approximation texture signal 211a 'and the first approximation spatiotemporal position transformation variable 211b' are matched, and the second approximation texture signal 213a 'and the second approximation spatiotemporal position transformation variable 213b'. Is matched, and the Nth approximated texture signal Na 'and the Nth approximated spatiotemporal position change variable Nb' are matched.

In operation S33, a visual sensor texture is generated using the matched texture signal and the space-time position transformation variable in operation S34. Specifically, a visual sensor texture composed of a plurality of frames for the texture signal is generated by applying a spatiotemporal position change variable that defines a time-dependent movement of feature points to the texture signal. Of course, a visual sensor texture may be generated using a matched approximation texture signal and an approximation spatiotemporal position transformation variable. In FIG. 4, a first visual sensor texture 211 composed of a plurality of frames is generated by using the first approximated texture signal 211a 'and the first approximated spatiotemporal position change variable 211b'. Then, a second visual sensor texture 213 consisting of a plurality of frames is generated using the second approximation texture signal 213a 'and the second approximation spatiotemporal position change variable 213b'. Further, the Nth visual sensor texture N including the plurality of frames is generated using the Nth approximated texture signal Na 'and the Nth approximated spatiotemporal position change variable Nb'.

In operation S34, the visual sensor textures generated in correspondence with the respective texture signals are combined (S35). By combining the visual sensor textures, a plurality of frames of the signal component unit signal are totally reconstructed. In FIG. 4, the first visual sensor texture 211, the second visual sensor texture 213, and the Nth visual sensor texture N are combined.

Artifacts at the coupling boundary of the plurality of visual sensor textures combined at step S35 are filtered and corrected at step S36. That is, the plurality of visual sensor textures combined in step S35 are restored as a simple sum, and a defect may occur at a boundary between the visual sensor textures. The removal filtering operation is performed on these defects to generate a corrected recovery signal.

Then, the reconstruction signal of the signal structural unit obtained through step S36 is decomposed to finally generate a reconstruction image signal 201 'and a reconstruction sensor signal 202' (S37). In the present invention, since a signal including an image signal and a sensor signal is represented using a model based on an excitation signal, signal decomposition is possible and cross estimation between the image signal and the sensor signal is possible.

Hereinafter will be described the configuration and operation of the multi-texture based signal processing apparatus according to the present invention.

5 is a block diagram showing the configuration of a multi-texture based signal processing apparatus according to the present invention.

Referring to FIG. 5, the multi-texture based signal processing apparatus 500 according to the present invention may include an encoder 510 and a decoder 520.

The encoder 510 includes a seed signal selector 511, a feature point detector 512, a variable calculator 513, a texture signal definer 514, and a texture block definer 516. Also, the encoding unit 510 may further include an approximation unit 515 and a compression unit 517.

The seed signal selector 511 classifies the received signal into a signal component unit signal, and selects one frame among a plurality of frames of the signal component unit signal as a seed signal. The seed signal selector 511 defines the remaining frames except the seed signal of the signal component unit signal as the remaining frame signal. That is, the seed signal selector 511 selects one seed signal when the signal configuration unit signal is composed of k frames, and defines the remaining k-1 frames as the remaining frame signal 220. In this case, the signal configuration unit signal refers to a signal constituting the corresponding signal. The feature point detector 512 detects a plurality of feature points from the seed signal selected by the seed signal selector 511. In this case, the feature point detector 512 may detect, as a feature point, a point having a change amount equal to or greater than a predetermined value in a plurality of frames of the signal component unit signal. That is, the feature point detector 512 may detect the specific point as the feature point if the specific point in the seed signal and the remaining frame signal changes more than a predetermined value.

The variable calculator 513 tracks a plurality of feature points in a plurality of frames of the signal component unit signal, and calculates a space-time position conversion variable for each feature point. That is, the variable calculator 513 calculates a spatiotemporal position change variable that defines a change of feature points in the seed signal and the remaining frame signal. The space-time position transformation variable may be in the form of a function indicating an amount of change of the position of the feature point over time.

The texture signal definition unit 514 defines a plurality of texture signals using feature points to which the spatiotemporal position conversion variables calculated by the variable calculator 513 correspond to each other. In this case, the texture signal defining unit 514 may define one texture signal by linking feature points having the same space-time position transformation variables.

The approximation unit 515 approximates a plurality of texture signals by combining texture signals having similar spatiotemporal position transformation variables into one texture signal. That is, the approximation unit 515 may generate a plurality of approximated texture signals and a plurality of approximated spatiotemporal position transformation variables to which the plurality of texture signals and the plurality of spatiotemporal position transformation variables are approximated. In this case, the approximation unit 515 may calculate the similarity between the spatiotemporal position transformation variables by obtaining correlation characteristics of the texture signal signals. In addition, the approximation unit 515 may combine texture signals having similarities between the spatiotemporal position transformation variables having values within a preset threshold into one texture signal.

The texture block definition unit 516 defines each of the plurality of texture signals as the sum of the plurality of texture blocks. In this case, the texture block may be defined as an output of the texture synthesis filter that receives the image sensor excitation signal. The image sensor excitation signal may be represented by a two-dimensional Gaussian function. Of course, the texture block definition unit 516 may define each of the plurality of approximated texture signals as the sum of the plurality of texture blocks.

The compression unit 517 compresses the image sensor excitation signal, the texture of the texture synthesis filter, and the plurality of space-time position conversion variables corresponding to each texture signal. Of course, the compression unit 517 may compress the image sensor excitation signal of the plurality of approximated texture signals, the variables of the texture synthesis filter, and the plurality of approximated spatiotemporal position conversion variables.

The decoder 520 includes a decompressor 521, a texture signal generator 522, a matcher 523, a visual sensor texture generator 524, and a visual sensor texture combiner 525. In addition, the decoder 520 may further include a corrector 526.

The decompressor 521 receives the compressed signal from the encoder 510 and decompresses the compressed signal. The decompressor 521 decompresses the image sensor excitation signal and the texture synthesis filter variable that define each of the plurality of compressed texture signals, and the plurality of spatiotemporal position change variables corresponding to each texture signal.

The texture signal generator 522 generates a plurality of texture blocks using the image sensor excitation signal and the variable of the texture synthesis filter, and generates a texture signal by adding the plurality of texture blocks.

The matching unit 523 matches the texture signal generated by the texture signal generation unit 522 and the space time position conversion variables corresponding to each texture signal and the corresponding texture signal in a one-to-one correspondence. Of course, the matching unit 523 may match the approximated spatiotemporal position transformation variable corresponding to each approximated texture signal and the corresponding approximated texture signal.

The visual sensor texture generator 524 generates a visual sensor texture using the matched texture signal and the space-time position transformation variable. In detail, the visual sensor texture generation unit 524 generates a visual sensor texture composed of a plurality of frames for the texture signal by applying a spatiotemporal position transformation variable that defines a time-dependent movement of feature points, etc., to the texture signal. Of course, the visual sensor texture generator 524 may generate a visual sensor texture by using the matched approximation texture signal and the approximated spatiotemporal position transformation variable.

The visual sensor texture combiner 525 combines the visual sensor textures generated by the visual sensor texture generator 524 corresponding to each texture signal. By combining the visual sensor textures, a plurality of frames of the signal component unit signal are totally reconstructed.

The correction unit 526 filters and corrects artifacts at the coupling boundary of the plurality of combined visual sensor textures. That is, the plurality of visual sensor textures combined by the visual sensor texture combiner 525 are simply restored as a combination, and a defect may occur at a boundary between the visual sensor textures. The correction unit 526 generates a corrected reconstruction signal by performing removal filtering on the defect.

The decomposing unit 527 decomposes the reconstruction signal of the signal structural unit obtained by the correction unit 526, and finally generates a reconstructed image signal and a reconstruction sensor signal. In the present invention, since a signal including an image signal and a sensor signal is represented using a model based on an excitation signal, signal decomposition is possible and cross estimation between the image signal and the sensor signal is possible.

As described above, the multiple texture-based signal processing method and apparatus according to the present invention are not limited to the configuration and method of the embodiments described as described above, but the embodiments may be modified in various ways so that various modifications may be made. All or part of the examples may be optionally combined.

500; Multi-texture based signal processing device
510; Encoding Section
511; Seed signal selector 512; Feature point detector
513; Variable calculator 514; Texture Signal Definition
515; Approximation section 516; Texture block definition
517; Compression part
520; Decoding unit
521; Decompression unit 522; Texture Signal Generator
523; A matching unit 523; Visual sensor texture generator
524; Visual sensor texture coupler 525; Compensator

Claims (18)

Classifying a signal including an input image signal and a sensor signal into a signal component unit signal, and selecting one frame among a plurality of frames of the signal component unit signal as a seed signal;
Detecting a plurality of feature points in the seed signal;
Calculating space-time position transformation variables for each of the feature points by tracking the plurality of feature points in the plurality of frames of the signal configuration unit signal;
Defining a plurality of texture signals using feature points corresponding to the spatiotemporal position transformation variables; And
And defining each of the plurality of texture signals as a sum of a plurality of texture blocks which are outputs of a texture synthesis filter having an image sensor excitation signal as an input. The method according to claim 1,
The image sensor excitation signal is a multiple texture-based signal processing method, characterized in that represented by a two-dimensional Gaussian function. The method according to claim 1,
Compressing the image sensor excitation signal of each of the plurality of texture blocks defining the plurality of texture signals, the variable of the texture synthesis filter, and the spatiotemporal position conversion variable corresponding to each of the plurality of texture signals. Characterized by the multiple texture-based signal processing method. The method according to claim 3,
The compressing step,
And compressing the image sensor excitation signal, the texture synthesis filter variable, and the space-time position conversion variable by a bitstream compression method. Claim 1
In the plurality of texture signals, approximating the texture signals having spatio-temporal position transformation variables having similar values calculated by obtaining correlation characteristics of the texture signal signals with values within a preset threshold, into one texture signal. The multi-texture-based signal processing method further comprising the step. The method according to claim 1,
Detecting the plurality of feature points,
And detecting a point having a change amount greater than or equal to a predetermined value as the feature point in the plurality of frames. The method according to claim 3,
Decompressing the compressed image sensor excitation signal, the variable of the texture synthesis filter, and the space-time position conversion variable corresponding to each of the texture signals;
Generating the plurality of texture blocks using the image sensor excitation signal and the variables of the texture synthesis filter, and generating the texture signal by adding the plurality of texture blocks;
Matching the space signal with the space-time displacement variable corresponding to the texture signal;
Generating a visual sensor texture using the texture signal and the spatiotemporal position transformation variable; And
And combining the visual sensor textures generated in response to the respective texture signals to generate a reconstruction signal. The method according to claim 7,
And correcting the reconstruction signal by filtering artifacts at the combined boundary of the visual sensor textures. The method according to claim 7,
And decomposing the reconstruction signal into a reconstruction image signal and a reconstruction sensor signal. A seed signal selecting unit classifying a signal including an input image signal and a sensor signal into a signal component unit signal, and selecting one frame among a plurality of frames of the signal component unit signal as a seed signal;
A feature point detector for detecting a plurality of feature points from the seed signal;
A variable calculator configured to track the plurality of feature points in the plurality of frames of the signal configuration unit signal and calculate a space-time position conversion variable for each feature point;
A texture signal definition unit defining a plurality of texture signals using feature points corresponding to the spatiotemporal position conversion variables; And
And a texture block definition unit defining each of the plurality of texture signals as a sum of a plurality of texture blocks which are outputs of a texture synthesis filter having an image sensor excitation signal as an input. The method according to claim 10,
The image sensor excitation signal is a multiple texture-based signal processing device, characterized in that represented by a two-dimensional Gaussian box. The method according to claim 10,
And a compression unit configured to compress the image sensor excitation signal of each of the plurality of texture blocks defining the plurality of texture signals, a variable of the texture synthesis filter, and a spatiotemporal position change variable corresponding to each of the plurality of texture signals. Characterized by the multiple texture-based signal processing device. The method of claim 12,
The compression unit,
And the image sensor excitation signal, the variable of the texture synthesis filter, and the space-time position conversion variable are compressed by a bitstream compression method. The method according to claim 10,
In the plurality of texture signals, approximating the texture signals having spatio-temporal position transformation variables having similar values calculated by obtaining correlation characteristics of the texture signal signals with values within a preset threshold, into one texture signal. Multiple texture based signal processing apparatus further comprising an approximation unit. The method according to claim 10,
The feature point detector,
The plurality of texture-based signal processing device, characterized in that for detecting the point having a change amount greater than a predetermined value as the feature point. The method of claim 12,
A decompression unit for decompressing the compressed image sensor excitation signal, the variable of the texture synthesis filter, and the space-time position conversion variable corresponding to each of the texture signals;
A texture signal generator configured to generate the plurality of texture blocks using the image sensor excitation signal and the variables of the texture synthesis filter, and generate the texture signal by adding the plurality of texture blocks;
A matching unit matching the texture signal and the space-time position conversion variable corresponding to the texture signal;
A visual sensor texture generation unit configured to generate a visual sensor texture using the texture signal and the spatiotemporal position transformation variable; And
And a visual sensor texture combiner for combining the visual sensor textures generated in correspondence with the respective texture signals. The method according to claim 16,
And a correction unit for filtering and correcting artifacts at the combined boundary of the visual sensor textures. The method according to claim 16,
And a decomposition unit for decomposing the reconstruction signal into a reconstruction image signal and a reconstruction sensor signal.

Images