WO2021221250A1

WO2021221250A1 - Electronic device for scaling image, and operating method therefor

Info

Publication number: WO2021221250A1
Application number: PCT/KR2020/015426
Authority: WO
Inventors: 안일준; 이태미
Original assignee: 삼성전자 주식회사
Priority date: 2020-04-29
Filing date: 2020-11-05
Publication date: 2021-11-04
Also published as: KR20210133802A; KR102484386B1

Abstract

Provided is a method by which an electronic device generates a second image by scaling a first image, the method determining, on the basis of each region-specific feature of the first image, the relative location relationship between pixels included in a first image and pixels included in a second image, acquiring location transform feature information on the basis of the relative location relationship and values of the pixels included in the first image, and performing a convolution calculation between the location transform feature information and a scaling kernel, thereby generating the second image acquired by scaling the first image.

Description

Electronic device for scaling an image and method for operating the same

The present disclosure relates to an electronic device for scaling an image and an operating method thereof.

In a convolutional neural network (CNN) or the like, a deconvolution layer may be used to generate an output image having a size different from that of an input image.

For example, a first pixel of an output image in which the input image is scaled may be generated based on at least one pixel of the input image. Also, pixels of the input image used to generate each pixel of the output image may be determined based on positions of each pixel of the output image.

However, when the pixel of the input image used to generate each pixel of the output image is determined simply according to the position of each pixel of the output image, pixels of the input image that are not suitable for generating each pixel of the output image are used. problems may arise.

An object of the present disclosure is to solve the above problem, and to provide an electronic device for scaling an image and an operating method thereof.

Another object of the present invention is to provide a computer-readable recording medium in which a program for executing the method in a computer is recorded. The technical problem to be solved is not limited to the technical problems as described above, and other technical problems may exist.

According to an embodiment, pixels of the input image suitable for generating the first pixel of the output image may be determined in consideration of the characteristics of each region of the input image.

1 is a diagram illustrating a method of generating an image by using a deconvolution operation by an image processing apparatus according to an exemplary embodiment.

2 is a diagram for explaining a case in which an image is upscaled by a non-integer multiple according to an embodiment.

3 is a diagram for describing a method of determining a tendency of weights included in a scaling kernel applied to deconvolution for upscaling a first image according to an embodiment.

4 is a diagram for describing a method of modeling a scaling kernel applied to deconvolution for upscaling a first image according to an embodiment.

5 is a diagram illustrating deconvolution for upscaling a first image according to an exemplary embodiment.

6 is a diagram for describing a process in which an image processing apparatus performs deconvolution to upscale a first image, according to an exemplary embodiment.

7 is a diagram for explaining a method of generating transform characteristic information, according to an embodiment.

8 is a block diagram illustrating an example of generating a second image by scaling a first image according to an embodiment.

9 is a diagram illustrating an example of obtaining an offset and a compensation weight based on a first kernel and a second kernel according to an embodiment.

10 is a diagram illustrating an example of obtaining an offset and a compensation weight based on a first kernel and a third kernel according to an embodiment.

11 is a block diagram illustrating an internal configuration of an electronic device according to an exemplary embodiment.

12 is a block diagram illustrating an internal configuration of an electronic device according to an exemplary embodiment.

13 is a flowchart illustrating a method of scaling an image according to an embodiment.

As a technical means for achieving the above technical problem, a first aspect of the present disclosure is a method of generating a second image by scaling a first image in an electronic device, based on characteristics of each region of the first image determining a relative positional relationship between pixels included in the first image and pixels included in a second image; obtaining position transformation characteristic information based on the determined relative positional relationship and values of pixels included in the first image; and generating a second image in which the first image is scaled by performing a convolution operation between the position transformation feature information and a scaling kernel.

Also, a second aspect of the present disclosure provides an electronic device for generating a second image by scaling a first image, comprising: a memory configured to store the first image; and a relative positional relationship between pixels included in the first image and pixels included in a second image is determined based on the characteristics of each region of the first image, and the determined relative positional relationship and At least one method for generating a second image in which the first image is scaled by obtaining position transformation characteristic information based on the values of included pixels and performing a convolution operation between the position transformation characteristic information and a scaling kernel An electronic device including a processor may be provided.

In addition, a third aspect of the present disclosure may provide a recording medium in which a program for performing the method of the first aspect is stored.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings so that those of ordinary skill in the art can easily implement them. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is "connected" with another part, this includes not only the case of being "directly connected" but also the case of being "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, this means that other components may be further included rather than excluding other components unless otherwise stated.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

Referring to FIG. 1 , an electronic device 1000 according to an embodiment may generate an image by using a neural network 30 . The electronic device 1000 according to an embodiment may use the neural network 30 to generate a second image 20, output, based on the first image 10, input.

The electronic device 1000 according to an embodiment generates a second image 20 based on the first image 10 , and displays the generated second image 20 on a display included in the electronic device 1000 or It can be displayed on an external display device.

The electronic device 1000 according to an embodiment may be implemented in various forms. For example, the electronic device 1000 described herein may include a digital camera, a smart phone, a laptop computer, a tablet PC, an electronic book terminal, a digital broadcasting terminal, and a personal digital assistant (PDA). , a Portable Multimedia Player (PMP), a navigation system, an MP3 player, a vehicle, and the like, but is not limited thereto. The electronic device 1000 described herein may be a wearable device that can be worn by a user. Wearable devices include accessory type devices (e.g., watches, rings, wristbands, ankle bands, necklaces, eyeglasses, contact lenses), head-mounted-devices (HMDs), textile or clothing-integrated devices (e.g., electronic clothing), a body attachable device (eg, a skin pad), or a bioimplantable device (eg, an implantable circuit).

The neural network 30 according to an embodiment may include one or more deconvolution layers, and in each of the deconvolution layers, an image input to the deconvolution layer and a scaling kernel (Scaling) A deconvolution operation of the kernel) may be performed, and an output image may be generated as a result of the deconvolution operation.

The deconvolution operation according to an embodiment is one of various types of convolution operations.

The scaling kernel according to an embodiment may be a matrix including values previously learned in order to generate an output image based on an input image. Not limited to the above-described example, the scaling kernel may be various types of data from which an output image may be generated through a deconvolution operation with an input image.

The deconvolution operation according to an embodiment may be used to upscale the input image 10 in a convolutional neural network (CNN), for example, super-resoultion It can be used in various fields such as image generation, auto-encoder, and style transfer. However, the present invention is not limited thereto.

According to an embodiment, the size of the upscaled second image 20 (output) generated as a result of the deconvolution operation may be larger than the size of the first image 10 (input).

Referring to FIG. 2 , a display on which an image is displayed may be configured in units of modules, and in this case, the size of the display may be variously determined according to a method of combining modules. For example, when the size of one module 201 is 9.8 inches and has a resolution of 480x270, the first display 210 having Ultra-HD (4K) resolution including 8x8 modules may be configured. Alternatively, the second display 220 having a resolution of 4320 x 2430 (eg, about 4.5K resolution) including 9 x 9 modules may be configured.

The electronic device 1000 according to an embodiment may output an image having a 4K resolution to the first display 210 by up-scaling the image having the 4K resolution by 9/8 times to the second display 220 . can be printed on

3 illustrates a cubic B-spline interpolation method to find a tendency of weights applied to pixel values of a first image according to a pixel position of a second image when calculating a pixel value of a second image according to an embodiment; An example of application is shown.

Referring to FIG. 3 , the first graph 310 is a pixel of the second image using four pixel values (x _i-1 , x _i , x _i+1 , x _i+2 ) of the first image. In the case of calculating the values _{, curves B 0} (u), B ₁ (u) indicating a weight applied to each of the _{four pixel values (x i-1} , x _i , x _i+1 , x _{i+2 )} ), B ₂ (u), B ₃ (u)). At this time, the curves B ₀ (u), B ₁ (u), B ₂ (u), and B ₃ (u)) are cubic B spline curves, and as shown in FIG. 3 , B ₀ (u) , B ₁ (u), B ₂ (u), and B ₃ (u) appear as a function of u. The meaning of the variable u will be described in detail below.

For example, assuming that the first image and the second image have the same size and a distance between adjacent pixels included in the first image is 1, positions of pixels in the first image are expressed as integers, and the positions of pixels in the second image are expressed as integers. The positions of the pixels of may be expressed as real numbers. For example, pixels of the first image _{may be expressed as x i} (i=0, 1, 2, 3,..., n), and index i may indicate a coordinate value (position) of the corresponding pixel. . Pixels of the second image _{may be expressed as y j} (j = real number), where j may indicate a coordinate value (position) of the corresponding pixel.

Meanwhile, referring to FIG. 3 , _{the value of the pixel y j} included in the second image is discarded on j to determine the integer i, and four pixels among the pixels included in the first image are selected. It may be determined based on (x _i-1 , x _i , x _i+1 , x _{i+2 ).}

For example, y _j =B ₀ (u)*x _i-1 +B ₁ (u)*x _i +B ₂ (u)*x _i+1 +B ₃ (u)*x _i+2 can

_{In this case, u is when the point corresponding to the pixel y j} included in the second image is located between the pixels xi and xi+1 of the first image, assuming that the first image and the second image have the same size. , can be expressed as the distance between the pixel xi and _{the point corresponding to the pixel y j .} Accordingly, the value of u is greater than or equal to 0 and less than 1. Referring to FIG. 3 , u may be expressed as a difference (ji) between a value (eg, an integer i) obtained by truncating j and j.

Meanwhile, the curves B ₀ (u), B ₁ (u), B ₂ (u), and B ₃ (u)) may be represented by a matrix B(u), as shown in FIG. 4 , Four pixels (x _i-1 , x _i , x _i+1 , x _i+2 ) of one image may be represented by a matrix I. Accordingly, _{the value of the pixel y j} may be determined by Equation 1 below.

In this case, the operation of symbols <A, B> _F means a convolution operation. That is, it means an operation of summing values obtained by multiplying elements at the same position in matrix A and matrix B.

On the other hand, the matrix B of FIG. 3 applies B-cubic spline curves to find the tendency of weights applied to the input I, and the remaining parts 330 except for the matrix for u in the matrix B of FIG. 3 . , the coefficient (1/6) and the elements of the matrix B') can be determined to an appropriate value through training.

Referring to FIG. 4 , the remaining parts 330 (eg, coefficients (1/6) and elements of matrix B′) except for the matrix for u in the matrix B of FIG. 3 are as shown in FIG. 4 . , can be modeled to be expressed as a trainable value θ _{ij .} Accordingly, the matrix K of FIG. 4 represents a trainable scaling kernel applied to deconvolution for upscaling the first image, and may be expressed as a product of a trainable matrix θ(410) and a matrix U. For example, the matrix K may be expressed as in Equation 2 below.

In this case, the matrix θ and the matrix U may be respectively expressed as shown in FIG. 4 .

Also, the deconvolution operation for upscaling the first image may be expressed as Equation 3 below.

In Equation 3, the matrix I _i represents pixel values of the first image, which are used to calculate _{a pixel value y j} included in the second image. In addition, the matrix U _j is, the matrix θ and, instead of being calculated, the matrix I _i and the operation such that the position of the matrix U _j can be changed.

Accordingly, the matrix K _j (=θU _j ) is a matrix expressed as a function _{of u j} , which is a matrix that varies according to the value of _{u j} , but the matrix θ does not change according to the value of _{u j (space-varient).} invariant) matrix. Also, since the order may be changed in the convolution operation, the pixel value yj of the second image may be expressed as Equation 4 below.

The meaning of the parameters in Equation 4 will be described in detail below with reference to FIG. 5 .

Referring to FIG. 5 , x denotes a first image, and y denotes a second image obtained by up-scaled by a non-integer multiple of x. In addition, i is a value indicating the index of pixels included in the first image, and when x and y are represented by the same size and the distance between adjacent pixels included in the first image is 1, A value indicating the position of pixels. i may be expressed as an integer value of 0, 1, ..., n. Also, j is a value indicating indexes of pixels included in the second image, and is a value indicating positions of pixels included in the second image. When positions (eg, coordinates) of pixels of the first image are expressed as integers, positions (eg, coordinates) of pixels of the second image may be expressed as real numbers. Therefore, j can be expressed as a real value.

As described in FIG. 4 , the value of _{the pixel (y j} _{) included in the second image may be expressed as <U j} ·I _i ^T , Θ> _F , where the matrix U _j is expressed as a function for _{u j .} and is determined according to the _{u j value.} In this case, the u _j value is a value determined according to the relative distance _{between the pixel y j} _{and the pixel x i} of the first image, and the _{value of u j} is 0 or more and less than 1. In addition, the index i of the _{pixel x i} is determined as a value obtained by performing a discard operation on the index j of the _{pixel y j .}

For example, _{when the value of pixel y 1.2} is obtained, j is 1.2, so the value 1 obtained by performing the rounding operation on 1.2 becomes the value i, and x _i becomes x ₁ . Also, u _1.2 becomes 0.2. Accordingly, the matrix Uj is determined differently depending on the index j of the pixel value yj to be calculated.

Also, the matrix Ii is determined differently depending on the index j of _{y j .} For example, the index i value is determined according to the index j, and the matrix Ii=[x _i-1 , x _i , x _i+1 , x _i+2 ] is determined according to the index i. For example, when the index j is 1.2 and the index i is 1, the matrix I ₁ =[x ₀ , x ₁ , x ₂ , x ₃ ] may be determined. However, the matrix Ii may be configured differently according to the number of elements included in the matrix Ii.

Meanwhile, the matrix Θ is a value determined by training, and may be equally applied regardless of the index j or u _j _{of the pixel value y j to be calculated.}

The image processing apparatus 100 according to an exemplary embodiment may determine a relative positional relationship between pixels included in the first image 610 and pixels included in the second image 620 and pixels included in the first image 610 . Based on the value, location transformation feature information 640 may be obtained.

For example, the image processing apparatus 100 may generate the transformation characteristic information 630 using pixel values included in the first image. As shown in FIG. 6 , the number of transformation feature information may be n, where n is equal to the number of pixels included in the first image used to calculate the value of one pixel included in the second image. . 6 illustrates a case in which one pixel value of the second image is calculated using four pixel values included in the first image.

The transformation characteristic information 630 according to an embodiment includes four transformation characteristic information (first transformation characteristic information (X _k-1 ), second transformation characteristic information (X _k ), and third transformation characteristic information (X)). _k+1 ) and fourth transform characteristic information (X _k+2 )). A method of generating the transform characteristic information 630 will be described in detail with reference to FIG. 7 .

Referring to FIG. 7 , assuming that the first image 610 and the second image 620 have the same size and the distance between adjacent pixels included in the first image 610 is 1, the pixels of the first image The positions of the pixels may be expressed as integers, and positions of pixels of the second image may be expressed as real numbers. For example, the pixels of the first image 610 _{may be expressed as x i} (i=0, 1, 2, 3,..., n), and the index i represents the coordinate value (position) of the pixel. can indicate Pixels of the second image _{may be expressed as y j} (j = j0, j1, j2, ..., jm, j are real numbers), and j may represent a coordinate value (position) of the corresponding pixel. Also, pixel values included in the first image may be expressed as x _i = x ₀ , x ₁ , x ₂ ,..., x _n , and pixel values included in the second image are y _j =y _j0 , y _It can be expressed as j1 , y _j2 ,..., y _{jm .}

The transformation characteristic information 630 according to an embodiment may be generated to have the same size as the second image 620 . For example, if the size of the second image is W X H, the size of the transform characteristic information may also be W x H.

Sample values included in each of the transformation characteristic information 630 according to an embodiment may be determined based on a position (index) of a pixel of the second image 620 corresponding to each of the samples. For example, the first pixel 740 of the second image 620 may have a value _{of index j 0} _{, and a value k 0} obtained by performing a discard operation on _{j 0} may be determined. Accordingly, the sample values S11 , S21 , S31 , and S41 of the position corresponding to the first pixel 740 are determined based on the value _{k 0 .}

Further, the sample values included in the conversion characteristics each of information 630 according to one embodiments, the determined addition to k _0, first on the basis of each section characterized in the first image 610, the determined at least one offset value ( Sample values S11 , S21 , S31 , and S41 of positions corresponding to the first pixel 740 may be determined according to d _k0-1 , d _k0 , d _k0+1 , and d _{k0+2 .}

The offset value according to an embodiment may be determined by the number of pixels of the first image 610 used to generate one pixel of the second image 620 for each pixel of the first image 610 . . For example, when one pixel is generated in the second image 620 and four different pixels of the first image 610 are used, for each pixel of the first image 610, four pixels are generated. Each offset value may be determined.

The offset value according to an embodiment may be determined according to a result of performing a convolution operation between the first kernel and the first image 610 learned in advance to determine the offset value. The first kernel according to an embodiment may be pre-learned so that a difference between the second image 620 generated by scaling the first image 610 using the first kernel and the second image set as a correct answer is minimized. can

According to the convolution operation according to an embodiment, values corresponding to the feature information may be extracted for each region of the first image 610 . Accordingly, according to an embodiment, by performing a convolution operation between the first image 610 and the pre-learned first kernel, the offset value determined differently according to the characteristics of each region of the first image 610 is the first It may be determined for each pixel of the image.

According to an embodiment, _{by determining a value obtained by adding an offset value to a value of k 0} as an index value for a pixel of the first image 610 , sample values S11 , S21 , and S31 of a position corresponding to the first pixel 740 are determined. , S41) may be determined.

For example, the first sample value S11 of the first transformation feature information 731 is the pixel value ( _{S11 ) of the first image having k 0} -1+d _{k0-1 as an index.}

), and the second sample value S21 of the second transformation feature information 732 is the pixel value ( _{S21 ) of the first image having k 0 +} d _{k0 as an index}

), and the third sample value S31 of the third transformation feature information 733 is the pixel value ( _{S31 ) of the first image having k 0} +1+ d _{k0+1 as an index.}

), and the fourth sample value S41 of the fourth transform characteristic information 734 is the pixel value ( _{S41 ) of the first image having k 0} +2+d _{k0+2 as an index.}

) is determined by

According to an embodiment, when the index value to which the offset value is applied is not an integer value, the sample value is a pixel value of the first image corresponding to a value obtained by raising the index value to an integer value and a value rounded down by an integer value. According to a value after a decimal point of the index value, it may be determined as a sum of weighted values.

For example, when the index value to which the offset value is applied is 2.3, x(2) and x(3), which are pixel values of the first image having 2 and 3 as index values, which are values obtained by rounding up and rounding down 2.3 to integer values. ), a sample value may be determined. In addition, since the value after the decimal point of 2.3 is 0.3, it is preferable that the sample value is determined as a value closer to the pixel value of index 2 among the pixel values of

indexes

2 and 3. Accordingly, a value obtained by multiplying the pixel values of x(2) and x(3) by 0.7 and 0.3, respectively, that is, a value of x(2)*0.7+x(3)*0.3 may be determined as a sample value.

Accordingly, after determining _{the index k 0} value of the reference pixel of the first image 610 at a position corresponding to the first pixel 740 of the second image 620 , the electronic device 1000 according to an embodiment , according to the offset values d _k0-1 , d _k0 , d _k0+1 , d _k0+2 with respect to the reference pixel k ₀ , sample values S11 at positions corresponding to the first pixel 740 , S21, S31, S41) may be determined.

According to an embodiment, before the offset value is applied, the pixel values of the first image, which are each determined as the sample value, may be determined as the values of pixels located at regular intervals of one pixel, respectively. However, according to the index value to which the offset value is applied, pixel values of the first image determined as sample values may be determined as values of pixels located at various intervals.

For example, among the regions of the first image 610 , offset values for pixel values included in a high-frequency region (eg, an edge, a boundary line) are used to generate one pixel of the second image 620 . , may be determined such that an interval between pixels of the first image 610 becomes relatively narrow. Accordingly, according to an embodiment, as pixels of the second image 620 are generated based on the pixel values of the first image 610 located in a narrow area, in the high-frequency region of the second image 620, the block is formed. Alternatively, an error in which the line area is blurred may be minimized.

On the other hand, among the regions of the first image 610 , offset values for pixel values included in a low-frequency region (eg, a background, a monochromatic surface) are used to generate one pixel of the second image 620 , It may be determined that an interval between pixels of the first image 610 is relatively wide.

The electronic device 1000 may acquire the first to fourth transformation

characteristic information

731 , 732 , 733 , and 734 having the same size as the second image 620 in the same manner.

Referring back to FIG. 6 , the image processing apparatus 100 _{multiplies the first to fourth transformation characteristic information 630 by u j} ³ , u _j ² , u _j ¹ , and u _j ⁰ , respectively, to obtain position transformation characteristic information. 640 may be created.

In addition, the electronic device 1000 performs a convolution operation on the position transformation feature information 640 and the scaling kernel 650 , so that the first image 610 is upscaled by a non-integer multiple of the second image 620 . can be obtained. In addition, the number of weights included in the scaling kernel according to an embodiment may be determined based on the number n of pixels included in the first image used to calculate one pixel value included in the second image.

According to an embodiment, in the same manner as when an offset value is determined for each pixel of the first image 610 , compensation weight values for each pixel of the first image 610 may be determined. According to an embodiment, the same number of compensation weight values as the number of offset values determined for each pixel of the first image 610 may be determined, and the compensation weight values may be applied to weight values included in the scaling kernel 650 . have.

Compensating weight values according to an embodiment are to correct the scaling kernel 650 in consideration of that the pre-learned scaling kernel 650 is a kernel learned in a state in which the offset value according to the embodiment is not used. value for

_{Compensation weight values w k0-1} , w _k0 , w _k0+1 , w _k0+2 according to an embodiment are the same as the offset value, based on the characteristics of each region of the first image 610 , For each pixel of the first image 610 , the number of pixels of the first image 610 used to generate one pixel of the second image 620 may be determined. For example, when one pixel is generated in the second image 620 and four different pixels of the first image 610 are used, for each pixel of the first image 610, four pixels are generated. Compensation weight values may be determined respectively.

The compensation weight value according to an embodiment may be determined according to a result of performing a convolution operation between the second kernel learned in advance and the first image 610 to determine the compensation weight value. The second kernel according to an embodiment uses the second kernel together with the first kernel, so that the difference between the second image 620 generated by scaling the first image 610 and the second image set as a correct answer is To be minimized, it can be learned in advance.

In addition, the compensation weight value according to an embodiment is based on a third kernel in which a convolution operation is performed with an offset value obtained by the first kernel instead of a convolution operation performed with the first image 610 . may be decided. For example, the compensation weight value may be determined according to a result of performing a convolution operation between the third kernel learned in advance to determine the compensation weight value and the offset values. Like the second kernel, the third kernel according to an embodiment uses the third kernel together with the first kernel, and the second image 620 generated by scaling the first image 610 and the It may be learned in advance so that the difference between the second images is minimized.

According to the convolution operation according to an embodiment, values corresponding to the feature information may be extracted for each region of the first image 610 . Accordingly, according to an embodiment, by performing a convolution operation between the first image 610 and the pre-learned second kernel, the compensation weight value differently determined according to the characteristics of each region of the first image 610 is It may be determined for each pixel of one image. Even when the third kernel is used, since the offset value obtained by the convolution operation of the first image 610 and the convolution operation are performed, the compensation weight is determined differently according to the characteristics of each region of the first image 610 . A value may be determined for each pixel of the first image.

According to an embodiment, after compensation weight values are applied to the weight values of the scaling kernel 650 , a convolution operation between the position transformation feature information 640 and the scaling kernel 650 may be performed.

_{For example, the value of the pixel y j0} of the second image 620 that is upscaled by performing the deconvolution operation of FIG. 6 on the first image 610 may be expressed as Equation 5 below.

8 is a block diagram illustrating an example of generating a second image 620 by scaling a first image 610 according to an embodiment.

Referring to FIG. 8 , the electronic device 1000 may generate the second image 620 by scaling the first image 610 in the scaling processor 810 .

The scaling processing unit 810 according to an embodiment includes an offset obtaining unit 811 , an index determining unit 812 , a compensation weight obtaining unit 813 , and a convolution calculating unit 814 , including the first image 610 . This scaled second image 620 may be generated.

The offset acquirer 811 according to an embodiment may be configured to, for each pixel of the first image 610 , as many as the number of pixels of the first image 610 used to generate one pixel of the second image 620 . may determine the offset values of . The offset acquirer 811 according to an embodiment may be configured to determine each pixel of the first image according to a result of performing a convolution operation between a pre-learned first kernel and the first image 610 to determine an offset value. It is possible to obtain an offset value for .

The index determiner 812 according to an exemplary embodiment may include an index value (ie, the first pixel 740 ) of a reference pixel of the first image 610 corresponding to the first pixel 740 of the second image 610 . When the value of the index value of ) is k ₀ , the offset values (d _k0-1 , d _k0 , d _k0+1 , d _k0+2 ) are applied to the index (eg, the offset value is added to the index value) ), index values for pixels of the first image 610 for determining sample values included in each of the transformation characteristic information 630 may be determined. For example, the first sample value S11 for the first transformation characteristic information 731 may be determined as a pixel value of the first image 610 having _{k 0} -1+d _{k0-1 as an index.} According to an embodiment, when the index value is a real value rather than an integer, a sample value may be determined as a weighted sum between a plurality of pixel values of the first image 610 .

Accordingly, according to an embodiment, an interval between pixels of the first image 610 corresponding to sample values for generating one pixel of the second image 620 is a region characteristic of the first image 610 . It may be adaptively determined according to an offset value determined according to (eg. frequency characteristic).

The compensation weight acquirer 813 according to an embodiment may generate the second image for each pixel of the first image 610 based on the characteristics of each region of the first image 610, identical to the offset value. Compensation weight values may be determined as many as the number of pixels of the first image 610 used to generate one pixel of 620 .

The compensation weight acquirer 813 according to an embodiment may determine the compensation weight value according to a result of performing a convolution operation between the pre-learned second kernel and the first image 610 to determine the compensation weight value. can be obtained

In addition, the compensation weight acquirer 813 according to an embodiment may perform a convolution operation between the offset values obtained by the third kernel and the first kernel learned in advance to determine the compensation weight value instead of the second kernel. According to the performed result, a compensation weight value may be obtained.

The compensation weight value obtained according to an embodiment is obtained by applying the compensation weight values to the weight values of the scaling kernel 650, and then, by the convolution operation unit 814, the position transformation feature information 640 and the scaling kernel 650 ) convolution operation can be performed.

According to one embodiment, for the pixel (x _i) of the index i of the pixel in the first image 610, a convolution operation offset 910 is performed. Thus, corresponding to the pixel (x _i) with the first kernel Values d _i-1 , d _i , d _i+1 , d _i+2 can be obtained.

The first kernel according to an embodiment may be configured as a 3x3 matrix including values previously learned in order to obtain offset values for each pixel of the first image 610 . The first kernel according to an embodiment may be pre-learned so that appropriate offset values may be output based on a characteristic (eg, a high-frequency region or a low-frequency region) of a region to which the _{pixel x i belongs.}

According to an embodiment, as a value obtained by discarding an index value of pixel y of the second image 620 is obtained as i according to the obtained offset value, the reference pixel of the first image 610 for the pixel y is x _{In the} case of i, the index values of the pixels of the first image 610 used to generate the pixel y are positions adjusted to offset values other than i-1, i, i+1, i+2, i It may be determined as -1+d _i-1 , i+d _i , i+1+d _i+1 , i+2+d _i+2 .

Therefore, instead of i-1, i, i+1, i+2, where the position interval is constant as 1 pixel, an offset value adaptively determined according to a characteristic (eg, a frequency characteristic) of each region of the first image 610 . A pixel value of the second image 620 may be generated based on the pixel values of the first image 610 whose position is adjusted to .

According to an embodiment, a convolution operation 920 with a second kernel is performed on _{a pixel (x i} ) of index i among pixels of the first image 610 , thereby compensating corresponding to the _{pixel (x i ).} Weight values w _i-1 , w _i , w _i+1 , w _i+2 may be obtained.

The second kernel according to an embodiment may be configured as a 3x3 matrix including values previously learned in order to obtain compensation weight values for each pixel of the first image 610 . The second kernel according to an embodiment may be pre-learned so that appropriate compensation weight values may be output based on a characteristic (eg, a high-frequency region or a low-frequency region) of a region to which the _{pixel x i belongs.}

Compensation weight values obtained based on the second kernel according to an embodiment may be applied to weight values of the scaling kernel 940 used to obtain the pixel y of the second image 620 . According to an embodiment, as shown in Equation 5 above, compensation weight values may be applied 930 to the pixels of the first image 610 to which the offset values are applied and corresponding components.

According to an embodiment, as in the example shown in FIG. 9 , the convolution operation 1010 with the first kernel is performed on _{the pixel (x i ) of index i among the pixels of the first image 610 .} , offset values d _i-1 , d _i , d _i+1 , d _i+2 corresponding to the pixel x _i may be obtained.

_{In addition, according to an embodiment, the offset values d i-1} , d _i , d _i+1 , d _i+2 , which are the results of the convolution operation 1010 with the first kernel, and the third kernel By performing the convolution operation 1020 of , compensation weight values w' _i-1 , w' _i , w' _i+1 , w' _i+2 _{corresponding to the pixel x i} may be obtained. .

The third kernel according to an embodiment _{performs a convolution operation with the offset values d i-1} , d _i , d _i+1 , and d _i+2 for each pixel of the first image 610 . In order to obtain the compensation weight values, it may be configured as a 3x3 matrix including pre-learned values. In the third kernel according to an embodiment, _{the offset values d i-1} , d _i , d _i+1 , d _{in which the characteristics of the region to which the pixel x i} belongs (eg, the high frequency region or the low frequency region) are already considered. _{Based on i+2} ), it can be learned in advance so that suitable compensation weight values can be output.

_{Compensation weight values w i-1} , w _i , w _i+1 , w _i+2 obtained based on the convolution operation 920 according to the second kernel of FIG. 9 and the third _{Compensation weight values w' i-1} , w' _i , w' _i+1 , w' _i+2 obtained based on the convolution operation 1020 according to the kernel may correspond to each other. The second kernel and the third kernel according to an embodiment are different kernels learned so that an optimal second image 620 can be generated by scaling the first image 610 , so the compensation weight values w _i-1 , w _i , w _i+1 , w _i+2 ) and the compensation weight values (w' _i-1 , w' _i , w' _i+1 , w' _i+2 ) are substantially the same can The above-described example is not limited, and compensation weight values applicable to each weight of the scaling kernel 1040 may be obtained based on a learned kernel according to various methods.

Compensation weight values obtained based on the third kernel according to an embodiment may be applied to weight values of the scaling kernel 1040 used to obtain the pixel y of the second image 620 . According to an embodiment, as shown in Equation 5 above, compensation weight values may be applied 1030 to the pixels of the first image 610 to which the offset values are applied and corresponding components.

11 is a block diagram illustrating an internal configuration of the electronic device 1000 according to an embodiment.

12 is a block diagram illustrating an internal configuration of the electronic device 1000 according to an embodiment.

Referring to FIG. 11 , the electronic device 1000 may include a processor 1300 and a memory 1700 . However, not all of the components shown in FIG. 11 are essential components of the electronic device 1000 . The electronic device 1000 may be implemented by more components than those illustrated in FIG. 11 , or the electronic device 1000 may be implemented by fewer components than those illustrated in FIG. 11 .

For example, as shown in FIG. 12 , the electronic device 1000 according to an embodiment includes a user input unit 1100 and a sensing unit 1400 in addition to the control unit 1300 and the memory 1700 . ), an A/V input unit 1600 , a communication unit 1500 , and an output unit 1200 may be further included.

The user input unit 1100 means a means for a user to input data for controlling the electronic device 1000 . For example, the user input unit 1100 includes a key pad, a dome switch, and a touch pad (contact capacitive method, pressure resistance film method, infrared sensing method, surface ultrasonic conduction method, integral type). There may be a tension measurement method, a piezo effect method, etc.), a jog wheel, a jog switch, and the like, but is not limited thereto.

According to an embodiment, the user input unit 1100 may receive a user input for generating a second image by scaling the first image.

The output unit 1200 may output an audio signal, a video signal, or a vibration signal, and the output unit 1200 may include a display unit 1210 , a sound output unit 1220 , and a vibration motor 1230 . have.

The display unit 1210 displays and outputs information processed by the electronic device 1000 . According to an embodiment, the display unit 1210 may display the second image generated by scaling the first image.

On the other hand, when the display unit 1210 and the touch pad form a layer structure to form a touch screen, the display unit 1210 may be used as an input device in addition to an output device. The display unit 1210 includes a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a three-dimensional display ( 3D display) and electrophoretic display (electrophoretic display) may include at least one. Also, depending on the implementation form of the electronic device 1000 , the electronic device 1000 may include two or more display units 1210 .

The sound output unit 1220 outputs audio data received from the communication unit 1500 or stored in the memory 1700 . The vibration motor 1230 may output a vibration signal. Also, the vibration motor 1230 may output a vibration signal when a touch is input to the touch screen.

According to an embodiment, the sound output unit 1220 and the vibration motor 1230 may output information related to a result of scaling an image.

The processor 1300 generally controls the overall operation of the electronic device 1000 . For example, the processor 1300 executes programs stored in the memory 1700 , and thus the user input unit 1100 , the output unit 1200 , the sensing unit 1400 , the communication unit 1500 , and the A/V input unit 1600 . ) can be controlled in general.

The electronic device 1000 may include at least one processor 1300 . For example, the electronic device 1000 may include various types of processors, such as a central processing unit (CPU), a graphics processing unit (GPU), and a neural processing unit (NPU).

The processor 1300 may be configured to process instructions of a computer program by performing basic arithmetic, logic, and input/output operations. The command may be provided to the processor 1300 from the memory 1700 or may be received through the communication unit 1500 and provided to the processor 1300 . For example, the processor 1300 may be configured to execute instructions according to program codes stored in a recording device such as a memory.

The processor 1300 according to an embodiment may generate a second image in which the first image is a scaled image by performing a convolution operation between the position transformation feature information between the first image and the second image and the scaling kernel. have. The position transformation characteristic information according to an embodiment may be obtained based on values of pixels included in the first image and a relative positional relationship between pixels of the first image and pixels of the second image. Also, the relative positional relationship according to an embodiment may be determined based on a characteristic of each region of the first image. For example, the relative positional relationship may be obtained based on a result of performing a convolution operation between at least one pixel of the first image and the first kernel.

According to an embodiment, a relative positional relationship may be determined by identifying n pixels included in the first image with respect to the position of one pixel included in the second image based on the characteristics of each region of the first image. have. According to an embodiment, the n pixels may be used to obtain one pixel of the second image.

According to an embodiment, the processor 1300 performs a discard operation on an index value indicated by one pixel of the second image, determines k _j (k _j = integer), and adds Based on this, n offset values corresponding to _{the k j may be obtained.} According to an embodiment, the n offset values may be obtained based on a convolution operation between at least one pixel of the first image including the pixel of the first image corresponding to _{k j and the first kernel.}

Also, the processor 1300 may identify n pixels to obtain the one pixel based on the n offset values. For example, a value obtained by adding n corresponding offset values to n index values (eg, k _j -1, k _j , k _j +1, k _j _{+2) determined based on k j is used as an index.} , n pixels can be identified.

As the relative positional relationship according to an embodiment is determined based on the offset value determined according to the characteristics of each region of the first image, the scaling kernel also provides a compensation weight value obtained based on the characteristics of each region of the first image. can be corrected according to Since the scaling kernel according to an embodiment is a kernel learned in a state in which the offset value is not taken into consideration, it may be used to generate the second image after being corrected according to the compensation weight value.

The compensation weight value according to an embodiment is determined according to a result of performing a convolution operation between at least one pixel of the first image and the second kernel, or a result of the above-described convolution operation of the first kernel and the third It may be determined according to a result of performing a convolution operation with the kernel.

The processor 1300 according to an embodiment may determine at least one compensation weight value to be applied to each weight value included in the scaling kernel, based on the characteristics of each region of the first image. For example, the compensation weight value may be obtained according to a result of a convolution operation performed by the second kernel or the third kernel. In addition, the processor 1300 applies at least one compensation weight value to at least one weight value of the scaling kernel, and then, based on the scaling kernel to which the at least one compensation weight value is applied, the second image to which the first image is scaled. can create

The sensing unit 1400 may detect a state of the electronic device 1000 or a state around the electronic device 1000 , and transmit the sensed information to the processor 1300 .

The sensing unit 1400 includes a geomagnetic sensor 1410 , an acceleration sensor 1420 , a temperature/humidity sensor 1430 , an infrared sensor 1440 , a gyroscope sensor 1450 , and a position sensor. (eg, GPS) 1460 , a barometric pressure sensor 1470 , a proximity sensor 1480 , and at least one of an illuminance sensor 1490 , but is not limited thereto.

The communication unit 1500 may include one or more components that allow the electronic device 1000 to communicate with the server 2000 or an external device (not shown). For example, the communication unit 1500 may include a short-range communication unit 1510 , a mobile communication unit 1520 , and a broadcast receiving unit 1530 .

Short-range wireless communication unit 1510, Bluetooth communication unit, BLE (Bluetooth Low Energy) communication unit, short-range wireless communication unit (Near Field Communication unit), WLAN (Wi-Fi) communication unit, Zigbee (Zigbee) communication unit, infrared ( It may include an IrDA, infrared Data Association) communication unit, a Wi-Fi Direct (WFD) communication unit, an ultra wideband (UWB) communication unit, an Ant+ communication unit, and the like, but is not limited thereto.

The mobile communication unit 1520 transmits/receives a radio signal to and from at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to transmission/reception of a voice call signal, a video call signal, or a text/multimedia message.

The broadcast receiver 1530 receives a broadcast signal and/or broadcast-related information from the outside through a broadcast channel. The broadcast channel may include a satellite channel and a terrestrial channel. According to an embodiment, the electronic device 1000 may not include the broadcast receiver 1530 .

According to an embodiment, the communication unit 1500 may transmit/receive data required for scaling an image. For example, the communication unit 1500 may receive a first image for scaling from the outside, or transmit a second image generated as a result of scaling the first image to the outside.

The A/V (Audio/Video) input unit 1600 is for inputting an audio signal or a video signal, and may include a camera 1610 , a microphone 1620 , and the like. The camera 1610 may obtain an image frame such as a still image or a moving image through an image sensor in a video call mode or a shooting mode. The image captured through the image sensor may be processed through the processor 1300 or a separate image processing unit (not shown).

The microphone 1620 receives an external sound signal and processes it as electrical voice data.

The memory 1700 may store a program for processing and control of the processor 1300 , and may also store data input to or output from the electronic device 1000 .

The memory 1700 according to an embodiment may store data required to generate a second image by scaling the first image. For example, the memory 1700 may store a first image and kernels (eg, a scaling kernel, a first kernel, a second kernel, and a third kernel) necessary to generate a second image through a convolution operation. can

The memory 1700 may include a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory), and a RAM. (RAM, Random Access Memory) SRAM (Static Random Access Memory), ROM (Read-Only Memory), EEPROM (Electrically Erasable Programmable Read-Only Memory), PROM (Programmable Read-Only Memory), magnetic memory, magnetic disk , may include at least one type of storage medium among optical disks.

Programs stored in the memory 1700 may be classified into a plurality of modules according to their functions, for example, may be classified into a UI module 1710 , a touch screen module 1720 , a notification module 1730 , and the like. .

The UI module 1710 may provide a specialized UI, GUI, or the like that interworks with the electronic device 1000 for each application. The touch screen module 1720 may detect a touch gesture on the user's touch screen and transmit information about the touch gesture to the processor 1300 . The touch screen module 1720 according to some embodiments may recognize and analyze a touch code. The touch screen module 1720 may be configured as separate hardware including a controller.

Various sensors may be provided inside or near the touch screen to detect a touch or a proximity touch of the touch screen. A tactile sensor is an example of a sensor for detecting a touch of a touch screen. A tactile sensor refers to a sensor that senses a touch of a specific object to the extent or higher than that felt by a human. The tactile sensor may sense various information such as the roughness of the contact surface, the hardness of the contact object, and the temperature of the contact point.

The user's touch gesture may include a tap, touch & hold, double tap, drag, pan, flick, drag and drop, swipe, and the like.

The notification module 1730 may generate a signal for notifying the occurrence of an event in the electronic device 1000 .

Referring to FIG. 13 , in operation 1310 , the electronic device 1000 determines the relative positional relationship between pixels included in the first image and pixels included in the second image based on the characteristics of each region of the first image. can decide The characteristic for each region of the first image according to an embodiment may include, for example, a frequency characteristic for each region. The relative positional relationship according to an embodiment may be determined based on a characteristic of each region of the first image. For example, the relative positional relationship may be obtained based on a result of performing a convolution operation between at least one pixel of the first image and the first kernel.

According to an embodiment, n pixels included in the first image with respect to the position of one pixel included in the second image are identified based on the characteristics (eg, frequency characteristics) of each region of the first image, A relative positional relationship may be determined. According to an embodiment, the n pixels may be used to obtain one pixel of the second image.

_{According to an embodiment, the electronic device 1000 determines k j} (k _j = an integer) by performing a discard operation on an index value indicated by one pixel of the second image, and features each region of the first image. Based on , n offset values corresponding to _{k j may be obtained.} According to an embodiment, the n offset values may be obtained based on a convolution operation between at least one pixel of the first image including the pixel of the first image corresponding to _{k j and the first kernel.}

Also, the electronic device 1000 may identify n pixels to obtain the one pixel based on the n offset values. For example, a value obtained by adding n corresponding offset values to n index values (eg, k _j -1, k _j , k _j +1, k _j _{+2) determined based on k j is used as an index.} , n pixels can be identified.

In operation 1320 , the electronic device 1000 may obtain position transformation feature information based on a relative positional relationship and values of pixels included in the first image. The location transformation characteristic information according to an embodiment may include sample values used to generate each pixel of the second image.

The sample value according to an embodiment may be determined according to the pixel value of the index value of the first image corresponding to each pixel of the second image according to the relative positional relationship determined in operation 1310 . According to an embodiment, the index value of the first image may be determined as a real value instead of an integer as an offset value is added. In this case, the sample value may be determined as the sum of pixel values of the first image to which a weight is applied according to a value below a decimal point of the index value.

In operation 1330 , the electronic device 1000 may perform a convolution operation between the location transformation feature information and the scaling kernel to generate a second image in which the first image is scaled.

According to an embodiment, as the relative positional relationship is determined based on the offset value determined according to the characteristic of each region of the first image, the scaling kernel also obtains a compensation weight value obtained based on the characteristic of each region of the first image. After being corrected according to , a convolution operation with the position transformation feature information may be performed.

The electronic device 1000 according to an embodiment may determine at least one compensation weight value to be applied to each weight value included in the scaling kernel based on a characteristic (eg, a frequency characteristic) of each region of the first image. . For example, the compensation weight value may be obtained according to a result of a convolution operation performed by the second kernel or the third kernel. In addition, the processor 1300 applies at least one compensation weight value to at least one weight value of the scaling kernel, and then, based on the scaling kernel to which the at least one compensation weight value is applied, the second image to which the first image is scaled. can create

The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory storage medium' is a tangible device and only means that it does not contain a signal (eg, electromagnetic wave). It does not distinguish the case where it is stored as For example, the 'non-transitory storage medium' may include a buffer in which data is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store (eg Play Store ^TM ) or on two user devices ( It can be distributed (eg downloaded or uploaded) directly, online between smartphones (eg: smartphones). In the case of online distribution, at least a portion of a computer program product (eg, a downloadable app) is stored at least in a machine-readable storage medium, such as a memory of a manufacturer's server, a server of an application store, or a relay server. It may be temporarily stored or temporarily created.

Also, in this specification, “unit” may be a hardware component such as a processor or circuit, and/or a software component executed by a hardware component such as a processor.

The description of the present invention described above is for illustration, and those of ordinary skill in the art to which the present invention pertains can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive. For example, each component described as a single type may be implemented in a distributed manner, and likewise components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the above detailed description, and all changes or modifications derived from the meaning and scope of the claims and their equivalent concepts should be interpreted as being included in the scope of the present invention. do.

Claims

A method of generating a second image by scaling a first image in an electronic device, the method comprising:

determining a relative positional relationship between pixels included in the first image and pixels included in a second image based on the characteristics of each region of the first image;

obtaining position transformation characteristic information based on the determined relative positional relationship and values of pixels included in the first image; and

and generating a second image in which the first image is scaled by performing a convolution operation between the position transformation feature information and a scaling kernel.
According to claim 1, wherein the characteristics of each area,

Including frequency characteristics for each region, the method.
The method of claim 1, wherein the relative positional relationship is

The method is determined based on a result of performing a convolution operation between at least one pixel of the first image and a first kernel.
The method of claim 3, wherein at least one compensation weight value to be applied to each weight value of the scaling kernel is

The method is determined based on a result of performing a convolution operation with at least one pixel of the first image and the first kernel and a result of performing a convolution operation with a third kernel.
According to claim 1,

determining at least one compensation weight value to be applied to each weight value included in the scaling kernel based on the characteristics of each region of the first image; and

applying the at least one compensation weight value to the at least one weight value of the scaling kernel;

The second image is generated based on the scaling kernel to which the at least one compensation weight value is applied.
6. The method of claim 5, wherein the at least one compensation weight value is

The method is determined based on a result of performing a convolution operation between at least one pixel of the first image and a second kernel.
The method of claim 1, wherein determining the relative positional relationship comprises:

determining the relative positional relationship of n pixels included in the first image with respect to the position of one pixel included in the second image based on the characteristics of each region of the first image,

wherein the n pixels are pixels used to obtain the one pixel value.
8. The method of claim 7,

Assuming that the first image and the second image have the same size, positions of pixels included in the first image are denoted by index i (i=integer), and positions of pixels included in the second image are denoted by index j (j = real number),

The step of determining the relative positional relationship is

k j (k j = integer) is determined by performing a discard operation on the index value indicated by the one pixel , and the first image corresponding to k j based on the characteristics of each region of the first image obtaining n offset values for pixels of ; and

based on the n offset values, identifying the n pixels.
An electronic device for generating a second image by scaling a first image, the electronic device comprising:

a memory for storing the first image; and

determining a relative positional relationship between pixels included in the first image and pixels included in a second image based on the characteristics of each region of the first image;

Based on the determined relative positional relationship and the values of pixels included in the first image, position transformation characteristic information is obtained,

and at least one processor configured to generate a scaled second image of the first image by performing a convolution operation between the position transformation feature information and a scaling kernel.
10. The method of claim 9, wherein the characteristics of each area,

An electronic device including frequency characteristics for each region.
10. The method of claim 9, wherein the relative positional relationship

The electronic device is determined based on a result of performing a convolution operation between at least one pixel of the first image and a first kernel.
The method of claim 11 , wherein at least one compensation weight value to be applied to each weight value of the scaling kernel is

The electronic device is determined based on a result of performing a convolution operation with at least one pixel of the first image and the first kernel and a result of performing a convolution operation with a third kernel.
10. The method of claim 9, wherein the at least one processor comprises:

determining at least one compensation weight value to be applied to each weight value included in the scaling kernel based on the characteristics of each region of the first image,

applying the at least one compensation weight value to the at least one weight value of the scaling kernel;

The second image is generated based on the scaling kernel to which the at least one compensation weight value is applied.
14. The method of claim 13, wherein the at least one compensation weight value is

The electronic device is determined based on a result of performing a convolution operation between at least one pixel of the first image and a second kernel.
10. The method of claim 9, wherein the at least one processor is

determining a relative positional relationship of n pixels included in the first image with respect to the position of one pixel included in the second image based on the characteristics of each region of the first image,

wherein the n pixels are pixels that are used to obtain the one pixel value.
The method of claim 15 , wherein, assuming that the first image and the second image have the same size, positions of pixels included in the first image are indicated by an index i (i=integer), and are included in the second image. The positions of pixels are denoted by index j (j = real number),

the at least one processor,

k j (k j = an integer) is determined by performing a discard operation on the index value indicated by the one pixel , and the first image corresponding to k j based on the characteristics of each region of the first image to obtain n offset values for the pixels of

identify the n pixels based on the n offset values.
A computer-readable recording medium in which a program for implementing the method of any one of claims 1 to 8 is recorded.