WO2024005590A1

WO2024005590A1 - Image scaling device and image scaling method

Info

Publication number: WO2024005590A1
Application number: PCT/KR2023/009230
Authority: WO
Inventors: 허훈; 박연숙
Original assignee: 주식회사 네패스
Priority date: 2022-07-01
Filing date: 2023-06-30
Publication date: 2024-01-04
Also published as: KR102611423B1

Abstract

An image scaling device, according to one embodiment of the present invention, comprises: two or more memories; and a processor which receives an input image, information related to the size of the input image, and information related to a conversion size, identifies a first conversion multiple corresponding to width and a second conversion multiple corresponding to height on the basis of the size of the input image and the conversion size, divides a plurality of pixels of the input image into a first plurality of groups on the basis of any one conversion multiple among the first conversion multiple and the second conversion multiple, and consecutively stores information related to the first plurality of groups in the two or more memories, groups the first plurality of groups into a second plurality of groups on the basis of the other conversion multiple among the first conversion multiple and the second conversion multiple, and then identifies the average values of the pixels included in the respective second plurality of groups, and thus generates a converted image.

Description

Image scaling device and image scaling method

The present invention relates to an image scaling device and an image scaling method.

With the development of artificial intelligence technology, AI learning or inference accelerators that learn images as input data are being actively developed. At this time, images may have different sizes, and the image resolution designed for each AI learning or inference accelerator is different, so it is necessary to convert the size of the input image to suit the design of the accelerator.

In addition, in the case of technology that uses images, there is always an issue of size conversion, so the development of technology for image scaling is required.

At this time, when image scaling is implemented through software programming, the processing speed is reduced and real-time processing is difficult, so it is necessary to design it based on hardware and perform real-time processing.

In addition, existing image scaling technology has limitations in conversion multiples such as 1/2 times, 1/3 times, and 1/4 times, so there are concerns about quality deterioration, such as arbitrarily cutting part of the image to convert the image to the desired size. also exists.

The purpose of the present invention is to provide an image scaling device and an image scaling method with a higher degree of freedom in conversion size.

The purpose of the present invention is to provide an image scaling device and an image scaling method that convert the size of an image to higher quality.

An image scaling device according to an embodiment of the present invention, comprising: two or more memories; Receive an input image, information about the size of the input image, and information about the transformation size, and based on the size of the input image and the transformation size, a first transformation multiplier corresponding to the width and a second transformation multiplier corresponding to the height Identifying and dividing the plurality of pixels of the input image into a first plurality of groups based on any one of the first transformation multiplier and the second transformation multiplier, and sequentially providing information about the first plurality of groups. is stored in the two or more memories, and the first plurality of groups are grouped into a second plurality of groups based on the remaining one of the first conversion multiplier and the second conversion multiple. It includes a processor that generates a converted image by identifying the average value of pixels included in each group.

The processor includes a division design block and a fixed-point design block, and the division design block divides the size of the input image and the transformation size and divides the quotient and the remainder into the first transformation multiplier and the second transformation multiplier. The conversion multiplier is identified as an integer value, and the fixed-point design block can identify decimal values of the first conversion multiple and the second conversion multiple using the remainder and the conversion size.

The fixed-point design block: (1) If the conversion size is less than twice the remainder, sets the most significant bit value of the decimal value to 1, and outputs a value obtained by subtracting the conversion size from two times the remainder. or, if the conversion size is not smaller than twice the remainder, the most significant bit value of the decimal value is set to 0, and a value obtained by doubling the remainder is output, and (2) the conversion size is equal to that of the previous step. If the output value is less than 2 times, the next bit value of the decimal value is set to 1, and a value obtained by subtracting the conversion size from 2 times the output value of the previous step is output, or the conversion size is equal to the output value of the previous step. If it is not less than twice, the next bit value of the decimal value is designated as 0, and a value twice the output value of the previous step is output. (3) When the decimal value has 8 binary bit values, You can repeat step (2) above until.

When the one transform multiple is the first transform multiple, the processor groups the pixels included in each line in the width direction of the input image according to the first transform multiple, and groups the pixels included in each line in the width direction of the input image according to the first transform multiple. When is the second transformation multiplier, the first plurality of groups can be generated by grouping pixels included in each line in the height direction of the input image according to the second transformation multiplier.

The processor is configured to perform a line in the width direction of the input image when the one transform multiple is the first transform multiple, and for each line in the height direction of the input image when the one transform multiple is the second transform multiple. Information about the first plurality of groups may be separately stored in the two or more memories.

The information about the first plurality of groups may include information about the sum of the number of pixels included in each group and the pixel values of the pixels included in each group.

The processor may group the first plurality of groups into the second plurality of groups while sequentially reading information about the first plurality of groups from the two or more memories based on the remaining conversion multiple.

The processor stores information about the second plurality of groups, including information on the sum of the number of pixels included in each group among the second plurality of groups and pixel values of pixels included in each group, in the two or more memories. You can save it.

The processor sequentially reads information about the first plurality of groups as many lines as the number of lines corresponding to the integer value of the remaining conversion multiple, and groups the first plurality of groups into the second plurality of groups, The sum of the decimal values of the remaining conversion multiples is calculated as the number of the second plurality of groups, and each time the sum of the decimal values exceeds 1, one more number than the number of lines is read to form the first plurality of groups. It can be grouped into the second plurality group.

The processor identifies pixel values of a plurality of pixels of the input image, identifies a plurality of intermediate pixel values between each pixel and at least one neighboring pixel, and based on the first transformation multiplier, the pixel value and Store first up-scaling information including the intermediate pixel value corresponding to the number of pixels to be added in the two or more memories, and store first up-scaling information including the intermediate pixel value corresponding to the number of pixels to be added, and corresponding to the number of pixels to be added based on the first up-scaling information and the second transform multiplier. The converted image may be generated by storing second up-scaling information including intermediate pixel values in the two or more memories, and sequentially reading the second up-scaling information stored in the two or more memories.

An image scaling method performed by an image scaling device according to an embodiment of the present invention, comprising: receiving an input image, information about the size of the input image, and information about the transformation size; identifying a first transform multiplier corresponding to the width and a second transform multiplier corresponding to the height based on the size of the input image and the transform size; Dividing a plurality of pixels of the input image into a first plurality of groups based on one of the first transformation multiplier and the second transformation multiplier and sequentially storing information about the first plurality of groups ; After grouping the first plurality of groups into a second plurality of groups based on the remaining one of the first transformation multiplier and the second transformation multiplier, the average value of the pixels included in each of the second plurality of groups is calculated. It includes the step of identifying and generating a converted image.

The step of identifying the first transform multiplier and the second transform multiplier includes dividing the quotient obtained by dividing the size of the input image and the transform size and dividing the quotient among the remainder into integers of the first transform multiplier and the second transform multiplier. identifying by value; It may include identifying decimal values of the first transform multiple and the second transform multiple using the remainder and the transform size.

The step of identifying the decimal value of the first transform multiple and the second transform multiple includes (1) when the transform size is less than twice the remainder, specifying the highest bit value of the decimal value as 1, and Output a value obtained by subtracting the conversion size from 2 times the remainder, or, if the conversion size is not smaller than 2 times the remainder, specify the most significant bit value of the decimal value as 0, and output a value obtained by subtracting the conversion size from 2 times the remainder. output step; (2) If the conversion size is less than twice the output value of the previous step, set the next bit value of the decimal value to 1, and output a value obtained by subtracting the conversion size from twice the output value of the previous step, Or, if the conversion size is not smaller than twice the output value of the previous step, designating the next bit value of the decimal value as 0 and outputting a value that is twice the output value of the previous step; (3) It may include repeating step (2) until the decimal value has 8 binary bit values.

The step of storing information about the first plurality of groups includes grouping pixels included in each line in the width direction of the input image according to the first transform multiplier when one of the transform multipliers is the first transform multiplier. (grouping), and when one of the transformation multipliers is the second transformation multiplier, the pixels included in each line in the height direction of the input image are grouped according to the second transformation multiplier to generate the first plurality of groups. May include steps.

The step of storing information about the first plurality of groups may include, if the one transform multiple is the first transform multiple, for each line in the width direction of the input image, the one transform multiple is the second transform multiple. If it is a multiple, the method may include storing information about the first plurality of groups for each line in the height direction of the input image.

The step of generating the conversion image includes sequentially reading information about the first plurality of groups stored based on the remaining conversion multiple and grouping the first plurality of groups into the second plurality of groups. It can be included.

The step of grouping the first plurality of groups into the second plurality of groups includes information on the sum of the number of pixels included in each group among the second plurality of groups and the pixel values of the pixels included in each group. It may include the step of storing information about the second plurality of groups.

The step of grouping the first plurality of groups includes sequentially reading information about the first plurality of groups as many lines as the number of lines corresponding to the integer value of the remaining conversion multiple, thereby dividing the first plurality of groups into the second plurality of groups. Grouping into a plurality of groups; calculating the sum of the decimal values of the remaining conversion multiples as many as the number of the second plurality of groups; The method may include grouping the first plurality of groups into the second plurality of groups by reading one more line than the number of lines whenever the sum of the decimal values exceeds 1.

identifying pixel values of a plurality of pixels of the input image; identifying a plurality of intermediate pixel values between each pixel and at least one neighboring pixel; storing first up-scaling information including the pixel value and the intermediate pixel value corresponding to the number of pixels to be added based on the first conversion multiplier; storing second up-scaling information including an intermediate pixel value corresponding to the number of pixels to be added based on the first up-scaling information and the second transform multiplier; It may include generating the converted image by sequentially reading the stored second up-scaling information.

According to an embodiment of the present invention, scaling can be performed faster and with fewer constraints on conversion multiples through a hardware configuration.

According to an embodiment of the present invention, the conversion multiplier can be freely configured, so the usability of the conversion image, such as for AI learning, can be increased. In particular, when used in an AI accelerator, learning performance can be improved.

1 is a diagram schematically showing an image scaling device according to an embodiment of the present invention.

Figure 2 is a block diagram showing the configuration of an image scaling device according to an embodiment of the present invention.

Figure 3 is a diagram illustrating an operation flowchart of an image scaling device according to an embodiment of the present invention.

FIG. 4 is a diagram illustrating an operation flowchart for identifying a conversion multiplier of an image scaling device according to an embodiment of the present invention.

Figure 5 is a diagram showing a downscaling operation of an image scaling device according to an embodiment of the present invention.

Figure 6 is a diagram showing the operation of an image scaling device according to an embodiment of the present invention.

Figure 7 is a diagram illustrating an upscaling operation of an image scaling device according to an embodiment of the present invention.

FIG. 8 is a diagram illustrating a lookup table for the number of pixels to be added according to the conversion multiplier during upscaling of the image scaling device according to an embodiment of the present invention.

Figure 9 is a diagram showing the operation of an image scaling device according to an embodiment of the present invention.

Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the attached drawings. The detailed description set forth below in conjunction with the accompanying drawings is intended to illustrate exemplary embodiments of the invention and is not intended to represent the only embodiments in which the invention may be practiced. In order to clearly explain the present invention in the drawings, parts unrelated to the description may be omitted, and the same reference numerals may be used for identical or similar components throughout the specification.

The image scaling device 100 (hereinafter referred to as the scaling device 100) shown in FIG. 1 is a device that converts the size of the input image 10 into a converted image 30 according to size information 20. As such, it can be implemented with a computer, server, etc.

According to an embodiment of the present invention, the input image 10 is not limited in format and may include dynamic images as well as static images. For dynamic images, size conversion may be applied on a frame-by-frame basis.

According to one embodiment of the present invention, the size information 20 may include information about the size of the input image 10 and information about the size to be converted (hereinafter referred to as conversion size).

The scaling device 100 may perform upscaling to enlarge the image 10 and downscaling to reduce the image 10 by comparing the transformation size with the size of the input image 10.

Information about the size of the input image 10 may include the width and height of the input image, and information about the converted size may include the width and height of the converted image. However, information about the size of the input image 10 can be identified from the input image 10 received by the scaling device 100.

According to one embodiment of the present invention, the converted image 30 is image data output by the scaling device 100.

In the case of conventional technology for converting the size of an input image, conversion was possible only for multiples that were set in the conversion program or multiples that were easy to convert in size, such as 1/2 times or 1/4 times.

However, when comparing the size of the input image 10 and the conversion size, in most cases, the multiple to be converted (hereinafter referred to as the conversion multiple) is not exactly the same.

The present invention proposes a scaling device 100 based on hardware design that has faster processing speed than software programming and has a high degree of freedom in conversion multiples.

Hereinafter, the configuration and operation of the scaling device 100 according to an embodiment of the present invention will be described in more detail with reference to the drawings.

The scaling device 100 according to an embodiment of the present invention includes an input unit 110, a communication unit 120, a display unit 130, a memory 140, and a processor 150.

The input unit 110 generates input data in response to user input of the scaling device 100. For example, in the case of user input for performing image scaling, such as user input regarding the conversion size, it can be applied without limitation.

The input unit 110 includes at least one input means. The input unit 110 includes a keyboard, key pad, dome switch, touch panel, touch key, mouse, menu button, etc. may include.

The communication unit 120 receives the input image, information about the size of the input image, and information about the conversion size, and performs communication with external devices such as an AI learner and a server to transmit the converted image. To this end, the communication unit 120 may perform communications such as 5th generation communication (5G), long term evolution-advanced (LTE-A), long term evolution (LTE), and wireless fidelity (Wi-Fi).

Meanwhile, the communication unit 120 includes a connector or port to which a cable capable of receiving the input image, information about the size of the input image, and information about the conversion size can be connected. Connectors or ports may include USB ports, HDMI ports, DisplayPort, DVI ports, SCART, etc. The communication unit 120 may include a connector or port according to a network transmission standard such as Ethernet. For example, the communication unit 120 may be implemented as a LAN card wired to a router or gateway.

The display unit 130 displays display data according to the operation of the scaling device 100. For example, the display unit 130 may display a screen including an input image, a screen for inputting a conversion size, a screen including a generated converted image, etc.

The display unit 130 includes a liquid crystal display (LCD), a light emitting diode (LED) display, an organic light emitting diode (OLED) display, and a micro electro mechanical systems (MEMS) display. and electronic paper displays. The display unit 130 may be combined with the input unit 110 and implemented as a touch screen.

The memory 140 stores operation programs of the scaling device 100. The memory 140 is a non-volatile storage that can preserve data (information) regardless of whether power is provided, and data to be processed by the processor 150 is loaded. If power is not provided, the data is stored. Includes memory with volatile properties that cannot be preserved. Storage includes flash memory, HDD (hard-disc drive), SSD (solid-state drive), ROM (Read Only Memory), etc., and memory includes buffer and RAM (Random Access Memory). etc.

According to one embodiment of the present invention, it may include two or more memories. At this time, two or more memories store information for each line in the width direction or height direction of the image, but if the number of lines is greater than the number of memories, the memory can be used like a ring buffer to sequentially store information.

When a plurality of pixels of an input image are divided into a plurality of groups according to one of the conversion multipliers corresponding to width or height, the memory 140 stores information about the plurality of groups or is necessary in the process of performing image scaling. Calculation programs, etc. can be stored.

The processor 150 may control at least one other component (eg, hardware or software component) of the scaling device 100 by executing software such as a program, and may perform various data processing or calculations.

The processor 150 may include a division design block 151 and a fixed point design block 152. At this time, the division design block 151 and the fixed point design block 152 may be configured as circuits electrically connected to each other.

According to one embodiment of the present invention, the conversion multiple can be identified using the division design block 151 and the fixed point design block 152. The transformation multiplier according to an embodiment of the present invention is a transformation multiplier corresponding to the width of the input image (hereinafter referred to as the first transformation multiplier) and a transformation multiplier corresponding to the height of the input image (hereinafter referred to as the second transformation multiplier). .) includes. At this time, the conversion multiple consists of an integer value and a decimal value.

The division design block 151 divides the size and transformation size of the input image 10 to obtain a quotient and a remainder, and identifies the obtained quotient as an integer value of the transformation multiple.

At this time, the decimal value may appear as a finite decimal number, but it may also appear as an infinite decimal number.

For example, if the width of the input image is 10 pixels and the width of the conversion size is 4 pixels, the integer value of the conversion multiplier is 2, the decimal value is 0.5, and the conversion multiple is 1/2.5 times.

On the other hand, if the width of the input image is 13 pixels and the width of the conversion size is 7 pixels, the integer value of the conversion multiplier is 1, and the decimal value is 0.85714285... which can be infinite. Additionally, in the present invention, decimal values must be expressed as binary numbers in a circuit.

Accordingly, the scaling device 100 uses a fixed-point design block 152 designed to express the decimal value of the conversion multiple in binary bit units.

The fixed point design block 152 may identify decimal values of the first and second conversion multiples using the remainder and conversion size identified by the division design block 151.

The processor 150 receives an input image, information about the size of the input image, and information about the transformation size, and based on the size of the input image and the transformation size, a first transformation multiple corresponding to the width and a second transformation multiple corresponding to the height. Identify the transformation multiplier, divide the plurality of pixels of the input image into first plurality groups based on either the first transformation multiplier or the second transformation multiplier, and sequentially store information about the first plurality groups in two or more memories. After storing the first plurality of groups into the second plurality of groups based on the remaining one of the first transformation multiplier and the second transformation multiplier, the average value of the pixels included in each of the second plurality of groups is identified to create a converted image. creates .

According to one embodiment of the present invention, the processor 150 receives the input image 10, information about the size of the input image 10, and information about the conversion size (S10).

As described above, the input image 10 can be received from the outside, but is not limited to this and can be pre-stored in the memory 140 and used. The processor 150 may identify information about the size of the input image 10 from the input image 10, and receive user input about the size or conversion size of the input image 10 through the input unit 110. , information about the conversion size can be received from the outside. As such, there are various reception paths for the input image 10, information about the size of the input image 10, and information about the conversion size, and the present invention is not limited by the reception path or method.

According to one embodiment of the present invention, the processor 150 identifies a first transform multiplier corresponding to the width and a second transform multiplier corresponding to the height based on the size and transform size of the input image (S20).

According to one embodiment of the present invention, the processor 150 includes a division design block 151 and a fixed point design block 152, and conversion is performed using the division design block 151 and the fixed point design block 152. Multiples can be identified.

More specifically, the division design block 151 may obtain a quotient and a remainder by dividing the width of the input image 10 and the width of the transformation size, and identify the quotient as an integer value of the first transformation multiple. Likewise, the division design block 151 may obtain a quotient and a remainder by dividing the height of the input image 10 and the height of the transformation size, and identify the quotient as an integer value of the second transformation multiple.

The fixed point design block 152 uses the remainder and the transform size identified by the division design block 151 to identify the decimal value of the transform multiple. At this time, the fixed point design block 152 may receive the remaining information from the division design block 151.

The specific operation process of the fixed-point design block 152 will be described with reference to FIG. 4.

According to an embodiment of the present invention, the processor 150 divides a plurality of pixels of the input image into a first plurality of groups based on either a first transform multiplier or a second transform multiplier and sequentially divides the plurality of pixels into the first plurality groups. Information about is stored in two or more memories (S30).

The scaling process consists of reduction/enlargement in the width and height directions, respectively. At this time, the scaling order in the width and height directions does not affect the final result.

For example, if the first conversion multiple is 1/2.5 and the second conversion multiple is 1/4.5, the converted image obtained by first reducing it by 1/2.5 times in the width direction and then reducing it by 1/4.5 times in the height direction, and The converted image obtained by reducing the image by 1/4.5 times in the height direction and then reducing it by 1/2.5 times in the width direction is the same.

Accordingly, the processor 150 may divide a plurality of pixels of the input image into a first plurality of groups according to a conversion multiplier in the width direction or the height direction, whichever scales first.

When one of the transformation multipliers is the first transformation multiplier, that is, when scaling is performed in the width direction first, the processor 150 selects the pixels included for each line in the width direction among the plurality of pixels forming the input image to the first transformation multiplier. A first plurality of groups can be created by grouping accordingly.

On the contrary, when one transformation multiplier is a second transformation multiplier, that is, when scaling is performed in the height direction first, the processor 150 divides the pixels included for each line in the height direction among the plurality of pixels constituting the input image into a second transformation multiplier. A first plurality of groups can be created by grouping according to the conversion multiple. A specific example of performing grouping will be described with reference to FIG. 5.

Additionally, the processor 150 may sequentially store information about the first plurality of groups in two or more memories. At this time, the information about the first plurality of groups includes information about the sum of the number of pixels included in each group and the pixel values of the pixels included in each group. The process of storing in memory will be described in detail with reference to FIG. 6.

According to an embodiment of the present invention, the processor 150 groups the first plurality of groups into a second plurality of groups based on the remaining one of the first transformation multiplier and the second transformation multiplier, and then groups each of the second plurality groups. The converted image 30 is generated by identifying the average value of the pixels included in (S40).

According to an embodiment of the present invention, the processor 150 sequentially reads information about the first plurality of groups as many lines as the number of lines corresponding to the integer value of the remaining conversion multiple, and converts the first plurality of groups into the second plurality of groups. Can be grouped into groups.

For example, when the first plurality of groups are created by the first transformation multiplier, the processor 150 sequentially reads information about the first plurality of groups as many lines as the number of lines corresponding to the integer value of the second transformation multiplier. .

At this time, the processor 150 generates a second plurality of groups by reading one more line than the number of lines corresponding to the integer value each time the sum of the decimal values of the conversion multiples exceeds 1 while reading.

The converted image 30 is ultimately created by continuously performing the process of reducing a plurality of pixels to one pixel in the case of downscaling, and the processor 150 determines the average value of the pixels included in each group for each of the second plurality of groups. The converted image 30 can be generated by identifying .

FIG. 4 shows an operational flow diagram of the fixed point design block 152 identifying a conversion multiple in relation to S20 of FIG. 3. Hereinafter, for convenience of explanation, it is assumed that the input image 10 has a size of 9 pixels * 13 pixels, and the conversion size has a size of 2 pixels * 7 pixels, and the first image corresponding to the height of the input image 10 is assumed to be 2 pixels * 7 pixels. 2Describes the process of calculating the decimal value of a conversion multiple. At this time, the decimal value is identified in bit units as described above.

According to one embodiment of the present invention, the fixed-point design block 152 receives information about the remainder from the division design block 151 and receives information about the transform size (S21). At this time, when 13 is divided by 7, the quotient is 1 and the remainder is 6. The corresponding transformation size is 7.

The fixed point design block 152 identifies whether the transform size is less than twice the remainder (S22). 7 is less than 12, which is twice 6 (Yes in S22).

If the conversion size is less than 2 times the remainder (Yes in S22), the fixed point design block 152 specifies the highest bit value of the decimal value as 1 and outputs (remainder * 2 - conversion size). (S23). According to one embodiment of the present invention, decimal values will be represented by 8 bits. At this time, the highest bit value means the first bit value among 8 bits.

If the conversion size is not less than twice the remainder (No in S22), the fixed-point design block 152 sets the highest bit value of the decimal value to 0 and outputs the remainder doubled (S24).

At this time, 7 is less than 12, which is twice 6 (Yes in S22), so the highest bit value is 1 (1XXX_XXXX), and the output value is (6*2-7)=5 (S23).

The fixed point design block 152 identifies whether the transform size is less than twice the previous output value (S25). Hereinafter, the previous output value refers to the value output by the fixed-point design block 152 before each step.

If the conversion size is less than twice the previous output value (Yes in S22), the fixed point design block 152 sets the next bit value of the decimal value to 1 and outputs (previous output value * 2 - conversion size) ( S26).

If the conversion size is not smaller than twice the previous output value (No in S22), the fixed point design block 152 sets the next bit value of the decimal value to 0 and outputs the previous output value by doubling it (S27).

In this case, the previous output value is 5, and 7 is less than 10, which is twice 5 (Yes in S25), so the next bit value is 1 (11XX_XXXX), and the output value is (5*2-7)=3 (S26) .

The fixed-point design block 152 checks whether the bit value for the decimal value has been completed with 8 bits (S28), and if not yet completed (No in S28), returns to step S25 to identify the next bit value.

When completed (Yes in S28), an 8-bit decimal value is output (S29).

Since the example is not yet 8 bits (11XX_XXXX) (No in S28), return to step S25.

In S25, 7 is greater than 6, which is twice 3 (No in S25), so the next bit value is 0 (110X_XXXX), and the output value is (3*2)=6 (S27).

Since it is not 8 bits (No in S28), return to step S25.

In S25, 7 is less than 12, which is twice 6 (Yes in S25), so the next bit value is 1 (1101_XXXX), and the output value is (6*2-7)=5 (S26).

Since it is not 8 bits (No in S28), return to step S25.

In S25, 7 is less than 10, which is twice 5 (Yes in S25), so the next bit value is 1 (1101_1XXX), and the output value is (5*2-7)=3 (S26).

Since it is not 8 bits (No in S28), return to step S25.

In S25, 7 is greater than 6, which is twice 3 (No in S25), so the next bit value is 0 (1101_10XX), and the output value is (3*2)=6 (S27).

Since it is not 8 bits (No in S28), return to step S25.

In S25, 7 is less than 12, which is twice 6 (Yes in S25), so the next bit value is 1 (1101_101X), and the output value is (6*2-7)=5 (S26).

Since it is not 8 bits (No in S28), return to step S25.

In S25, 7 is less than 10, which is twice 5 (Yes in S25), so the next bit value is 1 (1101_1011), and the output value is (5*2-7)=3 (S26).

Since 8 bits of the fixed point design block 152 are completed as 1101_1011 (Yes in S28), the 8 bits are output as a decimal value (S29).

That is, when the input image 10 is converted from a height of 13 pixels to a height of 7 pixels, the integer value of the second conversion multiple is 1, and the decimal value can be expressed as 0.11011011(2) in binary.

Converting 0.11011011(2) to decimal value, it is 0.85546875(10). At this time, if you divide 13 by 7, you get 0.85714285...(10) Invar, the error is very small.

FIG. 5 provides a detailed explanation of dividing a plurality of pixels of an input image into a first plurality of groups and a second plurality of groups with reference to S30 and S40 of FIG. 3 .

At this time, the example assumes that, as in FIG. 4, the input image 10 has a size of 9 pixels*13 pixels, the transformation size has a size of 2 pixels*7 pixels, and scaling is performed first based on the first transformation multiple. do.

As previously identified in FIG. 4, the first transformation multiple in the width direction is 1/4.5.

The processor 150 may divide the plurality of pixels of the input image 10 into a first plurality of groups based on the first transformation multiplier.

The processor 150 may generate a first plurality of groups by grouping pixels included in each line in the width direction among the plurality of pixels forming the input image 10 according to the first transformation multiplier.

Referring to FIG. 5, the processor 150 groups the 9 pixels included in line 1 of the lines in the width direction of the input image 10 into one group 11 with the integer value of the first transform multiple being 4. , the remaining 5 pixels can be grouped into another group (12).

At this time, since the integer 1 is obtained by combining the decimal value 0.5 of the first conversion multiple of group 11 and the decimal value 0.5 of the first conversion multiple of group 12, group 12 has the integer value 4 of the first conversion multiple. It consists of 5 pixels, including one more pixel.

Since this is performed equally for all lines, it consists of 26 groups of 4 pixels and 5 pixels for each line from line 1 to line 13 in the width direction. These are called the first plural group.

The processor 150 stores information about the first plurality of groups in the memory 140, including information on the sum of the number of pixels included in each group and the pixel values of the pixels included in each group.

According to one embodiment of the present invention, the pixel value may include brightness information, color information, etc. of the corresponding pixel. For example, if the input image has a gray scale, the pixel value can range between 0 and 255 by expressing 1 pixel in 8 bits. Black has a pixel value of 0 and white has a pixel value of 255.

If the input image is color, the pixel value can range between 0 and 255 for each color component by expressing 1 pixel with 8 bits for each color component of R (red), G (green), and B (blue). there is. For example, black has a pixel value of (0, 0, 0), and white has a pixel value of (255, 255, 255). For example, assuming that the pixel value of all pixels is 10, in the case of group 11, information on 4 pixels and the total of 40 pixel values is stored in the memory.

If the processor 150 immediately performs scaling, the image 510 can be generated reduced in the width direction. At this time, group 11 can be converted into pixel 511 and group 12 can be converted into pixel 512. However, the present invention does not directly scale, but sequentially stores information about the first plurality of groups in the memory 140.

Next, the processor 150 sequentially reads information about the first plurality of groups as many lines as the number of lines corresponding to the integer value of the remaining second conversion multiple and groups the first plurality of groups into the second plurality of groups. You can.

As identified in Figure 4, the second transformation multiplier in the height direction is 1/1.85546875.

The processor 150 sequentially reads information about the first plurality of groups as many lines as the number of lines corresponding to the integer value of the second conversion multiple. At this time, the processor 150 generates a second plurality of groups by reading one more line than the number of lines corresponding to the integer value each time the sum of the decimal values of the conversion multiples exceeds 1 while reading.

Referring to FIG. 5, since the integer value of the second conversion multiple is 1, the processor 150 can read group 11 and group it into group 31.

Next, the sum of the decimal value 0.85546875 of group (11) and the decimal value of group (13) 0.85546875 exceeds 1 (1.7109375), so read the two groups (13) and group (14) with one more than 1 of the integer value. It can be grouped into groups (32).

Next, adding the decimal value 0.85546875 of group (14) to the previous decimal value of 1.7109375 exceeds 1 (2.56640625), so two groups (15) and group (16) with one added more than 1 of the integer value are read to obtain group (33). ) can be tied.

If grouped in this way, the 26 first plural groups can be grouped into 14 second plural groups.

The processor 150 may generate the converted image 30 by identifying the average value of pixels included in each group for each of the second plurality of groups.

For example, group 31 has 4 pixels and 40 pixel values, and has a pixel value of 10, which is the average value of the pixels. Group 32 has 8 pixels and 80 pixel values, and has a pixel value of 10, which is the average value of the pixels. Group 33 has 8 pixels and 80 pixel values, and has a pixel value of 10, which is the average value of the pixels.

FIG. 6 shows the scaling process described with reference to FIG. 5 being performed using memory.

According to one embodiment of the present invention, the scaling device 100 has two or more memories 140. Referring to FIG. 6, the memory 140 consists of 17 memories. At this time, the number and structure of memories are not limited by this drawing.

Continuing the example of FIG. 5, the processor 150 groups the pixels included in each line in the width direction among the plurality of pixels forming the input image 10 according to the first transformation multiplier to form a first plurality of groups. can be created.

The processor 150 groups 9 pixels included in each line among the lines in the width direction of the input image 10 into one group with an integer value of the first transformation multiple of 4, and groups the remaining 5 pixels into another group. Can be grouped.

At this time, the processor 150 may sequentially store information about the first plurality of groups in two or more memories 140 for each line in the width direction of the input image 10.

Accordingly, the processor 150 can store information about the two groups, group 11, and group 12 constituting line 1 in memory 1, and sequentially store information about the groups for each line. There are 17 memories shown in FIG. 6, but they are composed of 13 lines, so information is stored up to memory 13. If the number of lines is greater than the number of memories, the memory can be used like a ring buffer and stored in the first memory order again.

When the information about the first plurality of groups is sequentially stored in the memory 140, the processor 150 sequentially reads the information about the first plurality of groups from the memory 140 based on the second conversion multiplier and The first plurality of groups can be grouped into the second plurality of groups.

Therefore, as explained in relation to FIG. 5, since the second conversion multiplier is 1/1.85546875, the information in memory 1 is read as is, and memory 2 and memory 3, and memory 4 and memory 5 are read based on the second conversion multiple. , memory 6 and memory 7, memory 8 and memory 9, memory 10 and memory 11, and memory 12 and memory 13 can be read together.

At this time, the processor 150 can immediately output the pixel value of the converted image 30 by reading information from the memories together and performing calculations immediately.

FIG. 7 is a diagram illustrating an upscaling operation of an image scaling device according to an embodiment of the present invention.

According to one embodiment of the present invention, the processor 150 may identify pixel values of a plurality of pixels of the input image 10 and identify a plurality of intermediate pixel values between each pixel and at least one neighboring pixel. .

For example, consider the case where the pixel values of a plurality of pixels of the input image 10 are 10, 20, and 30, respectively. A plurality of intermediate pixel values between a pixel with a pixel value of 10 and a neighboring pixel with a pixel value of 20 are obtained. At this time, the plurality of intermediate pixel values may be an average value of pixel values and may include an intermediate value between any one pixel value and the average value. That is, the middle pixel value between 10 and 20 may be 15, which is the average value, 12, which is the middle value between 10 and 15, and 17, which is the middle value between 15 and 20.

Accordingly, the plurality of intermediate pixel values between 10 and 20 may be 12, 15, and 17. At this time, the calculation can be done more precisely depending on the number of pixels to be added.

The processor 150 may store upscaling information including pixel values and intermediate pixel values for each number of pixels to be added in two or more memories 140 .

For example, if you want to add 1 pixel, upscaling information including 10 and 15 can be stored, and if you want to add 2 pixels, upscaling information including 10, 12, and 17 can be stored. At this time, the processor 150 may list and store the number of pixels and the corresponding pixel values.

At this time, as previously described in relation to downscaling, the memory 140 can be used as a ring buffer depending on the number of lines, and can also be used as a plurality of ring buffers divided by a certain number of memories.

According to one embodiment of the present invention, upscaling, like downscaling, identifies a conversion multiplier.

According to one embodiment of the present invention, the processor 150 may store first up-scaling information including a pixel value based on the first transform multiplier and an intermediate pixel value corresponding to the number of pixels to be added in two or more memories. .

Referring to FIG. 7, it is assumed that the input image 710 has a size of 3 pixels*2 pixels, the transformation size has a size of 9 pixels*6 pixels, and is first upscaled based on the first transformation multiplier.

According to one embodiment of the present invention, the processor 150 uses the division design block 151 and the fixed point design block 152 as described above to achieve a first conversion multiple of 3 and a second conversion multiple of 3. A breach of trust can be identified.

According to one embodiment of the present invention, the processor 150 identifies first up-scaling information including a pixel value and an intermediate pixel value corresponding to the number of pixels to be added based on the first transform multiplier.

For example, in line 1 of the input image 710, there are pixels with

pixel values

10, 20, and 30, and in order to upscale by 3 times in the width direction, there are 4 intermediate pixel values between 10 and 20 pixels, and 20 and 30. Between 30 pixels, two intermediate pixel values are needed. At this time, a method of identifying the number of pixels to be added will be described with reference to FIG. 8.

According to one embodiment of the present invention, the processor 150 outputs four intermediate pixel values 12, 15, 17, and 18 between

pixels

10 and 20 in line 1, and two intermediate pixel values 23 between

pixels

20 and 30. Identify 27, and in line 2 identify the four intermediate pixel values 22, 25, 27, and 28 between

pixels

20 and 30 and the two intermediate pixel values 32 and 37 between

pixels

30 and 40.

The processor 150 identifies first up-scaling information including the identified pixel values. If the processor 150 immediately performs scaling, an image 720 enlarged in the width direction can be generated. However, in the present invention, scaling is not performed immediately, and the first up-scaling information is sequentially stored in the memory 140.

Thereafter, the processor 150 identifies second up-scaling information including an intermediate pixel value corresponding to the number of pixels to be added based on the first up-scaling information and the second transform multiplier.

For example, line 1 of the middle image 720 has pixels with

pixel values

10, 12, 15, 17, 18, 20, 23, 27, and 30, and line 2 has pixel values 20, 22, 25, and 27. There are pixels numbered , 28, 30, 32, 37, and 40.

In order to upscale by 3 times in the height direction, 4 intermediate pixel values in the height direction are required between line 1 and line 2.

According to one embodiment of the present invention, the processor 150 identifies second up-scaling information including intermediate pixel values of 4 lines between line 1 and line 2, and sequentially reads the second up-scaling information to create a converted image. Generates (730).

The lookup table 800 shown in FIG. 8 indicates the number of additional pixels according to the conversion multiplier.

According to one embodiment of the present invention, the processor 150 may identify the number of pixels to be added according to the first transform multiplier and the second transform multiplier based on the lookup table 800.

For example, the first transform multiple of the input image 710 of FIG. 7 is 3 times and the second transform multiple is 3 times. Referring to the lookup table 800, when the conversion multiple is 3, the number of additional pixels between initial pixels is 4, and the number of additional pixels between subsequent pixels is 2.

Therefore, when upscaling the input image 710 in the width direction, 4 pixels are additionally required between

lines

10 and 20, and 2 pixels are additionally required between 20 and 30.

As another example, assume that the input image has a size of 4 pixels x 4 pixels, the conversion size has a size of 11 pixels x 6 pixels, the first conversion multiple is 2.75 times, and the second conversion multiple is 1.5 times.

Referring to the lookup table 800, when the conversion multiplier is 2.75 times, the number of additional pixels between initial pixels is 3, and the number of additional pixels between subsequent pixels is 1 or 2.

Therefore, the number of additional pixels between initial pixels is 3, and the number of additional pixels thereafter is set to 1 if the sum of the decimal values of the conversion multiples does not exceed 1, and 2 if the sum of the decimal values of the conversion multiples exceeds 1.

Therefore, among the 4 pixels in the width direction, 3 additional pixels are required between the first and second pixels, 2 additional pixels are required between the 2nd and 3rd pixels, and 2 additional pixels are required between the 3rd and 4th pixels.

FIG. 9 shows the scaling process described with reference to FIG. 7 being performed using memory.

According to one embodiment of the present invention, the scaling device 100 has two or more memories 140. Referring to FIG. 9, the memory 140 consists of 17 memories. At this time, the number and structure of memories are not limited by this drawing.

Continuing the example of FIG. 7, the processor 150 generates a first pixel value including a pixel value of a plurality of pixels constituting the input image 710 and an intermediate pixel value corresponding to the number of pixels to be added based on the first transform multiplier. Upscaling information can be stored in two or more memories 140.

Referring to FIG. 9 , the processor 150 sequentially stores first up-scaling information in the memory 140 for each line of the input image 710 of FIG. 7 .

The processor 150 stores the first upscaling information {(5, 10, 12, 15, 17, 18), (3, 20, 23, 27), (1, 30)} of line 1 in memory 1, and , the first upscaling information {(5,20,22,25,27,28), (3,30,33,37), (1,40)} of line 2 is stored in memory 2.

As described above, there are 17 memories shown in FIG. 9, but they are composed of two lines, and the first up-scaling information is stored up to memory 2. If the number of lines is greater than the number of memories, the memory can be used like a ring buffer and stored in the first memory order again.

When the first up-scaling information is sequentially stored in the memory 140, the processor 150 generates a second up-scaling information including an intermediate pixel value corresponding to the number of pixels to be added based on the first up-scaling information and the second transform multiplier. Scaling information can be stored in two or more memories 140.

The processor 150 may store second up-scaling information corresponding to each of the six lines increased according to the number of pixels added in the height direction in memories 3 to 8.

The processor 150 may sequentially read the second up-scaling information stored in memories 3 to 8 to generate the converted image 730. At this time, by storing the second up-scaling information for each line in each memory, information omission that may occur when generating a converted image due to communication failure, etc. can be prevented.

[National research and development project that supported this invention]

[Assignment number] 1711152982

[Assignment number] 2021-0-00131-002

[Ministry Name] Ministry of Science and ICT

[Project management (professional) organization name] Information and Communications Planning and Evaluation Institute

[Research project name] Artificial intelligence semiconductor application technology development (R&D)

[Research project name] Development of intelligent edge computing semiconductors for lightweight manufacturing inspection equipment

[Contribution rate] 1/1

[Name of project carrying out organization] Nepes Co., Ltd.

[Research period] 2022.01.01 ~ 2022.12.31

Claims

In the image scaling device,

2 or more memory;

Receiving an input image, information about the size of the input image, and information about the transformation size,

Identifying a first transform multiple corresponding to the width and a second transform multiple corresponding to the height based on the size of the input image and the transform size,

Based on any one of the first transform multiplier and the second transform multiple, a plurality of pixels of the input image are divided into a first plurality of groups and information about the first plurality of groups is sequentially provided to the two or more groups. save it in memory,

After grouping the first plurality of groups into a second plurality of groups based on the remaining one of the first transformation multiplier and the second transformation multiplier, the average value of the pixels included in each of the second plurality of groups is calculated. An image scaling device including a processor that identifies and generates a transformed image.
According to paragraph 1,

The processor includes a division design block and a fixed point design block,

The division design block is,

Identifying the quotient among the quotient and the remainder obtained by dividing the size of the input image and the transformation size as an integer value of the first transformation multiplier and the second transformation multiplier,

The fixed point design block is,

An image scaling device that identifies decimal values of the first transform multiple and the second transform multiple using the remainder and the transform size.
According to paragraph 2,

The fixed point design block is,

(1) If the conversion size is less than 2 times the remainder, set the most significant bit value of the decimal value to 1, and output a value obtained by subtracting the conversion size from 2 times the remainder,

Or, if the conversion size is not less than twice the remainder, the most significant bit value of the decimal value is set to 0, and a value obtained by doubling the remainder is output,

(2) If the conversion size is less than twice the output value of the previous step, set the next bit value of the decimal value to 1, and output a value obtained by subtracting the conversion size from twice the output value of the previous step,

Or, if the conversion size is not smaller than twice the output value of the previous step, designate the next bit value of the decimal value as 0 and output a value that is twice the output value of the previous step,

(3) An image scaling device that repeats step (2) until the decimal value has 8 binary bit values.
According to paragraph 1,

The processor,

When the one transform multiple is the first transform multiple, the pixels included in each line in the width direction of the input image are grouped according to the first transform multiple, and the one transform multiple is the second transform multiple. In the case of a transformation multiplier, an image scaling device for generating the first plurality of groups by grouping pixels included in each line in the height direction of the input image according to the second transformation multiplier.
According to clause 4,

The processor,

When the one transform multiple is the first transform multiple, the first transform multiple is applied for each line in the width direction of the input image, and for each line in the height direction of the input image when the one transform multiple is the second transform multiple. An image scaling device that stores information about a plurality of groups in the two or more memories.
According to clause 5,

The information about the first plurality of groups includes information about the sum of the number of pixels included in each group and the pixel values of the pixels included in each group.
According to paragraph 1,

The processor,

An image scaling device that groups the first plurality of groups into the second plurality of groups while sequentially reading information about the first plurality of groups from the two or more memories based on the remaining conversion multiple.
In clause 7,

The processor,

Image scaling for storing information about the second plurality of groups in the two or more memories, including information on the sum of the number of pixels included in each group among the second plurality of groups and the pixel values of the pixels included in each group. Device.
According to paragraph 4 or 7,

The processor,

Grouping the first plurality of groups into the second plurality of groups while sequentially reading information on the first plurality of groups as many lines as the number of lines corresponding to the integer value of the remaining conversion multiple,

Calculate the sum of the decimal values of the remaining conversion multiples as many as the number of the second plurality groups,

An image scaling device that groups the first plurality of groups into the second plurality of groups by reading one more line than the number of lines each time the sum of the decimal values exceeds 1.
According to paragraph 1,

The processor,

Identifying pixel values of a plurality of pixels of the input image,

Identifying a plurality of intermediate pixel values between each pixel and at least one neighboring pixel,

Store first up-scaling information including the pixel value and the intermediate pixel value corresponding to the number of pixels to be added based on the first conversion multiplier in the two or more memories,

Store second up-scaling information including an intermediate pixel value corresponding to the number of pixels to be added based on the first up-scaling information and the second transform multiplier in the two or more memories,

A scaling device that generates the converted image by sequentially reading the second up-scaling information stored in the two or more memories.
In an image scaling method performed by an image scaling device,

Receiving an input image, information about the size of the input image, and information about the conversion size;

Identifying a first transform multiplier corresponding to the width and a second transform multiplier corresponding to the height based on the size of the input image and the transform size;

Dividing a plurality of pixels of the input image into a first plurality of groups based on one of the first transformation multiplier and the second transformation multiplier and sequentially storing information about the first plurality of groups ;

After grouping the first plurality of groups into a second plurality of groups based on the remaining one of the first transformation multiplier and the second transformation multiplier, the average value of the pixels included in each of the second plurality of groups is calculated. An image scaling method comprising identifying and generating a transformed image.
According to clause 11,

The step of identifying the first conversion multiple and the second conversion multiple is,

identifying the quotient among the quotient and the remainder obtained by dividing the size of the input image and the transform size as integer values of the first transform multiplier and the second transform multiplier;

An image scaling method comprising identifying decimal values of the first transform multiple and the second transform multiple using the remainder and the transform size.
According to clause 12,

The step of identifying decimal values of the first conversion multiple and the second conversion multiple is,

(1) If the conversion size is less than 2 times the remainder, set the most significant bit value of the decimal value to 1, and output a value obtained by subtracting the conversion size from 2 times the remainder,

or when the conversion size is not less than twice the remainder, setting the most significant bit value of the decimal value to 0 and outputting a value obtained by doubling the remainder;

(2) If the conversion size is less than twice the output value of the previous step, set the next bit value of the decimal value to 1, and output a value obtained by subtracting the conversion size from twice the output value of the previous step,

Or, if the conversion size is not smaller than twice the output value of the previous step, designating the next bit value of the decimal value as 0 and outputting a value that is twice the output value of the previous step;

(3) An image scaling method comprising repeating step (2) until the decimal value has 8 binary bit values.
According to clause 11,

The step of storing information about the first plurality of groups includes:

When the one transform multiple is the first transform multiple, the pixels included in each line in the width direction of the input image are grouped according to the first transform multiple, and the one transform multiple is the second transform multiple. In the case of a transformation multiplier, an image scaling method comprising generating the first plurality of groups by grouping pixels included in each line in the height direction of the input image according to the second transformation multiplier.
According to clause 14,

The step of storing information about the first plurality of groups includes:

When the one transform multiple is the first transform multiple, the first transform multiple is applied for each line in the width direction of the input image, and for each line in the height direction of the input image when the one transform multiple is the second transform multiple. An image scaling method comprising storing information about a plurality of groups.
According to clause 15,

The information about the first plurality of groups includes information about the sum of the number of pixels included in each group and the pixel values of the pixels included in each group.
According to clause 11,

The step of generating the converted image is,

An image scaling method comprising grouping the first plurality of groups into the second plurality of groups while sequentially reading information about the first plurality of groups stored based on the remaining conversion multiplier.
According to clause 17,

The step of grouping the first plurality of groups into the second plurality of groups includes:

An image scaling method comprising storing information about the second plurality of groups, including information on the sum of the number of pixels included in each group and the pixel values of the pixels included in each group. .
According to claim 14 or 18,

The step of grouping into the second plurality of groups is,

Grouping the first plurality of groups into the second plurality of groups by sequentially reading information about the first plurality of groups as many lines as the number of lines corresponding to the integer value of the remaining conversion multiple;

calculating the sum of decimal values of the remaining conversion multiples as many as the number of the second plurality of groups;

An image scaling method comprising grouping the first plurality of groups into the second plurality of groups by reading one more line than the number of lines each time the sum of the decimal values exceeds 1.
According to clause 11,

identifying pixel values of a plurality of pixels of the input image;

identifying a plurality of intermediate pixel values between each pixel and at least one neighboring pixel;

storing first up-scaling information including the pixel value and the intermediate pixel value corresponding to the number of pixels to be added based on the first conversion multiplier;

storing second up-scaling information including an intermediate pixel value corresponding to the number of pixels to be added based on the first up-scaling information and the second transform multiplier;

An image scaling method comprising generating the converted image by sequentially reading the stored second up-scaling information.