JP6681078B2 - Image processing apparatus, image processing method, and image processing program - Google Patents

Description

  The present invention relates to image processing technology.

  Interframe predictive coding is one of the important techniques for compressing moving images. In inter-frame predictive coding, an image of a target frame is predicted based on a frame (hereinafter referred to as “reference frame”) that has been coded at a time different from that of the frame to be coded (hereinafter referred to as “target frame”). Encode the difference between the actual image of the frame and the predicted image. The data amount of the frame can be reduced by encoding the difference between the target frame and the encoded frame.

  In inter-frame predictive coding, the following processes (1) to (3) are performed in block units obtained by dividing a frame into a plurality of blocks.

(1) A block (hereinafter, referred to as “reference block”) similar to a block in the target frame (hereinafter, referred to as “target block”) is searched for in the reference frame using an image matching technique.

(2) A difference image between the target block and the reference block obtained by the search is obtained.

(3) A motion vector indicating the motion from the reference block obtained by the search to the target block is obtained.

  When decoding a frame of a moving image compressed in this way, the image of the target block can be restored from the reference block, the difference image, and the motion vector.

  As described above, in the above (1), the image matching technique is used when searching for a reference block similar to the target block. Various methods such as feature-based matching and area-based matching are known as image matching techniques. The movement of an object in a moving image includes rotation as well as movement and expansion / contraction (scale change). In the above (1), it is required to calculate the relative angle between the target block and the reference block in order to determine that the blocks in which the rotating object exists in the moving image are similar to each other. Further, as described above, the movement from the reference block to the target block calculated in (3) above includes movement, expansion and contraction, and rotation. Therefore, it is necessary to calculate the relative angle between the target block and the reference block.

  Patent Document 1 discloses a technique for calculating a motion in moving image compression. The coordinates of the three points in the area of the previous frame are obtained using the three points in the known area of the current frame on the image plane and the three translation vectors searched for these three points. Further, the horizontal and vertical components of each vector are calculated from the position coordinates of the six points in the current frame region and the previous frame region, and the horizontal and vertical rotation angles are calculated from the x and y components of these vectors. Six affine transformation parameters including expansion / contraction ratio and translation amount are calculated.

  Further, Non-Patent Document 1 discloses a technique related to image scale change and rotation. A histogram concerning the gradient of pixels included in the image block is created, and the peak having the maximum value of the histogram is assigned as the orientation of the key point. Then, the image is rotated in the orientation direction of the key point.

JP, 9-98424, A

Gradient-based Feature Extraction-SIFT and HOG- Hiroyoshi Fujiyoshi IPSJ Research Report CVIM 160, pp. 211-224, September, 2007.

  In the technique disclosed in Patent Document 1 and the technique disclosed in Non-Patent Document 1, there is a problem that the process of obtaining the relative angle between the target block and the reference block involves complicated calculation and a large processing load.

  An object of the present invention is to provide a technique for calculating a relative angle between regions of an image with a small amount of calculation.

An image processing apparatus according to one aspect of the present invention calculates a coefficient of the approximation function such that the brightness value of a pixel included in the first area is approximated by a curved surface of a predetermined approximation function, and sets the coefficient to the first value. A determining unit that calculates a coefficient of the approximation function so that the brightness value of a pixel included in the second region is approximated by the approximation function, and that determines the coefficient as a second coefficient; and the first coefficient. A first rotation angle indicating the direction of the approximation function is obtained using the above, a second rotation angle indicating the direction of the approximation function is obtained using the second coefficient, and the first rotation angle and the second rotation angle are obtained. A calculation unit that calculates a relative angle with
Have.

  According to the present invention, each of the regions for which the relative angle is calculated is approximated by a curved surface, and the relative angles in the directions of the two approximate curved surfaces (the first rotation angle and the second rotation angle) are set to the first region and the second region. Since the relative angle is used, it is possible to calculate the relative angle between the regions with a small calculation amount without requiring complicated calculation.

3 is a block diagram showing a functional configuration of the image processing apparatus according to the first embodiment. FIG. It is a block diagram which shows the hardware constitutions of an image processing apparatus. It is a figure for demonstrating one Example of a search process. FIG. 6 is a diagram for explaining a target area extraction process in the first embodiment. It is a figure explaining one Example of rotation processing. 6 is a flowchart showing a block matching process according to the first embodiment. FIG. 8 is a diagram for explaining a target area extraction process in a second modification of the first embodiment. 9 is a flowchart showing a block matching process according to a modified example 3 of the first embodiment. 9 is a block diagram showing a functional configuration of an image processing apparatus according to a second embodiment. FIG. 9 is a flowchart showing a block matching process according to the second embodiment. 9 is a flowchart showing a block matching process according to the second embodiment. FIG. 11 is a block diagram showing a functional configuration of an encoding device according to a third embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  The image processing apparatus 1 according to the first embodiment is a block matching device that performs block matching. Block matching is a process of searching for blocks that are similar to each other in each of two frames. Here, “similar” means that the similarity of images is equal to or more than a predetermined value.

  The block matching device can be used for, for example, interframe predictive coding of a moving image. It can be used to search for similar blocks that may exist in the target frame to be encoded and the encoded reference frame. Further, the block matching device can be used to find an object reflected in a certain still image frame (target frame) from an image of another still image frame (reference frame).

  FIG. 1 is a block diagram showing the functional arrangement of the image processing apparatus according to the first embodiment. FIG. 2 is a block diagram showing the hardware configuration of the image processing apparatus.

  Referring to FIG. 1, the image processing device 1 includes a control unit 10 and a storage unit 20. The storage unit 20 stores frame information 21 and similarity information 22. The frame information 21 is image data of each frame forming a moving image. The similarity information 22 is information indicating the similarity between regions, which is calculated by the block matching process described later. The control unit 10 performs block matching using the frame information 21 stored in the storage unit 20, and records the similarity calculated in the process as the similarity information 22 in the storage unit 20.

  Referring to FIG. 2, the hardware of the image processing device 1 is a computer device 400 as an example. The computer device 400 includes a control circuit 401, a storage device 402, a reading device 403, a recording medium 404, a communication interface 405, an input / output interface 406, an input device 407, and a display device 408. . The control circuit 401, the storage device 402, the reading device 403, the communication interface 405, the input / output interface 406, and the display device 408 are connected to each other by a bus 410. The recording medium 404 is connected to the reading device 403. The input device 407 is connected to the input / output interface 406. Further, the communication interface 405 is connected to the network 409.

  The control circuit 401 controls the entire computer device 400. The control circuit 401 is a processor as an example. The control circuit 401 operates as the control unit 10 illustrated in FIG. 1 by executing an image processing program (not shown) recorded in the storage device 402.

  The storage device 402 stores an image processing program that causes the control circuit 401 to function as the control unit 10. The storage device 402 also stores various data for operating the control circuit 401. The storage device 402 also functions as the storage unit 20 in FIG.

  The reading device 403 reads data from the removable recording medium 404 and writes data to the recording medium 404.

  The recording medium 404 is a non-temporary recording such as an SD (Secure Digital) memory card, an FD (Floppy Disk), a CDCompact Disk (DVD), a DVD (Digital Versatile Disk), a BD (Blu-ray Disk: registered trademark), and a flash memory. It is a medium. The recording medium 404 stores, for example, data to be given to the computer device 400, data generated by the computer device 400, and data obtained by the computer device 400. Further, the recording medium 404 may store the information in the storage unit 20 shown in FIG.

  The communication interface 405 communicatively connects the computer device 400 and another device (not shown) via the network 409.

  The input / output interface 406 is an interface connected to various input / output devices such as the input device 407. The input / output interface 406 outputs the signal input from the input device 407 to the control circuit 401 via the bus 410. The input / output interface 406 also outputs the signal output from the control circuit 401 to the input / output device via the bus 410. The input device 407 is an example of an input / output device.

  The input device 407 is an input device such as a keyboard and a mouse. The input device 407 receives an input operation of information by the user, and outputs a signal corresponding to the input information to the input / output interface 406. For example, when the user inputs an instruction to start block matching to the input device 407, the input device 407 outputs a signal indicating the instruction to the input / output interface 406.

  The display device 408 displays various information on the screen under the control of the control circuit 401.

  The network 409 connects the computer device 400 and other devices so that they can communicate with each other.

  Hereinafter, the details of the control unit 10 in FIG. 1 will be described.

  Referring to FIG. 1, the control unit 10 includes a search unit 11, an extraction unit 12, a determination unit 13, a calculation unit 14, and a rotation unit 15.

  The search unit 11 performs a search process. The search process is a process in which a reference block whose similarity to the target block in the target frame is determined is switched while sliding by a predetermined amount in the reference frame, and a reference block similar to the target block is searched.

  FIG. 3 is a diagram for explaining an example of the search process. The search unit 11 refers to the frame information 21 and selects one rectangular block included in the target frame 100 (first image) as the target block 101 (first block) for similarity determination. For example, the search unit 11 may sequentially select the target block 100 from left to right from top to bottom as the target block 101.

  Next, the search unit 11 sets the reference block 201 that determines the degree of similarity with the target block 101 in the search area 202 of the reference frame 200 (second image). The search unit 11 sequentially switches the reference blocks 201 while sliding the search area 202 in the direction of the arrow 204. In this embodiment, the search area 202, that is, the area in which the reference block 201 is set is limited to a range corresponding to the search area 102 in a predetermined range around the target block 101. This makes it possible to exclude a block having an excessive movement amount from candidates for calculating the degree of similarity with the target block 101, and suppress the processing amount of the search process. In this embodiment, a predetermined area around the target block 101 is set as the search area 102. A search area 202 having the same size as the search area 102 and located at the corresponding coordinates is set in the reference frame 200. The search areas 102 and 202 are areas to be subjected to block matching processing and can be set to any size.

  Next, the search unit 11 uses the result of the rotation process in which the rotation unit 15 described later relatively rotates the target region 103 (first region) and the reference region 203 (second region), and the calculation unit described later. Based on the similarity between the target area 103 and the reference area 203 calculated by 14, the reference block 201 corresponding to the reference area 203 similar to the target area 103 is determined. For example, the search unit 11 sequentially selects the reference blocks 201 over the entire search area 202, and the similarity with the target block 101 for all the reference blocks 201 included in the search area 202 is stored in the similarity information 22. Then, the highest similarity and the reference area 203 corresponding to the highest similarity may be extracted from the stored similarities. The reference block 201 corresponding to the extracted reference area 203 is the search result by the search unit 11. It should be noted that the search unit 11 extracts the reference region 203 corresponding to the similarity when the calculation unit 14 calculates the similarity of 1 or more and the calculated similarity has a similarity of a predetermined value or more. May be. Further, when there are a plurality of similarities equal to or higher than a predetermined value, the search unit 11 may extract the reference area 203 corresponding to the highest similarity among them.

  When the search unit 11 sets the reference block 201 for determining the similarity with the target block 101 in the search area 202 of the reference frame 200, the extraction unit 12 sets the target block 101 centered on the center of gravity of the rectangular target block 101. Is extracted as a target area 103, and the largest circle included in the reference block 201 is centered on the center of gravity of the reference block 201 that is a rectangle of the same size as the target block 101. An area in which is expressed by pixels is extracted as a reference area 203. The target area 103 and the reference area 203 have substantially the same circular shape.

  FIG. 4 is a diagram for explaining the extraction processing of the target area in the first embodiment. Here, an example of extracting the target area 103 from the target block 101 will be described, but the reference area 203 can be extracted from the reference block 201 by the same method.

  Referring to FIG. 4, in the present embodiment, the target block 101 includes 8 × 8 pixels. The extraction unit 12 extracts, as a target area 103, an area centered on the center of gravity of the target block 101 and representing the largest circle included in the target block 101 with pixels. In FIG. 4, pixels included in the target area 103 are hatched.

  In the present embodiment, since the target area 103 and the reference area 203 have the same circular shape, the target area 103 and the reference area 203 are determined in the determination process of determining the coefficient of the approximation function by the determining unit 13 described later. It is possible to reduce the influence of the corners of each rectangular region when relatively rotated. In the process of determining the coefficient of the approximation function, the determination unit 13 described below can obtain an approximate curved surface having a shape close to a circle that is less affected by the rotation angle, and can determine the coefficient with high accuracy.

  The determination unit 13 holds in advance information on a predetermined function as an approximation function. Here, the approximation function includes a coefficient, and the coefficient value can be determined so as to approximate the image curved surface represented by the brightness value of the pixel in the image. The determination unit 13 determines the coefficient value of the approximation function (hereinafter referred to as “first coefficient”) so that the brightness values of the plurality of pixels included in the target area 103 are approximated by the approximation function. At this time, for example, the least squares method may be used for the approximation. The approximation function in which the first coefficient is set indicates an approximated curved surface (including a case of a plane; the same applies hereinafter) that approximates the luminance value of each pixel of the target region 103 on the circular closed plane.

  Further, the determining unit 13 determines the coefficient value of the approximation function (hereinafter referred to as “second coefficient”) so that the brightness value of each pixel in the reference area 203 is approximated by the approximation function. At this time, for example, the least squares method may be used for the approximation. The approximation function in which the second coefficient is set indicates an approximate curved surface that approximates the brightness value of each pixel in the reference area 203 on the circular closed plane.

  An approximate curved surface of the target area 103 and the reference area 203 on the same circular closed plane (closed disk) is obtained.

  The calculating unit 14 obtains the direction of the approximation function (hereinafter referred to as “first rotation angle”) using the first coefficient, and uses the second coefficient to determine the direction of the approximation function (hereinafter referred to as “second rotation angle”). Is calculated and the relative angle between the first rotation angle and the second rotation angle is calculated.

  Expression (1) shows a two-variable quadratic function that is an example of an approximation function. The two-variable quadratic function of Expression (1) includes changeable coefficients a, b, and c. The two-variable quadratic function of the equation (1) should be converted into the standard two-variable quadratic function as shown in the equation (3) not including the rotation angle θ by the coordinate transformation shown in the equation (2). You can The target region 103 and the reference region 203 do not change the coefficients α and β of the equation (3) that does not include the rotation angle θ even when the rotation processing by the rotation unit 15 described below is performed.

仮に、閉円盤Ｄ上の２つの画像曲面が合同であり、ある相対角度Δθをもっている場合、各々の画像曲面を式（１）で最小二乗近似し、式（３）の標準形に変換すると、各々の画像曲面についての回転角を求めることができる。それらの回転角をθ、θ´とすると、回転角θ、θ´は各画像曲面の方向を示す特徴量といえる。そして、２つの画像曲面の相対角度Δθは式（４）により表すことができる。

If the two image curved surfaces on the closed disk D are congruent and have a certain relative angle Δθ, each image curved surface is subjected to least square approximation by the equation (1) and converted into the standard form of the equation (3), The rotation angle for each image curved surface can be obtained. When the rotation angles are θ and θ ′, it can be said that the rotation angles θ and θ ′ are feature amounts indicating the directions of the image curved surfaces. Then, the relative angle Δθ between the two image curved surfaces can be expressed by Expression (4).

算出部１４は、これを利用することにより、対象領域１０３と参照領域２０３のそれぞれの方向を示す回転角θ、θ´を算出し、さらに相対角度Δθを算出することができる。

By using this, the calculation unit 14 can calculate the rotation angles θ and θ ′ indicating the respective directions of the target region 103 and the reference region 203, and further can calculate the relative angle Δθ.

  Further, the calculation unit 14 calculates the similarity between the target region 103 and the reference region 203 after the rotation processing by the rotation unit 15 described later, and records it in the similarity information 22.

  FIG. 5 is a diagram illustrating an example of the rotation process.

  In the following description, it is assumed that only the reference area is rotated in the rotation processing, but the target area may be rotated, or the target area and the reference area may be rotated.

  The rotating unit 15 rotates the reference area 203 so that the relative angle Δθ becomes smaller. Preferably, the rotating unit 15 rotates the reference area 203 so that the relative angle Δθ becomes zero. Since the target area 103 and the reference area 203 have the same circular shape, when the rotating unit 15 rotates the reference area 203, the luminance value in the reference area 203 at the position corresponding to the plurality of pixels included in the target area 103 is It can be calculated from the brightness values of a plurality of pixels included in the reference area 203. For example, a weighted average value of the brightness values of pixels around the position where the brightness value is desired to be calculated may be used as the brightness value of the position.

  FIG. 6 is a flowchart showing the block matching process according to the first embodiment.

  The search unit 11 acquires the target frame 100 and the reference frame 200 from the frame information 21, and determines the target block 101 in the target frame 100 (step S101).

  The search unit 11 determines one reference block 201 in the search area 202 (step S102). As an example, it is assumed that the same position as the position of the target block 101 is set as a reference position and the reference block 201 is sequentially slid by a predetermined width within a predetermined range of the standard position. The reference block 201 of this time is determined at a position slid by a predetermined width from the reference block 201 of the previous time.

  The extraction unit 12 extracts a circular target area 103 corresponding to the target block 101 and a circular reference area 203 corresponding to the reference block 201 (step S103). In the present embodiment, the target area 103 corresponding to the target block 101 is an area centered on the center of gravity of the rectangular target block 101 and corresponding to the largest circle included in the target block 101. Similarly, the reference area 203 corresponding to the reference block 201 is an area corresponding to the largest circle included in the reference block 201 with the center of gravity of the rectangular reference block 201 as the center. The target block 101 and the reference block 201 have the same size, and the target area 103 and the reference area 203 have the same size.

  The determining unit 13 determines the coefficient of the bivariate quadratic function so as to approximate the luminance value of the target region 103 by the quadratic surface represented by the bivariate quadratic function, and sets the coefficient as the first coefficient, The coefficient of the bivariate quadratic variable is determined so that the luminance value of the reference area 203 is approximated by the quadratic surface represented by the bivariate quadratic function, and the coefficient is set as the second coefficient (step S104).

  The calculation unit 14 uses the first coefficient and the second coefficient for the two-variable quadratic function to calculate the relative angle Δθ between the quadratic surface that approximates the target area 103 and the quadratic surface that approximates the reference area 203. Calculate (step S105).

  The rotating unit 15 rotates the reference area 203 by using the relative angle Δθ so that the relative angle between the quadric surface that approximates the target area 103 and the quadric surface that approximates the reference area 203 becomes smaller (step S106). . Preferably, the reference area 203 is rotated so that the relative angle between the quadric surface that approximates the target area 103 and the quadric surface that approximates the reference area 203 becomes zero.

  The calculation unit 14 calculates the degree of similarity between the target area 103 and the reference area 203 after the rotation processing in step S106, and stores the similarity in the similarity information 22 (step S107).

  The search unit 11 determines whether the search is performed by sliding the reference block 201 over the entire search area 202 (step S108).

  The search part 11 returns to step S102, when not searching the whole search area 202 (No in step S108). On the other hand, when the search unit 11 finishes searching the entire search area 202 (Yes in step S108), the search unit 11 refers to the similarity information 22 and matches the reference block 201 having the highest similarity with the target block 101. The reference block 201 is set (step S109). Then, the image processing device 1 ends the block matching process.

  As described above, according to the present embodiment, the determining unit 13 calculates the coefficient of the approximation function so that the brightness value of the pixel included in the first area is approximated by the curved surface of the predetermined approximation function. The coefficient is used as the first coefficient, the coefficient of the approximation function is calculated so that the brightness value of the pixel included in the second region is approximated by the approximation function, and the coefficient is set as the second coefficient. The calculating unit 14 obtains a first rotation angle indicating the direction of the approximation function using the first coefficient, obtains a second rotation angle indicating the direction of the approximation function using the second coefficient, and calculates the first rotation angle as the first rotation angle. The relative angle with the second rotation angle is calculated. Since each of the regions for which the relative angle is calculated is approximated by a curved surface, and the relative angles of the two approximate curved surface directions (the first rotation angle and the second rotation angle) are the relative angles of the first region and the second region, The relative angle between regions can be calculated with a small amount of calculation without requiring complicated calculation.

  Further, according to the present embodiment, the extraction unit 12 includes the circular first area including at least a part of pixels of the rectangular first block and the circular first area including at least a part of pixels of the rectangular second block. 2 areas and are extracted. The rotating unit 15 executes a rotating process of rotating at least one of the first region and the second region so that the relative angle becomes small. The calculation unit 14 further calculates the similarity between the first area and the second area after the rotation processing. Since the circle corresponding to each rectangular block is extracted and the circular area is rotated to calculate the similarity, it is possible to favorably calculate the similarity between the images before and after the rotation.

  Further, according to the present embodiment, the extraction unit 12 matches the center point of the first area with the center of gravity of the first block, and matches the center point of the second area with the center of gravity of the second block. Since the rotation processing is performed by the circular area having the center of gravity of the rectangular block as the center point and the similarity is calculated, a good similarity between the blocks can be obtained by the similarity between the areas.

  In addition, according to the present embodiment, the extraction unit 12 defines the area in which the circle included in the first block is expressed in pixels as the first area and the area in which the circle included in the second block is expressed in pixels as the first area. There are two areas. Since the similarity is calculated from the circular area included in the block, the similarity not affected by the pixel value outside the block can be obtained.

  Further, according to the present embodiment, the search unit 11 moves the second block on the frame of the image, and while the second block moves, the similarity between the first region and the second region calculated by the calculation unit 14 after the rotation processing by the rotation unit 15 is performed. Based on the degree, the second block corresponding to the second area most similar to the first area is searched. As a result, the block matching can be performed with a small amount of calculation by using the rotation process that can calculate the relative angle between the regions with a small amount of calculation.

(Modification 1)
In this example, a two-variable quadratic function as shown in Expression (1) was used as an example of the approximation function, but the embodiment is not limited to this. It is also possible to use a two-variable linear function as a modification.

Expression (5) shows a two-variable linear function which is another example of the approximation function. The two-variable linear function of the equation (5) includes changeable coefficients a ₀ and b ₀ . The two-variable linear function of the equation (5) can be converted into a standard two-variable linear function as shown in the equation (6) not including the rotation angle by coordinate transformation. The target region 103 and the reference region 203 do not change in the coefficient γ of the equation (6) that does not include the rotation angle φ even when the rotation process is performed by the rotation unit 15.

仮に、閉円盤Ｄ上の２つの画像曲面が合同であり、ある相対角度Δφをもっている場合、各々の画像曲面を式（５）で最小二乗近似し、式（６）の標準形に変換すると、各々の画像曲面についての回転角を求めることができる。それらの回転角をφ、φ´とすると、回転角φ、φ´は各画像曲面の方向を示す特徴量といえる。そして、２つの画像曲面の相対角度Δφは式（７）により表すことができる。

If the two image curved surfaces on the closed disk D are congruent and have a certain relative angle Δφ, each image curved surface is least squares approximated by the equation (5) and converted into the standard form of the equation (6), The rotation angle for each image curved surface can be obtained. If the rotation angles are φ and φ ′, then the rotation angles φ and φ ′ can be said to be feature quantities indicating the directions of the respective image curved surfaces. Then, the relative angle Δφ between the two image curved surfaces can be expressed by Expression (7).

算出部１４は、これを利用することにより、対象領域１０３と参照領域２０３のそれぞれの方向を示す回転角φ、φ´を算出し、さらに相対角度Δφを算出することができる。

By using this, the calculation unit 14 can calculate the rotation angles φ and φ ′ indicating the respective directions of the target region 103 and the reference region 203, and further can calculate the relative angle Δφ.

(Modification 2)
Further, in the present embodiment, a reference block which is a rectangle having the same size as the target block 101 is defined as a target region 103, which is an area corresponding to the largest circle included in the target block 101, centered on the center of gravity of the rectangular target block 101. An example has been shown in which the area corresponding to the largest circle included in the reference block 201 is set as the reference area 203 with the center of gravity of 201 as the center. However, the embodiment of the present invention is not limited to this. As a modification, it is possible to set the area including all the pixels of the target block 101 as the target area 103 and the area including all the pixels of the reference block 201 as the reference area 203.

  FIG. 7 is a diagram for explaining the extraction processing of the target area in the second modification of the first embodiment. In the modification of FIG. 7, a region in which the pixels including all the pixels of the target block 101 represent the smallest circle is set as the target region 103, and a region in which the pixels including all the pixels of the reference block 201 represent the smallest circle is set as the target region 103. The reference area 203 is used.

  As described above, according to the present modification, the extraction unit 12 sets the area, which represents a circle including all the pixels of the first block by pixels, as the first area, and the circle including all the pixels of the second block. The area in which the pixel is expressed is the second area. Since the similarity is calculated by the circular area including the block, it is possible to obtain the similarity considering all the pixels of the block.

(Modification 3)
In the present embodiment, while rotating the reference block 201, the relative angle between the target area 103 and the reference area 203 is calculated and the rotation processing is performed, and the similarity between the target area 103 and the reference area 203 after the rotation processing is calculated. Block matching was performed by the procedure of finding the reference block 201 that gives the highest similarity. However, the embodiment of the present invention is not limited to this. As another example, first, a reference block that is a candidate for matching with the target block is narrowed down, a rotation process and a similarity calculation after the rotation process are performed on the narrowed down reference block, and the target block is calculated based on the similarity after the rotation process. You may decide to find a matching reference block. It is possible to reduce the number of executions of the rotation processing and reduce the amount of calculation.

  FIG. 8 is a flowchart showing the block matching process according to the third modification of the first embodiment. Referring to FIG. 8, step S101 is the same as the processing of the first embodiment shown in FIG.

  After step S101, in step S301, the calculation unit 14 calculates the degree of similarity between the target block 101 and all the reference blocks 201 in the search area 202, and the search unit 11 performs matching based on the calculated degree of similarity. One or more reference blocks 201 that are candidates are selected. The matching candidate is the reference block 201 that is the target of the block matching process involving the rotation process with the target block 101. For example, the reference block 201 whose similarity to the target block 101 is a predetermined value or more may be used as the matching candidate. Alternatively, a predetermined number of reference blocks 201 having a higher degree of similarity to the target block 101 may be used as a matching candidate. In the search unit 11, the calculation unit 14 calculates the degree of similarity between the target block 101 and some or all of the reference blocks 201 on the search area 202, and the reference number is equal to or greater than a predetermined value. When the block 201 is found, a predetermined number of reference blocks 201 may be used as matching candidates.

  The search unit 11 determines one reference block 201 from the matching candidates (step S302). Further, after performing steps S103 to S107 shown in FIG. 6, the search unit 11 determines whether or not all reference blocks 201 of matching candidates have been searched (step S303). When the search for all matching candidates has not been completed (No in step S303), the search unit 11 returns to step S302. On the other hand, when all the matching candidates have been searched (Yes in step S303), the search unit 11 refers to the similarity information 22 and matches the reference block 201 having the highest similarity with the target block 101. The reference block 201 is set (step S109). Then, the image processing device 1 ends the block matching process.

  As described above, according to the present modification, the calculation unit 14 calculates the similarity between the first block and each of the second blocks over the entire frame in advance, and the search unit 11 determines the similarity between the first block and the first block. Based on the degree, a matching candidate is selected from the second blocks over the entire frame. Then, the search unit 11 searches for the second block corresponding to the second area most similar to the first area while sequentially selecting the second blocks of the matching candidates. Since the second block of the matching candidates is narrowed down in advance without performing the rotation processing and the rotation processing is performed on the matching candidates, the number of executions of the rotation processing can be reduced and the processing amount of the block matching can be reduced.

  In the first embodiment, an example in which the relative angle between the target region 103 and the reference region 203 is calculated using the approximated curved surface by one approximation function has been shown. However, the embodiment of the present invention is not limited to this. In the second embodiment, an example will be described in which the relative angle between the target region 103 and the reference region 203 is calculated using an approximate curved surface formed by a plurality of approximation functions.

  The second embodiment exemplifies an image processing apparatus that performs the same block matching as in the first embodiment.

  FIG. 9 is a block diagram showing the functional arrangement of the image processing apparatus according to the second embodiment. The hardware configuration of the image processing apparatus 1 according to the second embodiment is the same as that of the first embodiment shown in FIG.

  Referring to FIG. 9, the image processing apparatus 1 according to the second embodiment differs from that of the first embodiment shown in FIG. 1 in that the image processing apparatus 1 according to the second embodiment further includes a selection unit 16 as a basic configuration. Further, the block matching processing executed by the image processing apparatus 1 of the second embodiment is different from that of the first embodiment. The block matching process of the second embodiment differs from that of the first embodiment in that two approximation functions are used. The image processing apparatus 1 performs rotation processing using each of the two approximation functions, calculates the degree of similarity for each, and then selects the relative angle for which a higher degree of similarity is obtained.

  Hereinafter, the operation and processing flow of each unit of the second embodiment will be described focusing on the differences from the first embodiment.

  The determination unit 13 of the second embodiment holds in advance information about two approximation functions. Here, each of the two approximation functions is a function having two variables. The two approximation functions have different orders. Specifically, one is a quadratic function and the other is a linear function. Hereinafter, the two approximation functions will be referred to as a first approximation function and a second approximation function, respectively. The two-variable quadratic function (Equation (1)) used in the first embodiment is used as the first approximation function, and the two-variable linear function used in the modification of the first embodiment is used as the second approximation function. (Formula (5)) is used.

  Then, the determining unit 13 determines the first coefficient of the first approximation function so that the brightness values of the plurality of pixels included in the target area 103 are approximated by the first approximation function. The first approximation function in which the first coefficient is set indicates an approximate curved surface that approximates the brightness value of each pixel of the target area 103 on the circular closed plane. Further, the determining unit 13 determines the second coefficient of the first approximation function so that the brightness values of the plurality of pixels included in the reference area 203 are approximated by the first approximation function. The second approximation function in which the second coefficient is set also indicates an approximate curved surface that approximates the brightness value of each pixel in the reference area 203 on the circular closed plane.

  Further, the determining unit 13 performs the same processing as above using the second approximation function. That is, the determination unit 13 determines the first coefficient of the second approximation function so that the brightness values of the plurality of pixels included in the target area 103 are approximated by the second approximation function. The second approximation function for which the first coefficient is set indicates an approximate curved surface that approximates the brightness value of each pixel of the target area 103 on the circular closed plane. Further, the determining unit 13 determines the second coefficient of the second approximation function so that the brightness values of the plurality of pixels included in the reference area 203 are approximated by the second approximation function. The second approximation function in which the second coefficient is set also indicates an approximate curved surface that approximates the brightness value of each pixel in the reference area 203 on the circular closed plane.

  The calculation unit 14 of the second embodiment uses the first coefficient to determine the direction of the approximation function (first rotation angle) for each of the case where the first approximation function is used and the case where the second approximation function is used. ) Is obtained, the direction of the approximation function (second rotation angle) is obtained using the second coefficient, and the relative angle between the first rotation angle and the second rotation angle is calculated. As a result, two relative angles are calculated for the target area 103 and the reference area 203.

  When the first approximation function is used, as described above, the first approximation function (quadratic function) shown in the equation (1) has changeable coefficients a, b, and c. include. The first approximation function of Expression (1) should be converted into a standard two-variable quadratic function as shown in Expression (3) that does not include the rotation angle θ by the coordinate conversion shown in Expression (2). You can The rotation angle of the approximate curved surface that approximates the image of the target area 103 can be set to θ, and the rotation angle of the approximate curved surface that approximates the image of the reference area 203 can be set to θ ′. The relative angle Δθ between the approximated curved surface of the target area 103 and the approximated curved surface of the reference area 203 can be expressed by Expression (4).

When the second approximation function is used, as described above, the second approximation function (linear function) shown in the equation (5) includes changeable coefficients a ₀ and b _0. Has been. The second approximation function of Expression (5) can be converted into the standard two-variable linear function as shown in Expression (6) not including the rotation angle φ by the coordinate conversion shown in Expression (2). . The rotation angle of the approximate curved surface approximating the image of the target region 103 can be set to φ, and the rotation angle of the approximate curved surface approximating the image of the reference region 203 can be set to φ ′. The relative angle Δφ between the approximated curved surface of the target area 103 and the approximated curved surface of the reference area 203 can be expressed by Expression (7).

  As shown in FIG. 4, an orthogonal coordinate system whose origin is the center of a circle D corresponding to the target area 103 is set, and pixels whose center is inside the circle are included in the target area 103. By using a polynomial (shown in Expression (8)) obtained by adding a constant term d to Expressions (1) and (5) as an objective function and finding a least squares solution thereof (shown in Expression (9)), the first The coefficients of the approximation function and the second approximation function can be calculated.

式（３）の係数α、βは、算出された係数ａ、ｂ、ｃから、式（１０）を用いて算出される。式（６）の係数γは、算出された係数ａ_０、ｂ_０により、式（１１）で算出される。更に、回転角θは式（１２）で算出され、回転角φは式（１３）で算出される。なお、θおよびφはｘ軸を基準に反時計回りの角度とする。

The coefficients α and β of the equation (3) are calculated from the calculated coefficients a, b and c using the equation (10). The coefficient γ of Expression (6) is calculated by Expression (11) using the calculated coefficients a ₀ and b ₀ . Further, the rotation angle θ is calculated by the equation (12), and the rotation angle φ is calculated by the equation (13). Note that θ and φ are counterclockwise angles with respect to the x axis.

算出部１４は、参照領域２０３についても、上記対象領域１０３と同様に計算することで、回転角θ´、φ´を算出する。更に、算出部１４は、式（４）により相対角度Δθを算出し、式（７）により相対角度Δφを算出する。

The calculation unit 14 also calculates the rotation angles θ ′ and φ ′ for the reference area 203 as in the case of the target area 103. Further, the calculation unit 14 calculates the relative angle Δθ by the equation (4) and the relative angle Δφ by the equation (7).

  Furthermore, the calculation unit 14 calculates the similarity between the target region 103 and the reference region 203 after the rotation processing by the rotation unit 15 for the first approximation function and the second approximation function (first similarity and second similarity). Of the first similarity and the second similarity, and the similarity showing the higher value (more similar) is recorded in the similarity information 22.

  The rotating unit 15 rotates the reference region 203 so that the relative angle Δθ becomes small, preferably the relative angle Δθ becomes zero. Further, the rotating unit 15 rotates the reference region 203 so that the relative angle Δφ becomes small, preferably the relative angle Δφ becomes zero. Here, in the rotation process, only the reference area is described as being rotated, but the target area may be rotated, or the target area and the reference area may be rotated.

  The selection unit 16 determines which of the relative angle Δθ and the relative angle Δφ is higher in similarity after the rotation processing, and determines the higher similarity as the relative angle between the target region 103 and the reference region 203. Then, it is recorded in the similarity information 22.

  10 and 11 are flowcharts showing the block matching processing according to the second embodiment.

  Referring to FIG. 10, steps S101 to S103 are the same as the processing of the first embodiment shown in FIG. After step S103, the determination unit 13 approximates the brightness value of the target area 103 by the curved surface and the plane represented by the approximation function for each of the first approximation function and the second approximation function. The coefficient of the approximation function is determined, and the coefficient is set as the first coefficient. Further, the determining unit 13 approximates each of the first approximation function and the second approximation function so that the brightness value of the reference region 203 is approximated by the curved surface and the plane represented by the approximation function. Is determined and the coefficient is set as the second coefficient (step S201).

  The calculating unit 14 uses the first coefficient and the second coefficient for each of the first approximation function and the second approximation function, and uses the respective first and second coefficients to approximate the target area 103 and the reference area 203. And the relative angle Δθ and the relative angle Δφ are calculated (step S202). Since the first approximation function is a quadratic function, when the first approximation function is used, each image curved surface of the target region 103 and the reference region 203 is approximated by a quadric surface. Since the second approximation function is a linear function, when the second approximation function is used, each image curved surface of the target region 103 and the reference region 203 is approximated by a linear plane. The relative angle calculated using the first approximation function is Δθ, and the relative angle calculated using the second approximation function is Δφ.

  In step S106, the rotation unit 15 reduces the relative angle between the quadric surface that approximates the target region 103 and the quadric surface that approximates the reference region 203 by using the relative angle Δθ, as in the first embodiment. When the reference area 203 is rotated, the calculation unit 14 calculates the similarity between the target area 103 and the reference area 203 after the rotation processing in step S106 as the first similarity (step S203). Step S203 of the second embodiment differs from step S107 of the first embodiment in that the first similarity calculated here is not yet recorded in the similarity information 22.

  Subsequently, the same processing as the processing executed using the relative angle Δθ in steps S106 to S203 is executed using the relative angle Δφ.

  That is, in step S204, the rotation unit 15 rotates the reference area 203 so that the relative angle between the primary plane that approximates the target area 103 and the primary plane that approximates the reference area 203 becomes smaller using the relative angle Δφ. . In step S205, the calculation unit 14 calculates the similarity between the target area 103 and the reference area 203 after the rotation processing of step S204 as the second similarity (step S205). The second similarity calculated here is not yet recorded in the similarity information 22.

  Referring to FIG. 11, the selection unit 16 determines whether the first similarity is equal to or higher than the second similarity (step S206). That is, the selection unit 16 uses the result of the rotation process that relatively rotates the target region 103 and the reference region 203, and the relative angle at which the target region 103 and the reference region 203 are more similar is Δθ or Δφ. Determine if there is.

  If the first similarity is equal to or higher than the second similarity, the selection unit 16 records the first similarity in the similarity information 22 (step S207). If the first similarity is not equal to or higher than the second similarity, the selection unit 16 records the second similarity in the similarity information 22 (step S208). That is, the selection unit 16 stores the similarity between the target area 103 and the reference area 203, which is obtained by rotating the reference area 203 at the relative angle with which the similarity becomes higher, in the similarity information 22.

  Steps S108 to S109 are the same as the processing of the first embodiment shown in FIG. Then, the image processing device 1 ends the block matching process.

  As described above, according to the present embodiment, the determining unit 13 determines the first coefficient and the second coefficient of each two-variable function using a plurality of two-variable functions having different orders. The calculation unit 14 calculates each relative angle using the first coefficient and the second coefficient for each of the two variable functions. The rotating unit 15 performs a rotating process on each of the relative angles. The selection unit 16 selects a relative angle among the relative angles in which the first region and the second region are more similar after the rotation process. Since a more similar relative angle is selected from the relative angles obtained by approximation using a plurality of functions having different orders, the accuracy of the relative angle can be improved.

  As described above, the image processing illustrated in the first embodiment can be used for moving image compression by interframe predictive coding. This embodiment exemplifies an encoding device for performing interframe predictive encoding of video. The differences between the encoding apparatus according to the third embodiment and the image processing apparatus 1 according to the first embodiment will be mainly described below.

  FIG. 12 is a block diagram showing the functional arrangement of the encoding device according to the third embodiment.

  The encoding device 2 encodes the moving image stored in the frame information 21 by the inter-frame predictive encoding method, and records the encoded frame in the encoded frame information 23. The moving image is composed of time-series frames. In inter-frame prediction, the image of the target frame to be encoded is predicted based on the reference frame that has been encoded at a different time, and the difference between the actual image of the target frame and the predicted image is encoded to create a moving image. It is for compressing image data.

  The encoding device 2 processes in units of blocks obtained by dividing a frame into a plurality of parts. In the processing on a block-by-block basis, the encoding device 2 searches the reference frame for a reference block similar to the target block in the target frame, and obtains a difference image between the searched reference block and the target block. The encoding device 2 further obtains a motion vector indicating the relative position between the target block and the reference block. Then, the encoding device 2 records the difference image and the motion vector in the encoded frame information 23.

  As shown in FIG. 12, the encoding device 2 according to the third embodiment differs from the image processing device 1 according to the first embodiment in that the encoding device 2 further includes an encoding unit 17.

  The processing of the search unit 11 in the encoding device 2 will be described with reference to FIG. In order to perform the inter-frame encoding of the moving image, the search unit 11 uses the rotating unit 15 while sliding the reference block 201 on the reference frame 200 on which the moving image has been encoded, more specifically, in the search area 202. The reference block 201 from which the reference area 203 similar to the target area 103 is extracted is searched based on the similarity between the target area 103 and the reference area 203 calculated after the rotation processing.

  Further, the calculation unit 14 in the present encoding device 2 stores the information on the degree of similarity between the target block 101 and the reference block 201 and the relative angle Δθ between the target region 103 and the reference region 203 when the similarity is calculated. To record.

  The encoding unit 17 performs interframe predictive encoding processing on each frame of the frame information 21, generates a difference image and a motion vector for a block in each frame, and records the difference image and the motion vector in the encoded frame information 23. The generated difference image is a difference image between the target block 101 and the reference block 201 corresponding to the reference area 203 similar to the target area 103. The generated motion vector is vector information indicating the relative position of the target block 101 and the reference block 201. The relative position between the target block 101 and the reference block 201 includes the rotation angle and the translation amount. The encoding unit 17 sets the relative angle Δθ calculated by the calculation unit 14 and recorded in the storage unit 20 as the rotation angle of the target block 101 and the reference block 201.

  As described above, according to the present embodiment, the search unit 11 moves the second block on the reference frame that has been subjected to the encoding process of the moving image, and the first region and the second region calculated after the rotation process. The second block in which the second region similar to the first region is extracted is searched for based on the similarity degree of. The encoding unit 17 determines the difference image between the first block and the second block corresponding to the second region similar to the first region, and the relative position information indicating the relative positional relationship between the first block and the second block ( Motion vector) is generated. By using the rotation process that can calculate the relative angle between regions with a small amount of calculation, a moving image can be encoded by the inter-frame predictive encoding method with a small amount of calculation.

  Further, according to the present embodiment, the relative position information (motion vector) includes the relative angle between the second block and the first block, and the encoding unit 17 determines that the first area and the second area similar to the first area. The relative angle with the area is the relative angle between the first block and the second block. In the inter-frame predictive coding, the relative angle calculated with a small amount of calculation by block matching is used as the relative angle between the blocks included in the coding result, so that the inter-frame coding can be performed with a small amount of calculation.

  The above-described embodiments of the present invention are examples for explaining the present invention, and are not intended to limit the scope of the present invention only to those embodiments. Those skilled in the art can implement the present invention in various other modes without departing from the scope of the present invention. The apparatus of each of the above embodiments has been illustrated as being realized by executing a software program on a processor, but the present invention is not limited to this. A part or all of the functions of the control unit 10 may be realized by hardware. Further, a part or all of the functions of the control unit 10 may be realized by a dedicated processor.

DESCRIPTION OF SYMBOLS 1 ... Image processing device, 2 ... Encoding device, 10 ... Control part, 11 ... Search part, 12 ... Extraction part, 13 ... Determination part, 14 ... Calculation part, 15 ... Rotation part, 16 ... Selection part, 17 ... Code Coding unit, 20 ... storage unit, 21 ... frame information, 22 ... similarity information, 23 ... coding frame information, 100 ... target frame, 101 ... target block, 102 ... search area, 103 ... target area, 200 ... reference frame , 201 ... Reference block, 202 ... Search area, 203 ... Reference area, 204 ... Arrow, 400 ... Computer device, 401 ... Control circuit, 402 ... Storage device, 403 ... Reading device, 404 ... Recording medium, 405 ... Communication interface, 406 ... Input / output interface, 407 ... Input device, 408 ... Display device, 409 ... Network, 410 ... Bus

Claims (8)

The brightness of the pixels included in the second region is calculated by calculating the coefficient of the approximation function so that the brightness value of the pixels included in the first region is approximated by a curved surface using a predetermined approximation function. A determining unit that calculates a coefficient of the approximation function so that the value is approximated by the approximation function, and sets the coefficient as a second coefficient;
A first rotation angle indicating the direction of the approximation function is obtained using the first coefficient, a second rotation angle indicating the direction of the approximation function is obtained using the second coefficient, and the first rotation angle A calculation unit that calculates a relative angle with the second rotation angle;
An image processing apparatus having. An extraction unit that extracts the circular first region including at least a portion of pixels of the rectangular first block and the circular second region including at least a portion of pixels of the rectangular second block;
A rotation unit that performs a rotation process of rotating at least one of the first region and the second region so that the relative angle becomes smaller,
The calculator further calculates the degree of similarity between the first area and the second area after the rotation processing,
The image processing apparatus according to claim 1. The determining unit determines the first coefficient and the second coefficient of each of the two-variable functions using a plurality of two-variable functions of different orders,
The calculation unit calculates the relative angles using the first coefficient and the second coefficient for each of the two-variable functions,
The rotation unit performs the rotation process for each of the relative angles,
The image processing apparatus further includes a selection unit that selects, from the relative angles, a relative angle at which the first region and the second region are more similar to each other after the rotation process.
The image processing device according to claim 2. While moving the second block on the frame of the image, based on the similarity between the first region and the second region calculated by the calculation unit after the rotation processing by the rotation unit, A search unit that searches for a second block corresponding to the most similar second region;
The image processing apparatus according to claim 2. The calculation unit calculates in advance the degree of similarity between the first block and each of the second blocks over the search region on the frame determined based on the position of the first block on the frame, and the search unit However, a matching candidate is selected from the second blocks over the entire frame based on the similarity with the first block,
The search unit searches for a second block corresponding to the second area that is most similar to the first area while sequentially selecting the second blocks of the matching candidates.
The image processing apparatus according to claim 4. The image processing device is an encoding device that encodes a moving image by interframe predictive encoding,
The search unit moves the second block on a reference frame that has been subjected to encoding processing of the moving image, and based on the similarity between the first area and the second area calculated after the rotation processing, Searching for a second block from which the second region similar to the first region is extracted,
The image processing device,
Generating a difference image between the first block and a second block corresponding to the second region similar to the first region, and relative position information indicating a relative positional relationship between the first block and the second block An encoding unit that
The image processing apparatus according to claim 4, further comprising: The determining unit calculates a coefficient of the approximation function so that the brightness value of a pixel included in the first area is approximated by a curved surface by a predetermined approximation function, sets the coefficient as a first coefficient, and includes the coefficient in the second area. The coefficient of the approximation function is calculated so as to approximate the brightness value of the pixel to be approximated by the approximation function, and the coefficient is set as the second coefficient,
The calculation unit obtains a first rotation angle indicating the direction of the approximation function using the first coefficient, obtains a second rotation angle indicating the direction of the approximation function using the second coefficient, and calculates the first rotation angle. Calculating a relative angle between one rotation angle and the second rotation angle,
Image processing method. A software program for causing a processor included in a computer to execute,
The computer is executed by the processor,
The brightness of the pixels included in the second region is calculated by calculating the coefficient of the approximation function so that the brightness value of the pixels included in the first region is approximated by a curved surface using a predetermined approximation function. A determining unit that calculates a coefficient of the approximation function so that the value is approximated by the approximation function, and sets the coefficient as a second coefficient;
A first rotation angle indicating the direction of the approximation function is obtained using the first coefficient, a second rotation angle indicating the direction of the approximation function is obtained using the second coefficient, and the first rotation angle A calculation unit that calculates a relative angle with the second rotation angle;
Image processing program to operate as.

Images