JP4846719B2 - Panorama motion estimation and compensation - Google Patents

Description

  The present invention relates to panoramic video motion estimation and compensation, and more particularly to a panoramic video motion estimation method, motion estimation device, motion compensation method, and motion compensation device including 360 ° omnidirectional video information.

  An omnidirectional video camera system refers to a camera system capable of shooting 360-degree omnidirectional video with a fixed viewpoint as a reference. In the omnidirectional video camera system, a specially shaped mirror such as a hyperboloid mirror or a special lens such as a fisheye lens is attached to the camera, or a plurality of cameras are used to shoot in all directions.

  As an example to which the omnidirectional video coding method is applied, there is 3D reality broadcasting. As an example of a three-dimensional reality broadcast, video information about scenes from various viewpoints is given to a viewer's terminal in a baseball game or the like. In other words, the viewer is provided with a variety of video information from the viewpoint of the pitcher, the viewpoint from the catcher, the viewpoint from the batter's position, the viewpoint from the position of the first-seat audience, and the like. The viewer can determine the viewpoint he / she wants to see and view the video from that viewpoint.

  An image captured by the omnidirectional camera system has characteristics corresponding to a three-dimensional spherical environment. Therefore, the 3D image captured by the omnidirectional camera system is converted into a 2D plane image. At this time, the two-dimensional planar image is a panoramic image including a 360 ° omnidirectional image. Omnidirectional video coding is performed on the two-dimensional panoramic image.

  On the other hand, motion estimation, which is one of the video encoding techniques, uses a predetermined evaluation function to search for the data unit most similar to the data unit in the current frame in the previous frame, and locates both data units. This is a process of obtaining a motion vector representing the difference between the two. Here, a macro block having a size of 16 × 16 is mainly used as a data unit, but blocks of various sizes such as 16 × 8, 8 × 16, and 8 × 8 blocks can be used.

  A motion estimation process according to an example of the conventional technique performed in units of 16 × 16 macroblocks will be described in more detail. First, a motion vector of a current macroblock is predicted using motion vectors of a plurality of previous macroblocks adjacent to the current macroblock. FIG. 1 is a diagram illustrating a plurality of previous macroblocks used for motion vector prediction of a current macroblock. X represents the current macroblock, and A, B, C, and D are previous macroblocks encoded before the current macroblock X.

  However, depending on the position of the current macroblock X in the current frame, a previous macroblock that cannot be used is generated. FIG. 2A shows a case where there are no previous macroblocks B, C, and D used to predict the motion vector of the current macroblock X. In such a case, the motion vector of the current macroblock X is set to zero.

  FIG. 2B shows a case where there are no previous macroblocks A and D used for prediction of the motion vector of the current macroblock X. In such a case, the motion vectors of the previous macroblocks A and D are set to 0, respectively. The motion vector of the current macroblock X is set to the intermediate value of the previous macroblocks A, B, C and D.

  FIG. 2C shows a case where there is no previous macroblock C used for prediction of the motion vector of the current macroblock X. In such a case, the motion vector of the previous macroblock C is set to 0, respectively. The motion vector of the current macroblock X is set to the intermediate value of the previous macroblocks A, B, C and D.

  If the predicted motion vector of the current macroblock X is obtained, the degree of similarity between the reference macroblock in the reference frame and the current macroblock represented by the predicted motion vector is calculated using a predetermined evaluation function. Thereafter, a reference macroblock in a reference frame most similar to the current macroblock is searched for within a predetermined search range. As the predetermined evaluation function, a SAD (Sum of Absolute Difference) function, a SATD (Sum of Absolute Transformed Difference) function, or an SSD (Sum of Squared Difference) function is generally used.

  When searching for a reference macroblock most similar to the current macroblock within a predetermined search range, some or all pixels of the reference macroblock may exist outside the reference video. In such a case, as shown in FIG. 3, the pixel value located at the left boundary of the reference image is padded to the outside area of the left boundary of the reference image, and the pixel value located at the right boundary of the reference image is After padding the outer region of the right boundary, motion prediction and compensation are performed. The motion prediction and compensation by such a method is called motion prediction and compensation by the UMV (Unrestricted Motion Vector) mode.

  4A is a drawing showing a cylindrical image including a 360 ° omnidirectional image, and FIG. 4B is a 360 ° image generated by cutting the cylindrical image shown in FIG. 4A along the reference line X. Represents panoramic video including omnidirectional video. As shown in FIG. 4B, it can be seen that the left portion A of the human-like object is located at the right boundary portion of the panoramic image, and the right portion B of the human-like object is located at the left boundary portion of the panoramic image. . That is, the spatial correlation between the right boundary portion and the left boundary portion of the panoramic image including the 360 ° omnidirectional image is very high.

  Without considering such characteristics of panoramic video including omnidirectional video information, it is not efficient to apply the above-described conventional motion prediction and compensation method to panoramic video including omnidirectional video information as it is. Therefore, a motion prediction and compensation method suitable for panoramic video including omnidirectional video information is required.

  The technical problem to be solved by the present invention is to provide a more efficient and accurate motion estimation method and apparatus that match the characteristics of panoramic video including omnidirectional video information.

  Another technical problem to be solved by the present invention is to provide a more efficient and accurate motion compensation method and apparatus that match the characteristics of panoramic video including omnidirectional video information.

  One feature of the present invention for solving the above problem is that in a panoramic video motion estimation method including 360 ° omnidirectional video information, a left padding area of a basic reference video for motion estimation of a current panoramic video is used. Is padded using the right boundary area, and the right reference padding area of the basic reference video is padded using the left boundary area, and then the padded video is expanded to be the reference video. And using the motion vectors of a plurality of previous data units adjacent to the current data unit included in the current panoramic image to predict the motion vector of the current data unit; If a sub-pixel included in the reference data unit represented by the predicted motion vector belongs to the reference video, the pixel value of the sub-pixel is set as the sub-pixel. If the sub-pixel is located outside the reference image, the distance on the x-axis between one boundary of the basic reference image near the sub-pixel and the sub-pixel is determined as the basic pixel. The reference data unit is obtained from the reference image by adding or subtracting the value to the x coordinate of the other boundary of the reference image to set the x coordinate of the subpixel and obtaining the pixel value of the subpixel. And obtaining a similarity between the current data unit and the reference data unit using a predetermined evaluation function.

  According to another aspect of the present invention, in the panoramic video motion estimation method including 360 ° omnidirectional video information, the left padding region of the basic reference video for motion estimation of the current panoramic video has a left boundary pixel value. And padding in the right padding area of the basic reference video using the right boundary pixel value, and then extending the padded video to set as a reference video; Predicting a motion vector of the current data unit using motion vectors of a plurality of previous data units adjacent to the current data unit included in the current panoramic image; and the predicted motion vector If the subpixel included in the reference data unit represented by belongs to the basic reference video of the reference video, obtain the pixel value of the subpixel, When the subpixel belongs to the padding area of the reference image or outside the reference image, the distance on the x-axis between one boundary of the basic reference image near the subpixel and the subpixel is determined. Adding or subtracting the x coordinate value of the subpixel to the x coordinate of the other boundary of the basic reference image to obtain the pixel value of the subpixel, thereby obtaining the pixel value of the subpixel. Obtaining all pixel values of the reference data unit, and determining a similarity between the current data unit and the reference data unit using a predetermined evaluation function.

  According to another aspect of the present invention, in the panoramic video motion estimation apparatus including 360 ° omnidirectional video information, the left padding area of the basic reference video for motion estimation of the current panoramic video has a right boundary area. In the right padding area of the basic reference image, the padding image is expanded using the left boundary area and the reference image generated by extending the padded image and the current reference image. A memory storing motion vectors of a plurality of previous data units adjacent to the current data unit included in the panoramic image; and a motion of the current data unit using the motion vectors of the plurality of previous data units When a vector is predicted and a sub-pixel included in a reference data unit represented by the predicted motion vector belongs to the reference image, If the pixel value of the sub-pixel is obtained as it is and the sub-pixel is located outside the reference image, an x-axis between one boundary of the basic reference image near the sub-pixel and the sub-pixel By adding the distance above to the x coordinate of the other boundary of the basic reference image or setting the subtracted value to the x coordinate of the sub pixel to obtain the pixel value of the sub pixel, A motion estimator that determines a similarity between the current data unit and the reference data unit using a predetermined evaluation function after obtaining all pixel values of the reference data unit from a reference video. It is.

  According to another aspect of the present invention, in the panoramic video motion estimation apparatus including 360 ° omnidirectional video information, the left padding region of the basic reference video for motion estimation of the current panoramic video has a left boundary pixel. In the right padding area of the basic reference image, padding is performed using the right boundary pixel value, and then the reference image generated by expanding the padded image and A memory storing motion vectors of a plurality of previous data units adjacent to a current data unit included in a current panoramic image; and the current data unit using the motion vectors of the plurality of previous data units A sub-pixel included in a reference data unit represented by the predicted motion vector is the base of the reference video If the sub-pixel belongs to a reference image, the pixel value of the sub-pixel is obtained, and if the sub-pixel belongs to the padding area of the reference image or outside the reference image, one of the basic reference images near the sub-pixel is obtained. The distance on the x-axis between the boundary and the sub-pixel is added to the x-coordinate of the other boundary of the basic reference image, or the subtracted value is set as the x-coordinate value of the sub-pixel, and After obtaining all pixel values of the reference data unit from the reference image by obtaining a pixel value of a sub-pixel, the similarity between the current data unit and the reference data unit using a predetermined evaluation function And a motion estimator for determining the degree.

  According to another aspect of the present invention, in the panoramic video motion compensation method including 360 ° omnidirectional video information, the left padding area of the basic reference video for motion compensation of the current panoramic video has a right boundary area. And padding the right reference padding area of the basic reference image using the left boundary area, and then expanding the padded image to generate a reference image; and A step of inputting a motion vector of a current data unit included in a current panoramic image, and a sub-pixel included in a reference data unit represented by the motion vector of the current data unit belongs to the reference image; When the pixel value of the sub-pixel is obtained as it is and the sub-pixel is located outside the reference image, the basic reference image near the sub-pixel is obtained. The distance on the x-axis between one boundary of the sub-pixel and the sub-pixel is added to the x-coordinate of the other boundary of the basic reference image, or the subtracted value is set as the x-coordinate of the sub-pixel. Obtaining all pixel values of the reference data unit from the reference image by obtaining a pixel value of the sub-pixel, and reproducing the current data unit using the pixel value of the reference data unit Steps.

  According to another aspect of the present invention, in the panoramic video motion compensation method including 360 ° omnidirectional video information, the left padding region of the basic reference video for motion compensation of the current panoramic video has a left boundary pixel. Padding using a value, padding a right padding area of the basic reference video using a right boundary pixel value, and then generating a reference video by extending the padded video A step of inputting a motion vector of the current data unit included in the current panoramic image, and a sub-pixel included in the reference data unit represented by the motion vector of the current data unit is the basic of the reference image If it belongs to the reference video, the pixel value of the sub-pixel is acquired, and the sub-pixel belongs to the padding area of the reference video or outside the reference video In this case, the distance on the x-axis between one boundary of the basic reference image near the subpixel and the subpixel is added to or subtracted from the x coordinate of the other boundary of the basic reference image. Obtaining all pixel values of the reference data unit from the reference video by setting the obtained value to the x-coordinate value of the sub-pixel and obtaining the pixel value of the sub-pixel; and Replaying the current data unit using pixel values.

  According to another aspect of the present invention, in the panorama video motion compensation apparatus including 360 ° omnidirectional video information, the left padding region of the basic reference video for motion compensation of the current panoramic video has a right boundary region. In the right padding area of the basic reference video, padding is performed using the left boundary area, and the generated reference video is stored by extending the padded video. When a motion vector of a current data unit included in the current panoramic image is input to a memory and a sub-pixel included in a reference data unit represented by the input motion vector belongs to the reference image, If the pixel value of the sub-pixel is obtained as it is and the sub-pixel is located outside the reference image, one of the basic reference images near the sub-pixel The distance on the x-axis between the field and the sub-pixel is added to the x-coordinate of the other boundary of the basic reference image, or the subtracted value is set as the x-coordinate of the sub-pixel, and the sub-pixel is set. A motion compensation unit that obtains all pixel values of the reference data unit from the reference image by acquiring a pixel value of the pixel and then reproduces the current data unit using the pixel value of the reference data unit And providing.

  According to another aspect of the present invention, in the panoramic video motion compensation apparatus including 360 ° omnidirectional video information, the left padding region of the basic reference video for current panoramic video motion compensation includes a left boundary pixel. In the right padding area of the basic reference image, padding is performed using the right boundary pixel value, and then the reference image generated by expanding the padded image is displayed. A memory to be stored and a motion vector of the current data unit included in the current panoramic image are input, and a sub-pixel included in the reference data unit represented by the motion vector of the current data unit is If it belongs to the basic reference video, the pixel value of the sub-pixel is acquired, and the sub-pixel belongs to the padding area of the reference video or outside the reference video. The distance on the x-axis between one boundary of the basic reference image near the subpixel and the subpixel is added to the x coordinate of the other boundary of the basic reference image, or After obtaining all pixel values of the reference data unit from the reference image by setting the subtracted value to the x coordinate value of the subpixel and obtaining the pixel value of the subpixel, the reference data unit A motion compensation unit that reproduces the current data unit using the pixel value of the current data unit.

  According to the present invention, panoramic image motion prediction and compensation are performed using the very high nature of the spatial correlation between the right and left boundary portions of panoramic images including 360 ° omnidirectional images. Therefore, the efficiency and accuracy of motion estimation and compensation can be improved, and the image quality can be improved. Particularly, according to the present invention, the efficiency and accuracy of motion prediction and compensation of the right and left boundary portions of panoramic images including 360 ° omnidirectional images are improved, and the image quality of the right and left boundary portions is improved. Can be improved.

  Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 5 is a block diagram of a motion vector encoder for panoramic video according to an embodiment of the present invention. As shown in FIG. 5, the motion vector encoder for panoramic video according to an embodiment of the present invention includes a transform unit 110, a quantization unit 115, an inverse quantization unit 120, an inverse transform unit 125, an addition unit 130, and a clipping unit 140. , A frame memory 150, a panoramic video motion estimation unit 160, a panoramic video motion compensation unit 170, a subtraction unit 180, and a variable length coding unit (VLC) 190.

  The conversion unit 110 receives a panoramic image, converts the input panoramic image by a predetermined conversion method, and then outputs a conversion coefficient value. The input panoramic image is a panoramic image including a 360 ° omnidirectional image as illustrated in FIG. 4B generated by cutting the cylindrical image illustrated in FIG. 4A along the reference line X. An example of the predetermined conversion method performed by the conversion unit 110 is DCT (Discrete Cosine Transform) in units of 8 × 8 blocks.

  The quantization unit 115 quantizes the transform coefficient value input from the transform unit 110. The input panorama image is reproduced after the inverse quantization and inverse transform by the inverse quantization unit 120 and the inverse transform unit 125, respectively. The reproduced panoramic image is normalized by the clipping unit 140 and then stored in the frame memory 150. The panoramic video stored in the frame memory 150 is used as a reference video for motion estimation and compensation of the newly input panoramic video.

  The panoramic video motion estimation unit 160 performs motion estimation according to the present invention using the reference panoramic video stored in the frame memory 150. That is, the panoramic video motion estimation unit 160 receives the current panoramic video information, performs a motion estimation of the current panoramic video using the reference panoramic video stored in the frame memory 150, and performs the current panoramic video motion estimation. After generating the motion vector of the panoramic image, it is output to the VLC 190. Motion estimation and compensation are performed in units of a predetermined size block. A unit block for motion estimation and compensation is called a data unit. The data unit in this embodiment is assumed to be a 16 × 16 macroblock.

  The panoramic image motion compensation unit 170 performs motion compensation according to the present invention. That is, the motion vector of the current macroblock generated by the panoramic video motion estimation unit 160 is input, and the reference macroblock corresponding to the current macroblock is output to the subtraction unit 180. The subtraction unit 180 outputs a residual signal between the current macro block and the reference macro block to the conversion unit 110. The residual signal is transformed and quantized by the transforming unit 110 and the quantizing unit 115, respectively, and then subjected to variable length coding by the VLC 190. On the other hand, the motion vector of the current macroblock generated by the panoramic video motion estimation unit 160 is directly input to the VLC 190 and subjected to variable length coding.

  Hereinafter, the operation of the panoramic image motion estimation unit 160 will be described in more detail with reference to the drawings.

  6A and 6B are flowcharts of a panoramic image motion estimation method for integer pixel search according to an embodiment of the present invention. The panoramic image motion estimation unit 160 predicts a motion vector of the current data unit using motion vectors of a plurality of previous data units adjacent to the current data unit (310). As shown in FIG. 1, when data unit X is the current data unit, data units A, B, C, and D represent the previous data necessary for the motion vector prediction of current data unit X. Is a unit. In the present embodiment, the data unit is a 16 × 16 macro block.

  The panorama video motion estimation unit 160 searches for motion vectors of the previous macroblocks A, B, C, and D stored in the internal memory. If there are any motion vectors of the previous macroblocks A, B, C, and D, the motion vector of the current macroblock X is predicted using a predetermined method or a conventional technique using these motion vectors.

  However, at least one of the motion vectors of the previous macroblocks A, B, C, and D may not exist. FIG. 7A shows a case where the previous macroblocks A and D do not exist and the motion vectors of the previous macroblocks A and D cannot be used, and FIG. 7B shows that the previous macroblock C does not exist and the previous macroblocks A and D cannot be used. This represents a case where the motion vector of block C cannot be used.

  As described above, the spatial correlation between the right boundary portion and the left boundary portion of the panoramic image including the 360 ° omnidirectional image is very high. That is, the distance in the actual space between the right boundary portion and the left boundary portion of the panoramic image is zero. Therefore, in the present invention, if there is a previous macroblock that does not exist among the previous macroblocks A, C, and D required for the prediction of the motion vector of the current macroblock X, the characteristics of the panoramic image described above are used. To determine the required motion vector of the previous macroblock. That is, as shown in FIG. 7A, it can be said that the previous macroblock D ′ existing at the right boundary portion on the same Y axis as the previous macroblock D is the same as the previous macroblock D. Therefore, the motion vector of the previous macroblock D ′ is regarded as the motion vector of the previous macroblock D and is used for the prediction of the motion vector of the current macroblock X. However, since the previous macroblock existing in the right boundary portion on the same Y axis as the previous macroblock A is estimated after the motion estimation of the current macroblock X, there is no motion vector to use. Therefore, the motion vector of the previous macroblock A used for prediction of the motion vector of the current macroblock X is set to zero.

  As shown in FIG. 7B, it can be seen that the previous macroblock C ′ and the previous macroblock C existing at the left boundary portion on the same Y axis as the previous macroblock C are the same. Therefore, the motion vector of the previous macroblock C ′ is regarded as the motion vector of the previous macroblock C and is used for the prediction of the motion vector of the current macroblock X.

  After predicting the motion vector of the current macroblock X in step 310, the panoramic video motion estimation unit 160 determines whether the reference macroblock represented by the predicted motion vector exists in the reference video (315). ). The reference video is stored in the frame memory 150.

  If every pixel of the reference macroblock represented by the motion vector of the current macroblock X is present in the reference image, all pixel values of the reference macroblock are obtained from the frame memory 150 (330), and then the current macroblock X And the reference macroblock are evaluated using a predetermined evaluation function (335).

  However, if some or all pixels of the reference macroblock represented by the motion vector of the current macroblock X exist outside one of the left boundary and the right boundary of the reference video, the reference video An image from another boundary to a predetermined range is padded outside the one boundary (320).

  FIG. 8A shows a case where the reference macroblock exists over the boundary of the reference video, and FIG. According to the conventional technique shown in FIG. 3, the pixel value located at the left boundary of the reference image is padded to the outside area of the left boundary of the reference image, and the pixel value located at the right boundary of the reference image is padded to the right side of the reference image. Motion prediction and compensation were performed after padding outside the boundary.

  However, in the present invention, by using the very high nature of the spatial correlation between the right and left boundary portions of a panoramic image including 360 ° omnidirectional images, as shown in FIG. Padding the outer area. As shown in FIG. 9, the outer region 480 on the left boundary of the reference image 400 is padded with pixel values located in the right boundary region 470 of the reference image 400. The outer region 460 at the right boundary of the reference image 400 is padded with the pixel value located in the left boundary region 450 of the reference image 400.

  The panoramic image motion estimation unit 160 pads the reference image in step 320, and then acquires all pixel values of the reference macroblock from the padded reference image from the frame memory 150 (325). Thereafter, the similarity between the current macroblock X and the reference macroblock is evaluated using a predetermined evaluation function (335). As the predetermined evaluation function, a SAD function, a SATD function, or an SSD function is generally used.

  On the other hand, without performing the padding process as described above, by connecting the left boundary and the right boundary of the reference image, and assuming that the reference image is a cylindrical image, from the cylindrical image All pixel values of the reference data unit may be obtained. More specifically, the reference image is a two-dimensional plane image as shown in FIG. 4B. If the left boundary and the right boundary of such a two-dimensional planar image are connected, a cylindrical image as shown in FIG. 4A is obtained. Assuming that the reference image is a cylindrical image, all pixel values of the reference data unit are obtained from the cylindrical image.

  The panoramic video motion estimation unit 160 changes the position of the reference macroblock within a predetermined search range, and evaluates the similarity between the changed reference macroblock and the current macroblock X (340, 345). When the evaluation of the similarity between the plurality of reference macroblocks and the current macroblock X is completed within a predetermined search range, the panoramic video motion estimation unit 160 may select the current macroblock X from among the plurality of reference macroblocks. A reference macroblock most similar to is determined, and a motion vector representing the reference macroblock is generated (350).

  FIG. 10 shows the motion vector of the current macroblock X. As shown in FIG. 10, the reference macroblock on the padded reference video that is most similar to the current macroblock X510 is the reference macroblock 530, which corresponds to the reference macroblock 530, and is not padded. The macro block on the reference image 500 is a macro block 540. When the reference macroblock most similar to the current macroblock X510 is the reference macroblock 530, the reference number 550 represents the motion vector of the current macroblock X510. If the reference macroblock most similar to the current macroblock X510 is the reference macroblock 540, the reference number 560 represents the motion vector of the current macroblock X510. That is, the motion vector of the current macroblock X510 can be expressed by the above two. However, since a motion vector representing a macroblock that deviates from a predetermined search range for motion detection cannot be transmitted to the decoder side, the motion vector 550 representing the reference macroblock 530 is determined as the motion vector of the current macroblock X510.

  11A and 11B are flowcharts of a panoramic video motion estimation method targeted for sub-pixel search according to an embodiment of the present invention.

  The panorama video motion estimation unit 160 predicts a motion vector of the current data unit using motion vectors of a plurality of previous data units adjacent to the current data unit (1101). Such motion vector prediction is the same as the panoramic video motion estimation method for searching for positive pixels.

  The panoramic video motion estimation unit 160 adds a padding area to the basic input video, which is a basic reference video used for motion estimation of the current panoramic video, using the pixel values at the boundary (1102). That is, the left boundary pixel value is padded in the left padding area of the basic input video, and the right boundary pixel value is padded in the right padding area of the basic input video.

  Next, the panoramic video motion estimation unit 160 expands the basic input video with the padding area added in step 1102 and uses it as a reference video (1103). For example, in the case of 1/2 pixel, it is expanded by 2 times, and in the case of 1/4 pixel, it is expanded by 4 times.

  For example, when the current panoramic video size is 352 * 288 and a half-pixel search is performed, the current panoramic video size is doubled to (352 * 2) as shown in FIG. * (288 * 2). As shown in FIG. 13A, in the case of a reference video, if a padding area having a padding size 4 is added to the basic input video (352 * 288) for reference, the padded basic input video is displayed. The size is (4 + 352 + 4) * (4 + 288 + 4), and the basic input video padded in this way is doubled to a size of (4 + 352 + 4) * 2 * (4 + 288 + 4) * 2.

  The panoramic video motion estimation unit 160 predicts the motion vector of the current data unit, and then determines whether or not the pixel X of the reference data unit represented by the predicted motion vector belongs to the basic input video region from the reference video. (1104).

  If the pixel X of the reference data unit belongs to the basic input video area of the reference video, the pixel value of the pixel X is obtained as it is (1105).

  When the pixel X of the reference data unit belongs to the padding area of the reference image or outside the reference image, the x coordinate of the pixel X is set to one boundary of the basic reference image close to the sub pixel, the sub pixel, Is added to the x coordinate of the other boundary of the basic reference image or the subtracted value is set as the x coordinate of the sub-pixel to obtain the corresponding pixel value ( 1106). That is, when the pixel X belongs to the left padding region of the reference image or the outer region on the left side of the reference image, the x coordinate of the pixel X is set to one boundary of the basic reference image close to the sub pixel and the sub image. A value obtained by subtracting the distance on the x axis from the pixel from the x coordinate of the other boundary of the basic reference image is set as the x coordinate of the sub-pixel. When the pixel X belongs to the right padding region of the reference image or the outer region on the right side of the reference image, the x coordinate of the pixel X is set to one boundary of the basic reference image close to the sub pixel and the sub image. A value obtained by adding the distance on the x-axis to the pixel to the x coordinate of the other boundary of the basic reference image is set as the x coordinate of the sub-pixel.

  For example, as shown in FIG. 13B, the coordinates of the pixel A in the reference data unit are (30, 50) and belong to the basic input video area of the reference video, so if the pixel value of the pixel A is acquired as it is. Good.

  The coordinates of the pixel B in the reference data unit are (4, 50) and belong to the reference video padding area. The x coordinate of one boundary of the basic input image near the pixel B is 7, and the distance between the pixel B and one boundary of the basic input image is 3. Since the x coordinate of the other boundary of the basic input video area is 711 and 711-3 = 708, the x coordinate of the pixel B is set to 708. That is, the pixel value of the pixel B is obtained from the coordinates (708, 50) of the pixel B ′.

  The coordinates of the pixel C in the reference data unit are (−5, 100) and are located outside the reference image. The x coordinate of one boundary of the basic input image near the pixel C is 7, and the distance between the pixel C and one boundary of the basic input image is 12. Since the x coordinate of the other boundary of the basic input video area is 711 and 711-12 = 699, the x coordinate of the pixel C is set to 699. That is, the pixel value of the pixel C is obtained from the coordinates (699, 100) of the pixel C ′.

  As described above, in this embodiment, when the pixel value existing in the padding area is obtained, the pixel value existing in the padding area is a meaningless value. Therefore, the pixel value of the opposite boundary area is used. I have to. In other words, when the padding area is used in a conventional manner, the pixel values of the opposite boundary area must be used for both the pixels existing in the padding area and the pixels existing outside the reference image.

  Next, the panoramic image motion estimation unit 160 determines whether or not all pixel values of the reference data unit have been acquired (1107). When all the pixel values have been acquired, a predetermined evaluation function is used. The similarity between the current data unit and the reference data unit is evaluated (1108). If all the pixel values have not been acquired, the process proceeds to step 1104 to acquire the next pixel value. As the predetermined evaluation function, a SAD function, a SATD function, or an SSD function is generally used.

  Next, the panoramic video motion estimation unit 160 changes the position of the reference data unit within a predetermined search range, and evaluates the similarity between the changed reference data unit and the current data unit (1109, 1110). The panorama video motion estimation unit 160 may determine the most current data unit among the plurality of reference data units when the evaluation of the similarity between the plurality of reference data units and the current data unit is completed within a predetermined search range. A similar reference data unit is determined, and a motion vector representing the reference data unit is generated (1111).

  14A and 14B are flowcharts of a panoramic video motion estimation method targeted for sub-pixel search according to another embodiment of the present invention.

  The panoramic image motion estimation unit 160 predicts a motion vector of the current data unit using motion vectors of a plurality of previous data units adjacent to the current data unit (1401). Such motion vector prediction is the same as the panoramic video motion estimation method for searching for positive pixels.

  The panorama image motion estimation unit 160 performs padding using the pixel value of the right boundary region in the left padding region of the basic input image, and the pixel value of the left boundary region in the right padding region of the basic input image. Is used for padding (1402).

  Next, the panoramic video motion estimation unit 160 expands the basic input video with the padding area added in step 1402 and uses it as a reference video (1403). For example, in the case of 1/2 pixel, it is expanded by 2 times, and in the case of 1/4 pixel, it is expanded by 4 times.

  For example, when the current panoramic image size is 352 * 288 and the search for sub-pixels is performed, the current panoramic image size is doubled and, as shown in FIG. 12, (352 * 2) * (288 * 2). As shown in FIG. 15A, in the case of a reference video, if a padding area of padding size 4 is added to the basic input video for reference (352 * 288), the padded basic input video The size is (4 + 352 + 4) * (4 + 288 + 4), and the basic input video padded in this way is doubled to a size of (4 + 352 + 4) * 2 * (4 + 288 + 4) * 2. Of course, if the padding area is described here, the pixel value of the right boundary area is padded in the left padding area of the reference image, and the pixel value of the left boundary area is padded in the right padding area of the reference image. Is padded.

  After predicting the motion vector of the current data unit, the panoramic image motion estimation unit 160 determines whether or not the pixel B of the reference data unit represented by the predicted motion vector belongs to the reference image (1404).

  If the pixel X of the reference data unit belongs to the reference video, the pixel value of the pixel X is obtained as it is (1405).

  When the pixel X of the reference data unit is located outside the reference image, the x coordinate of the pixel X is set on the x-axis between one boundary of the basic reference image and the subpixel near the subpixel. Is set to the x coordinate of the pixel X to obtain the pixel value (1406). That is, when the pixel X belongs to the outer region on the left side of the reference image, the x coordinate of the pixel X is set to the x axis between one boundary of the basic reference image near the subpixel and the subpixel. A value obtained by subtracting the above distance from the x coordinate of the other boundary of the basic reference image is set as the x coordinate of the sub-pixel. If the pixel X belongs to the outer region on the right side of the reference image, the x coordinate of the pixel X is set to the x axis between one boundary of the basic reference image and the sub pixel near the sub pixel. A value obtained by adding the above distance to the x coordinate of the other boundary of the basic reference image is set as the x coordinate of the sub-pixel.

  For example, as shown in FIG. 15B, the reference data unit includes a pixel A having coordinates (30, 50), a pixel B having coordinates (4, 50), and a pixel D having coordinates (−5, 100). To do.

  Pixel A and pixel B belong to the reference image, and pixel C does not belong to the reference image and belongs to the outside of the reference image. Also, the pixel A belongs to the basic input video area in the reference video, and the pixel B belongs to the padding area. In the case of the pixel A, since it is a pixel belonging to the basic input video as a matter of course, the pixel value at that position may be obtained.

  In the case of the pixel B, the value belongs to the area padded with the pixel value of the right boundary area in consideration of the spatial correlation of the panoramic image when the reference image is generated. Alternatively, the pixel value at that position may be acquired.

  The coordinates of the pixel D in the reference data unit are (−5, 100) and are located outside the reference image. The x coordinate of one boundary of the basic input image near the pixel D is 7, and the distance between the pixel D and one boundary of the basic input image is 12. Since the x coordinate of the other boundary of the basic input video area is 711 and 711-12 = 699, the x coordinate of the pixel D is set to 699. That is, the pixel value of the pixel D is obtained from the coordinates (699, 100) of the pixel D ′. As described above, in this embodiment, when the pixel value existing in the padding area is obtained, the pixel value existing in the padding area is a value that has already been padded using the pixel value of the opposite boundary area. Therefore, it is sufficient to use it as it is, and it is only necessary to calculate and use the pixel value of the boundary region on the opposite side only for the pixels outside the reference video.

  Next, the panoramic image motion estimation unit 160 determines whether or not all pixel values of the reference data unit have been acquired (1407), and when all the pixel values have been acquired, a predetermined evaluation function is used. The degree of similarity between the current data unit and the reference data unit is evaluated (1408). If all the pixel values have not been acquired, the process proceeds to step 1404 to acquire the next pixel value. As the predetermined evaluation function, a SAD function, a SATD function, or an SSD function is generally used.

  Next, the panoramic image motion estimation unit 160 changes the position of the reference data unit within a predetermined search range, and evaluates the similarity between the changed reference data unit and the current data unit (1409, 1410). The panorama video motion estimation unit 160 may determine the most current data unit among the plurality of reference data units when the evaluation of the similarity between the plurality of reference data units and the current data unit is completed within a predetermined search range. A similar reference data unit is determined, and a motion vector representing the reference data unit is generated (1411).

  Hereinafter, a panoramic video motion compensation method and apparatus according to an embodiment of the present invention will be described.

  FIG. 16 is a block diagram of a panoramic video motion vector decoder according to an embodiment of the present invention. As shown in FIG. 16, a motion vector decoder for panoramic video according to an embodiment of the present invention includes a variable length decoder (VLD) 710, an inverse quantization unit 720, an inverse transform unit 730, and an addition unit. 740, a panoramic image motion compensation unit 750, a clipping unit 760, and a frame memory 770.

  The VLD 710 performs variable length decoding on the input bitstream. Of the outputs from the VLD 710, the motion vector is output to the panoramic video motion compensation unit 750, and the residual signal between the current macroblock and the reference macroblock is output to the inverse quantization unit 720.

  The frame memory 770 stores the reference video reproduced through the inverse quantization unit 720, the inverse transform unit 730, and the clipping unit 760. The reference video stored in the frame memory 770 is used for motion compensation of a newly input panoramic video.

  The panoramic video motion compensation unit 750 performs motion compensation according to the present invention using the reference panoramic video stored in the frame memory 770. The motion vector of the current macroblock transmitted from the panoramic video motion vector encoder as shown in FIG. 5 is input, and a reference macroblock corresponding to the current macroblock is read from the frame memory 770 and output to the adder 740. . Further, the adding unit 740 receives the residual signals of the current macro block and the reference macro block that have been inverse quantized and inverse transformed by the inverse quantizing unit 720 and the inverse transform unit 730, respectively.

  The adder 740 receives the residual signal between the current macroblock and the reference macroblock and the reference macroblock from the panoramic video motion compensator 750 and reproduces the current macroblock. The clipping unit 760 performs a function of normalizing the reproduced current macroblock output from the adding unit 740.

  Hereinafter, the operation of the panoramic image motion compensation unit 750 will be described in more detail with reference to the drawings. FIG. 17 is a flowchart of a panoramic video motion compensation method according to an embodiment of the present invention.

  The panorama video motion compensation unit 750 receives the motion vector of the current data unit for which motion compensation is performed from the VLD 710 (910). In this embodiment, the data unit is a 16 × 16 macroblock.

  The panoramic image motion compensation unit 750 determines whether a reference macroblock represented by the current motion vector of the macroblock exists in the reference image (920). The reference video is stored in the frame memory 770.

  In the case of the positive pixel search, if all the pixels of the reference macroblock represented by the motion vector of the current macroblock are present in the reference video, all the pixel values of the reference macroblock are acquired from the frame memory 770 ( 950), the current macroblock is played back (960).

  The adder 740 receives the residual signal between the current macroblock and the reference macroblock output from the inverse converter 730 and the reference macroblock output from the panoramic video motion compensator 750 and receives the current macro. Play the block.

  However, some or all of the pixels of the reference macroblock represented by the motion vector of the current macroblock are present outside one of the left and right boundaries of the reference image, as shown in FIG. 8A or 8B. For example, the video from one other boundary to the predetermined range of the reference video is padded outside the one boundary (930). In the present invention, as shown in FIG. 9, the outer region of the reference image is utilized by utilizing the very high nature of the spatial correlation between the right and left boundary portions of the panoramic image including 360 ° omnidirectional images. Padding.

  The panoramic video motion compensation unit 750, after padding the reference video in step 930, acquires all the pixel values of the reference macroblock from the padded reference video from the frame memory 770 (940).

  On the other hand, without performing the padding process as described above, by connecting the left boundary and the right boundary of the reference image, and assuming that the reference image is a cylindrical image, from the cylindrical image All pixel values of the reference data unit may be obtained. More specifically, the reference image is a two-dimensional plane image as shown in FIG. 4B. If the left boundary and the right boundary of such a two-dimensional planar image are connected, a cylindrical image as shown in FIG. 4A is obtained. Assuming that the reference image is a cylindrical image, all pixel values of the reference data unit are obtained from the cylindrical image.

  Finally, the addition unit 740 receives the residual signal between the current macroblock and the reference macroblock and the reference macroblock output from the panoramic video motion compensation unit 750 and reproduces the current macroblock (960). ).

  In the case of half-pixel search, according to an embodiment of the present invention, the left padding region of the basic reference image used for motion compensation of the current panoramic image is padded using the right boundary region, In the right padding area of the basic reference video, the reference video generated by extending the padded video after padding using the left boundary area is stored in the frame memory, and the input motion When a sub-pixel that is included in the reference data unit represented by the vector and is a motion search target belongs to the reference video, the pixel value of the sub-pixel is directly used for motion compensation, and the basic reference video near the sub-pixel is used. The distance on the x-axis between one boundary of the sub-pixel and the sub-pixel is added to the x-coordinate of the other boundary of the basic reference image, or the subtracted value is the x-coordinate value of the sub-pixel Set, obtain the pixel values.

  In the case of half pixel search, according to another embodiment of the present invention, the left padding region of the basic reference image used for motion compensation of the current panoramic image is padded using the left boundary pixel value, In the right padding area of the basic reference video, the right border pixel value is used for padding, and the reference video generated by expanding the padded video is saved in the frame memory, and the current data When a sub-pixel that is included in the reference data unit represented by the motion vector of the unit and is a motion search target belongs to the basic input video area, the pixel value of that pixel is obtained as it is, and the sub-pixel acquires the reference video pack. A distance on the x-axis between one boundary of the basic reference image near the sub-pixel and the sub-pixel. Or added to the x coordinate of the other boundary of the reference image, or the subtracted value is set to x-coordinate value of the sub-pixels to obtain a pixel value.

  In the above description, the pixel value of the right boundary area is mainly used as the pixel value of the right area using the pixel value of the left boundary area of the reference image, and the pixel value of the left area is used as the pixel value of the right boundary area of the reference image. As described above, only the horizontal direction of the reference video has been mainly explained, but this is explained in the horizontal direction because of the high spatial correlation between the left and right boundaries due to the characteristics of the panoramic video. Is. However, such a method according to the present invention can be applied not only in the horizontal direction but also in the vertical direction if the spatial correlation between the upper and lower boundary regions of the image is high. A person skilled in the art will fully understand.

  The present invention can also be embodied as computer readable code on a computer readable recording medium. Computer-readable recording media include all types of recording devices that can store data which can be read by a computer system. Examples of the computer-readable recording medium include a ROM (Read Only Memory), a RAM (Random Access Memory), a CD-ROM, a magnetic tape, a flexible disk, an optical data storage device, and a carrier wave, for example, And those embodied in the form of transmission over the Internet. The computer-readable recording medium can be distributed in a computer system connected to a network, stored as a computer-readable code in a distributed manner, and executed.

  In the above, this invention was demonstrated centering on the desirable embodiment. Those skilled in the art will appreciate that the present invention may be embodied in variations that do not depart from the essential characteristics of the invention. The scope of the present invention is described not in the above description but in the claims, and all differences within the equivalent scope should be construed as being included in the present invention.

2 is a diagram illustrating a plurality of previous macroblocks used for predicting a motion vector of a current macroblock according to the prior art. 6 is a diagram illustrating a case where there is no previous macroblock used for prediction of a motion vector of a current macroblock. 6 is a diagram illustrating a case where there is no previous macroblock used for prediction of a motion vector of a current macroblock. 6 is a diagram illustrating a case where there is no previous macroblock used for prediction of a motion vector of a current macroblock. 6 is a diagram illustrating a reference video padding method according to a conventional technique. It is a drawing showing a cylindrical image including a 360 ° omnidirectional image. It is drawing which shows the two-dimensional panoramic image corresponding to the cylindrical image | video shown to FIG. 4A. It is a block diagram of a motion vector encoder for panoramic video according to an embodiment of the present invention. 5 is a flowchart of a panoramic video motion estimation method according to an embodiment of the present invention. 5 is a flowchart of a panoramic video motion estimation method according to an embodiment of the present invention. It is a figure which shows an example of the previous macroblock selection utilized for the prediction of the motion vector of the present macroblock. It is a figure which shows another example of the previous macroblock selection utilized for the prediction of the motion vector of the present macroblock. 6 is a diagram illustrating a case where a reference macroblock exists across a reference video boundary. 6 is a diagram illustrating a case where a reference macroblock exists in an outer region of a reference video. 3 is a diagram illustrating a reference video padding method according to an exemplary embodiment of the present invention. 3 is a diagram illustrating a motion vector of a current macroblock X. 5 is a flowchart illustrating an embodiment of a half-pixel motion search method according to the present invention. 5 is a flowchart illustrating an embodiment of a half-pixel motion search method according to the present invention. 12 is a diagram showing an example of a basic input frame for explaining a process of the motion search method shown in FIGS. 11A and 11B. FIG. 11A and 11B are diagrams illustrating an example of a reference frame in which a padding region is added to the basic input frame illustrated in FIG. 12 and enlarged twice to describe one embodiment of FIG. 11A and FIG. 11B. 11A and 11B are reference diagrams for explaining processing of pixel values belonging to a padding region in a reference frame enlarged twice according to an embodiment of FIGS. 11A and 11B. 6 is a flowchart illustrating another embodiment of the process of the half-pixel motion search method according to the present invention. 6 is a flowchart illustrating another embodiment of the process of the half-pixel motion search method according to the present invention. 14A and 14B are diagrams illustrating an example of a reference frame in which a padding region is added to the basic input frame illustrated in FIG. 12 and enlarged twice to describe another embodiment of FIGS. 14A and 14B. 14A and 14B are reference diagrams for explaining processing of a pixel value belonging to a padding region and a pixel value positioned outside the reference frame in a reference frame that is doubled according to another embodiment of FIGS. 14A and 14B. It is a block diagram of a motion vector decoder for panoramic video according to an embodiment of the present invention. 3 is a flowchart of a panoramic video motion compensation method according to an embodiment of the present invention.

Explanation of symbols

DESCRIPTION OF SYMBOLS 110 Conversion part 115 Quantization part 120 Inverse quantization part 125 Inverse conversion part 130 Adder part 140 Clipping part 150 Frame memory 160 Panoramic video motion estimation part 170 Panoramic video motion compensation part 180 Subtraction part 190 Variable length encoding part 710 Variable length decoding Conversion unit 720 inverse quantization unit 730 inverse conversion unit 740 addition unit 750 panoramic image motion compensation unit 760 clipping unit 770 frame memory

Claims (13)

In a panoramic video motion estimation method including 360 ° omnidirectional video information,
The left padding area adjacent to the left outer side of the basic reference video for motion estimation of the current panoramic video is padded using the right boundary area of the basic reference video, and the right outer side of the basic reference video. A padding area adjacent to the right padding area using the left boundary area of the basic reference video, and then extending the padded video to set it as a reference video; and
Predicting a motion vector of the current data unit using motion vectors of a plurality of previous data units adjacent to the current data unit included in the current panoramic image;
When a sub-pixel included in the reference data unit represented by the predicted motion vector belongs to the reference video, the pixel value of the sub-pixel is obtained as it is, and the sub-pixel is located on the right outside of the reference video. , The distance on the x-axis between the right padding area near the sub-pixel and the boundary of the basic reference image and the sub-pixel is expressed as x of the boundary between the left padding area and the basic reference image. If the sub-pixel is added to the coordinates, or if the sub-pixel is located outside the left side of the reference image, an x between the left padding area near the sub-pixel and the boundary of the basic reference image and the sub-pixel A value obtained by subtracting the distance on the axis from the x coordinate of the boundary between the right padding region and the basic reference image is set to the x coordinate of the sub pixel, and the pixel value of the sub pixel is obtained. ,in front A step of the reference image to obtain all pixel values of the reference data unit,
Determining a similarity between the current data unit and the reference data unit using a predetermined evaluation function;
Determining a reference data unit most similar to the current data unit within a predetermined search range;
Determining a motion vector representing the determined reference data unit;
A motion estimation method comprising: Predicting a motion vector of the current data unit comprises:
If at least one of the plurality of previous data units is present outside the left boundary of the current panoramic image, the previous data unit adjacent to the right boundary on the same Y axis of the current panoramic image Or at least one of the plurality of previous data units is adjacent to the outside of the right boundary of the current panoramic image, and the left boundary on the same y-axis of the current panoramic image. The motion estimation of claim 1, comprising determining the plurality of previous data units to determine a motion vector of the current data unit using adjacent previous data units. Method. In a panoramic video motion estimation method including 360 ° omnidirectional video information,
The left padding area adjacent to the left outer side of the basic reference video for motion estimation of the current panoramic video is padded using the left boundary area of the basic reference video, and the right outer side of the basic reference video. A padding area adjacent to the right padding area using the right boundary area of the basic reference video, and then extending the padded video to set it as a reference video; and
Predicting a motion vector of the current data unit using motion vectors of a plurality of previous data units adjacent to the current data unit included in the current panoramic image;
If a sub-pixel included in the reference data unit represented by the predicted motion vector belongs to the basic reference video, the sub-pixel acquires the pixel value of the sub-pixel as it is, and the sub-pixel receives the right padding of the reference video. When the region is located on the right outside of the reference image, the distance on the x-axis between the right padding region near the sub-pixel and the boundary of the basic reference image and the sub-pixel, When the left pixel is added to the x coordinate of the boundary between the left padding region and the basic reference image, or the subpixel is located outside the left padding region or the left side of the reference image, the left padding near the subpixel. The distance on the x-axis between the border between the padding area and the basic reference image and the sub-pixel is subtracted from the x coordinate of the boundary between the right padding area and the basic reference image. And is set to x-coordinate value of the sub-pixels, by acquiring pixel values of the sub-pixels, and obtaining all the pixel values of the reference data unit from the reference image,
Determining a similarity between the current data unit and the reference data unit using a predetermined evaluation function;
Determining a reference data unit most similar to the current data unit within a predetermined search range;
Determining a motion vector representing the determined reference data unit;
A motion estimation method comprising: Predicting a motion vector of the current data unit comprises:
If at least one of the plurality of previous data units is present outside the left boundary of the current panoramic image, the previous data unit adjacent to the right boundary on the same y-axis of the current panoramic image Or at least one of the plurality of previous data units is adjacent to the outside of the right boundary of the current panoramic image, and the left boundary on the same y-axis of the current panoramic image. 4. The motion estimation of claim 3 , comprising determining the plurality of previous data units to determine a motion vector of the current data unit utilizing adjacent previous data units. Method. In a panoramic video motion estimation device including 360 ° omnidirectional video information,
The left padding area adjacent to the left outer side of the basic reference video for motion estimation of the current panoramic video is padded using the right boundary area of the basic reference video, and the right outer side of the basic reference video. The right padding area adjacent to the reference video is padded using the left boundary area of the basic reference video, and then included in the current panoramic video and the reference video generated by extending the padded video. A memory for storing motion vectors of a plurality of previous data units adjacent to the current data unit
A motion vector of the current data unit is predicted using the motion vectors of the plurality of previous data units, and a sub-pixel included in a reference data unit represented by the predicted motion vector belongs to the reference image If the pixel value of the sub-pixel is obtained as it is and the sub-pixel is located on the right outer side of the reference image, the right padding area near the sub-pixel and the boundary of the basic reference image, The distance on the x axis between the sub-pixel and the x-coordinate of the boundary between the left padding region and the basic reference image is added, or when the sub-pixel is located outside the left side of the reference image, The distance between the left padding area near the sub-pixel and the basic reference video and the distance between the sub-pixel and the sub-pixel is defined as the right padding area and the basic reference video. All the pixel values of the reference data unit were obtained from the reference image by setting the value subtracted from the x coordinate of the boundary of the subpixel to the x coordinate of the subpixel and obtaining the pixel value of the subpixel. Later, a similarity between the current data unit and the reference data unit is determined using a predetermined evaluation function, and a reference data unit most similar to the current data unit within a predetermined search range is determined. A motion estimator that determines a motion vector representing the determined reference data unit . In a panoramic video motion estimation device including 360 ° omnidirectional video information,
The left reference padding area adjacent to the left outer side of the basic reference image for motion estimation of the current panoramic image is padded using the left boundary pixel value of the basic reference image, and the basic reference The right padding area adjacent to the right outer side of the video is padded using the right boundary pixel value of the basic reference video, and then the reference video generated by extending the padded video and the current video A memory for storing motion vectors of a plurality of previous data units adjacent to the current data unit included in the panoramic image;
A motion vector of the current data unit is predicted using the motion vectors of the plurality of previous data units, and a sub-pixel included in a reference data unit represented by the predicted motion vector is the reference image of the reference image. If it belongs to the basic reference video, the pixel value of the sub-pixel is obtained as it is, and when the sub-pixel is located on the right padding area of the reference video or on the right outer side of the reference video, the sub-pixel is close to the sub-pixel. The x-axis distance between the right padding area and the basic reference video boundary and the subpixel is added to the x coordinate of the left padding area and the basic reference video boundary, or the subpixel Is located on the left padding region or the left outer side of the reference image, the left padding region near the sub-pixel and the basic reference image A value obtained by subtracting the distance on the x-axis between the boundary and the sub-pixel from the x-coordinate of the boundary between the right padding area and the basic reference image is set as the x-coordinate value of the sub-pixel. , Obtaining all pixel values of the reference data unit from the reference image by obtaining the pixel values of the sub-pixels, and then using a predetermined evaluation function to determine the current data unit and the reference data unit. A motion estimation unit that determines a similarity, determines a reference data unit that is most similar to the current data unit within a predetermined search range, and determines a motion vector that represents the determined reference data unit. A motion estimation apparatus characterized by the above. The motion estimator is
If at least one of the plurality of previous data units is present outside the left boundary of the current panoramic image, the previous data unit adjacent to the right boundary on the same y-axis of the current panoramic image Or at least one of the plurality of previous data units is adjacent to the outside of the right boundary of the current panoramic image, and the left boundary on the same y-axis of the current panoramic image. The motion estimation apparatus according to claim 6 , wherein the previous data units for determining a motion vector of the current data unit are determined using adjacent previous data units. In a panoramic video motion compensation method including 360 ° omnidirectional video information,
The left padding area adjacent to the left outer side of the basic reference video for motion compensation of the current panoramic video is padded using the right boundary area of the basic reference video, and the right outer side of the basic reference video. A padding area adjacent to the right padding area using the left boundary area of the basic reference video, and then expanding the padded video to generate a reference video; and
A step of inputting a motion vector of a current data unit included in a current panoramic image transmitted by a transmission side encoder from a decoding unit ;
A step of inputting, from the decoding unit, a residual signal transmitted by the transmission side encoder between the current data unit and a reference data unit represented by a motion vector of the current data unit;
When a sub-pixel included in a reference data unit represented by a motion vector of the current data unit belongs to the reference video, the pixel value of the sub-pixel is acquired as it is, and the sub-pixel is positioned outside the right side of the reference video. In this case, the distance on the x-axis between the right padding area near the subpixel, the boundary of the basic reference image, and the subpixel is defined as the distance between the left padding area and the basic reference image. When added to the x coordinate of the boundary, or when the sub-pixel is located outside the left side of the reference image, the boundary between the left padding area near the sub-pixel and the basic reference image, and the sub-pixel The value obtained by subtracting the distance on the x-axis between the right padding area and the x coordinate of the boundary of the basic reference image is set as the x coordinate of the sub pixel, and the pixel value of the sub pixel is obtained. To do By the steps of obtaining all the pixel values of the reference data unit from the reference image,
Adding the pixel value of the reference data unit and the residual signal to reproduce the current data unit. In a panoramic video motion compensation method including 360 ° omnidirectional video information,
The left padding area adjacent to the left outer side of the basic reference image for motion compensation of the current panoramic image is padded using the left boundary pixel value of the basic reference image, and the right side of the basic reference image The right padding area adjacent to the outside is padded using the right boundary pixel value of the basic reference video, and then the padded video is expanded to generate a reference video; and
A step of inputting a motion vector of a current data unit included in a current panoramic image transmitted by a transmission side encoder from a decoding unit ;
A step of inputting, from the decoding unit, a residual signal transmitted by the transmission side encoder between the current data unit and a reference data unit represented by a motion vector of the current data unit;
When a sub-pixel included in a reference data unit represented by a motion vector of the current data unit belongs to the basic reference video, the pixel value of the sub-pixel is obtained as it is, and the sub-pixel is acquired in the right padding area or the When located on the right outer side of the reference image, the distance on the x-axis between the right padding area near the sub-pixel and the boundary of the basic reference image and the sub-pixel is defined as the left-hand padding. The left padding area near the subpixel when the subpixel is located on the left padding area or the left outer side of the reference video. A value obtained by subtracting the distance on the x axis between the boundary of the basic reference image and the sub-pixel from the x coordinate of the boundary between the right padding area and the basic reference image , Is set to x-coordinate value of the sub-pixels, by acquiring pixel values of the sub-pixels, and obtaining all the pixel values of the reference data unit from the reference image,
Adding the pixel value of the reference data unit and the residual signal to reproduce the current data unit. In a motion compensation device for panoramic video including 360 ° omnidirectional video information,
The left padding area adjacent to the left outer side of the basic reference video for motion compensation of the current panoramic video is padded using the right boundary area of the basic reference video, and the right outer side of the basic reference video. In the right padding area adjacent to the memory, the padding is performed using the left boundary area of the basic reference video, and then the reference video generated by expanding the padded video is stored.
Sent by the sending encoder, a motion vector of the current data unit, wherein included in the current panoramic image, Rutotomoni inputted from the decoding unit, transmitted by the transmitting-side encoder, said the current data unit The residual signal from the reference data unit represented by the motion vector of the current data unit is input from the decoding unit, and the sub-pixel included in the reference data unit represented by the input motion vector is the reference video. If the pixel value of the sub-pixel is obtained as it is and the sub-pixel is located outside the right side of the reference image, the right padding area near the sub-pixel and the boundary of the basic reference image Whether the distance on the x-axis between the sub-pixel and the sub-pixel is added to the x-coordinate of the boundary between the left padding area and the basic reference image When the sub-pixel is located outside the left side of the reference image, a distance on the x-axis between the left padding region near the sub-pixel and the boundary of the basic reference image and the sub-pixel A value obtained by subtracting from the x coordinate of the boundary between the right padding area and the basic reference image is set to the x coordinate of the sub pixel, and the pixel value of the sub pixel is obtained, thereby obtaining the pixel value from the reference image. A motion compensation unit that obtains all the pixel values of the reference data unit and then adds the pixel value of the reference data unit and the residual signal to reproduce the current data unit. Motion compensation device. In a motion compensation device for panoramic video including 360 ° omnidirectional video information,
The left padding area adjacent to the left outer side of the basic reference image for motion compensation of the current panoramic image is padded using the left boundary pixel value of the basic reference image, and the right side of the basic reference image In the right padding area adjacent to the outside, after padding using the right boundary pixel value of the basic reference video, a memory for storing the reference video generated by extending the padded video, and
Sent by the sending encoder, a motion vector of the current data unit, wherein included in the current panoramic image, Rutotomoni inputted from the decoding unit, transmitted by the transmitting-side encoder, said the current data unit The residual signal with the reference data unit represented by the motion vector of the current data unit is input from the decoding unit, and the sub-pixel included in the reference data unit represented by the motion vector of the current data unit is the If it belongs to the basic reference picture, the pixel value of the sub-pixel is obtained as it is, and if the sub-pixel is located on the right padding area or the right outside of the reference picture, the right padding near the sub-pixel The distance on the x-axis between the region and the boundary of the basic reference image and the sub-pixel is defined as the left padding region. When the sub-pixel is located outside the left-side padding region or the left-side reference image, the left-side padding region near the sub-pixel and the sub-pixel The value obtained by subtracting the distance on the x-axis between the boundary of the basic reference image and the sub-pixel from the x-coordinate of the boundary between the right padding region and the basic reference image is the x-coordinate value of the sub-pixel. After obtaining all pixel values of the reference data unit in the reference video by acquiring the pixel value of the sub-pixel, and adding the pixel value of the reference data unit and the residual signal And a motion compensation unit that reproduces the current data unit. In order to perform a panoramic video motion estimation method including 360 ° omnidirectional video information, in a computer-readable recording medium recorded with a program including an instruction word adapted to control a computer, the program includes:
The left padding area adjacent to the left outer side of the basic reference video for motion estimation of the current panoramic video is padded using the right boundary area of the basic reference video, and the right outer side of the basic reference video. A padding area adjacent to the right padding area using the left boundary area of the basic reference video, and then extending the padded video to set it as a reference video; and
Predicting a motion vector of the current data unit using motion vectors of a plurality of previous data units adjacent to the current data unit included in the current panoramic image;
When a sub-pixel included in the reference data unit represented by the predicted motion vector belongs to the reference video, the pixel value of the sub-pixel is obtained as it is, and the sub-pixel is located on the right outside of the reference video. The distance between the right padding area near the subpixel and the basic reference image and the distance between the subpixel and the subpixel on the X axis is the boundary between the left padding area and the basic reference image. When the sub-pixel is added to the x-coordinate or the sub-pixel is located outside the left side of the reference image, the boundary between the left padding area near the sub-pixel and the basic reference image and the sub-pixel A value obtained by subtracting the distance on the x-axis from the x coordinate of the boundary between the right padding area and the basic reference image is set as the x coordinate of the sub pixel to obtain the pixel value of the sub pixel. By A step of the serial reference image to obtain all pixel values of the reference data unit,
Determining a similarity between the current data unit and the reference data unit using a predetermined evaluation function;
Determining a reference data unit most similar to the current data unit within a predetermined search range;
Determining a motion vector representing the determined reference data unit;
A computer-readable recording medium characterized by causing a computer to execute. The program is
In the step of predicting a motion vector of the current data unit, if at least one of the plurality of previous data units exists adjacent to the outside of a left boundary of the current panoramic image, the current panoramic image If a previous data unit adjacent to the right boundary on the same Y-axis is used, or if at least one of the plurality of previous data units exists outside the right boundary of the current panoramic image Determining the plurality of previous data units for determining a motion vector of the current data unit using a previous data unit adjacent to a left boundary on the same Y axis of the current panoramic image; readable recording a computer according to claim 12, characterized by causing a computer to execute the Body.