JP2022068983A - Video processing device and program thereof - Google Patents

Abstract

To provide a video processing device which can perform interpolation operation at a high speed to convert a video.SOLUTION: A video processing device 1 comprises: memory block generation means 12 which divides a frame image constituting an input video into memory blocks each of which includes the required number of pixels for interpolation operation of one pixel after conversion and has predetermined sizes in a horizontal direction and a vertical direction respectively; memory block storage means 13 which consists of a memory allowing for pixel-by-pixel readout and in which the memory blocks are stored; and interpolation means which reads out pixel values of peripheral pixels of a pre-conversion pixel position corresponding to a post-conversion pixel position from the memory blocks and calculates a pixel value of the post-conversion pixel position by interpolation operation.SELECTED DRAWING: Figure 1

Description

The present invention relates to a video processing device for generating virtual reality video and a program thereof.

In recent years, a virtual reality (VR) service that allows a user to experience a remote location as if he / she is there by using a head-mounted display (hereinafter, HMD), a dome-shaped display, or the like has become widespread.
Conventionally, for live circular celestial motion images for VR, multiple cameras using wide-angle lenses (fisheye lenses) are arranged, and the images taken by each camera are equirectangular projection (wide-angle image) for each frame image (wide-angle image). It is generated as a flat image by expanding the plane with an equirectangular (equirectangular) and then joining (stitching) the same region on a pixel-by-pixel basis.
In the equirectangular projection, the converted pixels are interpolated using the unconverted pixels. As a method of this interpolation, a method of performing bilinear interpolation or the like using a plurality of pixels before conversion is disclosed (see Patent Document 1).
Then, the HMD or the like displays the VR image by inversely converting the planar image to the entire 360 degrees of one celestial sphere and mapping it.

Japanese Unexamined Patent Publication No. 2015-215817

When converting an image into an equirectangular projection using equirectangular projection, it is necessary to read a plurality of pixels before conversion from the input image when bilinear interpolation or the like is used as in the conventional case.
In particular, as a memory for storing an input image, a DRAM that is read out in units of one line is generally used. For example, in order to perform bilinear interpolation, when calculating the pixel value of one pixel after conversion, one line is read out for each pixel in order to refer to the pixel value of four pixels (2 × 2 pixels) from the DRAM. Is required.
Therefore, there is a problem that the conventional method cannot perform the interpolation calculation at high speed.

The present invention has been made in view of such problems, and provides a video processing apparatus and a program thereof capable of reducing access to DRAM and performing video conversion processing accompanied by interpolation at high speed. Is the subject.

In order to solve the above problems, the video processing apparatus according to the present invention is a video processing apparatus that converts an input video into an output video by a conversion process accompanied by interpolation, and is a video input means, an input image storage means, and a memory. The configuration includes a block generation means, a memory block storage means, an interpolation means, an output image storage means, and a video output means.

In such a configuration, the video processing apparatus inputs video by the video input means, and sequentially stores the frame images constituting the input video in the input image storage means.
Then, the video processing device uses the memory block generation means to put the frame image in the memory blocks having predetermined sizes in the horizontal direction and the vertical direction, which are larger than the number of pixels required for the interpolation calculation of one pixel after the conversion by the conversion process. To divide.

Then, the memory block generation means stores the generated memory block in the memory block storage means. This memory block storage means is composed of a memory that can be read in pixel units such as a block RAM.

Then, the video processing device reads the pixel values of the peripheral pixels of the pixel position before conversion corresponding to the pixel position after conversion from the memory block by the interpolation means, and calculates the pixel value of the pixel position after conversion by the interpolation calculation. do. At this time, the interpolation means is required at high speed by reading the pixels required for the interpolation calculation from the memory block that can be read in pixel units instead of the frame image stored in the input image storage means that performs line reading. The pixel value can be acquired.

Then, the image processing apparatus stores the pixel value of the pixel position after conversion in the output image storage means as the pixel value of the corresponding pixel position of the frame image constituting the output image.
Then, the image processing device outputs an image in which the image stored in the output image storage means is used as a frame image by the image output means.

The video processing device can be operated by a video processing program for operating the computer as each of the above-mentioned means.

The present invention has the following excellent effects.
According to the present invention, since the pixels required for the interpolation calculation are stored as one memory block in the memory accessible in pixel units and the interpolation calculation is performed, frequent reading to the DRAM is suppressed and the speed is higher than before. Image conversion can be performed.
Thereby, the present invention can realize the creation of a real-time VR image in VR.

It is a functional block diagram which shows the structure of the image processing apparatus which concerns on embodiment of this invention. It is a pixel arrangement diagram which shows an example of the pixel arrangement of the input image (fish-eye image). It is a pixel arrangement diagram which shows the structural example of a memory block. It is explanatory drawing for demonstrating the corresponding pixel before and after conversion, (a) shows the pixel position of a fisheye image, (b) shows the pixel position of an equirectangular image. It is explanatory drawing which shows the pixel position before conversion and the peripheral pixel position for demonstrating bilinear interpolation. It is a flowchart which shows the operation of the image processing apparatus which concerns on embodiment of this invention. It is explanatory drawing for demonstrating the modification of the memory block, (a) corresponds to the pixel arrangement figure of YCbCr4: 2: 2 in the input image (fisheye image), and (b) corresponds to YCbCr4: 2: 2. It is a pixel arrangement diagram of a memory block.

<Configuration of video processing equipment>
First, the configuration of the video processing apparatus 1 according to the embodiment of the present invention will be described with reference to FIG.

The image processing device 1 converts an input image into an output image by a conversion process accompanied by interpolation.
Here, the image processing device 1 converts an image taken by a wide-angle camera (fish-eye camera) into an equirectangular projection format (equirectangular format) image as an image conversion process accompanied by interpolation. do.
As shown in FIG. 1, the video processing apparatus 1 includes a video input means 10, an input image storage means 11, a memory block generation means 12, a memory block storage means 13, an interpolation means 14, and an output image storage means 15. And the video output means 16.

The video input means 10 inputs video. Here, the image input means 10 inputs an image composed of the fisheye image G1 taken by the wide-angle camera.
The video input means 10 sequentially writes and stores the fisheye image G1, which is a frame of the input video, in the input image storage means 11. Further, the video input means 10 notifies the memory block generation means 12 of the end of writing when the writing of one frame to the input image storage means 11 is completed.

The input image storage means 11 stores a frame image constituting the input video. Here, the input image storage means 11 stores the fisheye image G1 input by the video input means 10.
The input image storage means 11 can be configured with a semiconductor memory. For example, the input image storage means 11 can be configured with a DRAM (Dynamic Random Access Memory) that is generally used as an image memory.
The fisheye image G1 stored in the input image storage means 11 is read out by the memory block generation means 12.

The memory block generation means 12 has predetermined sizes of the frame image stored in the input image storage means 11 in the horizontal direction and the vertical direction, which are larger than the number of pixels required for the interpolation calculation of one pixel after conversion by the conversion process. It is divided into memory blocks of.
This memory block has a size in which at least the pixels required for the interpolation calculation by the interpolation means 14 described later, for example, 4 pixels (2 × 2 pixels) in the case of the bilinear interpolation calculation, are included in one block.

Further, in the memory block, the pixels required for the interpolation calculation of one pixel always exist in one memory block, and the adjacent memory blocks so that all the pixels in the memory block correspond to the pixels of the fisheye image G1. It is assumed that an overlapping area is included with and.
For example, the memory block generating means 12 divides the entire fisheye image G1 by including two or more overlapping pixels in units of 64 horizontal pixels and 32 vertical pixels.

Here, a specific example of the memory block will be described with reference to FIGS. 2 and 3.
FIG. 2 is a pixel arrangement diagram showing an example of the pixel arrangement of the input image (fisheye image G1). Here, the fisheye image G1 is a full HD (High Definition) horizontal 1920 image and a vertical 1080 pixel image. Of course, the fisheye image G1 is not limited to full HD, and may be an image of HD, 4K, 8K, or the like. In FIG. 2, for convenience of explanation, the pixel positions are designated by reference numerals such as P0, P1, ....
FIG. 3 is a pixel arrangement diagram showing a configuration example of the memory blocks B0, B1, ... By dividing the image of FIG. 2. Here, an example in which the memory block has 64 pixels horizontally and 32 pixels vertically is shown.

For example, the memory block B0 is composed of pixels P0 to P63, P1920 to P19833, ..., P59520 to P59583 of the fisheye image G1 of FIG.
Note that FIG. 3 shows an example in which the fisheye image G1 is simply divided in units of 64 pixels × 32 pixels in order to facilitate understanding of the drawings, but overlapping pixels are shown at the boundaries of each memory block. Is included. For example, the memory block B1 is P62, P63, ..., P125, P1982, P19833 ..., P2945, P59582, P59583, ..., P59645 so as to overlap the horizontally adjacent memory blocks B0 by two pixels. The same applies to memory blocks that are adjacent in the vertical direction.
Returning to FIG. 1, the description of the configuration of the video processing apparatus 1 will be continued.

The memory block generation means 12 reads a pixel value of the number of lines (32 lines) corresponding to the number of pixels in the vertical direction of the memory block (here, 32 pixels) from the input image storage means 11 for each line, and of the memory block. Each number of pixels in the horizontal direction (64 pixels in this case) is written as an individual memory block of the memory block storage means 13.
After the reading of the 32 lines is completed, the memory block generating means 12 reads the next 32 lines according to the instruction of the interpolation means 14, and updates the memory block.
The memory block generation means 12 notifies the interpolation means 14 of the end of writing when the memory block storage means 13 has finished writing the memory block.

The memory block storage means 13 stores the memory blocks generated by the memory block generation means 12.
The memory block storage means 13 is a memory that can be accessed in pixel units, and is composed of, for example, a block RAM of an FPGA (Field Programmable Gate Array).
The memory block storage means 13 arranges a plurality of block RAMs, for example, block RAMs for the number of blocks in the horizontal direction shown in FIG. 3 (in this case, 30 blocks from B0 to B29), and sequentially updates the memory blocks. To use.

When the output port of the block RAM is "2", only the pixel value of two pixels can be read from the memory block at a time. In that case, the memory block generation means 12 writes the same memory block twice in the memory block storage means 13, and the interpolation means 14 reads out the pixel values of a total of 4 pixels, 2 pixels from each memory block. And.

The interpolation means 14 reads the pixel values of the peripheral pixels of the pixel position before conversion corresponding to the pixel position after conversion from the memory block, and calculates the pixel value of the pixel position after conversion by interpolation calculation. Here, the interpolation means 14 is generated by interpolating the pixel values of the equirectangular image with reference to the memory blocks stored in the memory block storage means 13.
The interpolation means 14 includes a pre-conversion coordinate calculation means 140 and an interpolation pixel value calculation means 141.

The pre-conversion coordinate calculation means 140 calculates the pre-conversion pixel position (coordinates) corresponding to the post-conversion pixel position. Here, the pre-conversion coordinate calculation means 140 calculates the positions (coordinates) of the pixels of the fish-eye image before conversion corresponding to the pixels of the equirectangular image after conversion.
That is, as shown in FIG. 4, the pre-conversion coordinate calculation means 140 sets the pixel position (coordinates p) of the equirectangular image G2 when the fisheye image G1 of (a) is converted into the equirectangular image G2. The pixel position (coordinates p') of the corresponding fisheye image G1 is calculated.

The conversion from the fisheye image G1 to the regular distance cylindrical image G2 may be performed by using a general equirectangular transformation, and the pre-conversion coordinate calculation means 140 is positive by performing the inverse conversion of the equirectangular transformation. The pixel position (coordinates p') of the fisheye image G1 before conversion corresponding to the pixel position (coordinates p) of the distance cylindrical image G2 is calculated.
The pre-conversion coordinate calculation means 140 outputs the pixel position (coordinates p) after conversion and the pixel position (coordinates p') before conversion to the interpolated pixel value calculation means 141.

The interpolation pixel value calculation means 141 calculates the pixel value of the pixel position before conversion calculated by the pre-conversion coordinate calculation means 140 by the interpolation calculation with the pixel values of the peripheral pixels of the pixel.
Here, the interpolation pixel value calculation means 141 calculates the pixel value by the bilinear interpolation calculation.
The interpolation pixel value calculation means 141 reads the pixel values of the four peripheral pixels of the pixel position before conversion from the memory block stored in the memory block storage means 13, and calculates the pixel values by bilinear interpolation calculation. This calculated pixel value becomes the pixel value of the pixel position after conversion.

As shown in FIG. 5, when the coordinate p'of the pixel position before conversion is (x, y), the interpolated pixel value calculation means 141 sets the coordinates of the peripheral four pixels to p1 ([x], [y]]. ), P2 ([x] + 1, [y]), p3 ([x], [y] +1), p4 ([x] + 1, [y] +1). Note that [x] and [y] are integer parts of x and y, respectively.
The interpolation pixel value calculation means 141 reads the pixel values of the coordinates p1, p2, p3, p4 from the memory block corresponding to the coordinates p1, p2, p3, p4, and sets the pixel values of the coordinates p'to the following by bilinear interpolation calculation. It is calculated by the formula (1) of.

Here, the pixel value of the coordinate p'is Dst (x, y), the pixel value of the coordinate p1 is Src ([x], [y]), and the pixel value of the coordinate p2 is Src ([x] + 1, [y]. ), The pixel value of the coordinate p3 is Src ([x], [y] +1), and the pixel value of the coordinate p4 is Src ([x] + 1, [y] +1).

The interpolated pixel value calculation means 141 writes the calculated pixel value as a pixel value at the pixel position (coordinates p) of the converted equirectangular image G2 in the output image storage means 15.
It is preferable that the interpolated pixel value calculating means 141 writes the pixel value for one line in the output image storage means 15 every time the pixel value for one horizontal line of the equirectangular image G2 is calculated.
When the pixel referred to by the memory block reaches the final line, the interpolation means 14 instructs the memory block generation means 12 to generate the next memory block.
Further, the interpolation means 14 notifies the video output means 16 of the end of writing of the output image storage means 15 after the interpolation of the equirectangular image G2 is completed.

The output image storage means 15 stores the pixel value of the converted pixel position calculated by the interpolation means 14 as the pixel value of the corresponding pixel position of the frame image constituting the output video.
The output image storage means 15 stores the equirectangular image G2 as the converted image.
The output image storage means 15 can be configured with a semiconductor memory. For example, the output image storage means 15 can be configured by a general DRAM as an image memory.
The equirectangular image G2 stored in the output image storage means 15 is read out by the video output means 16.

The video output means 16 outputs an output video having an image (equirectangular image G2) stored in the output image storage means 15 as a frame image.
The video output means 16 outputs the equirectangular image G2 as a frame image at the timing when the interpolation means 14 notifies the completion of writing of the equirectangular image G2.

By configuring the image processing device 1 as described above, the image processing device 1 performs a bilinear interpolation calculation when converting a fish-eye image into a regular-distance cylindrical image, so that a high-quality image with less noise is performed. Can be generated.

Further, when performing the interpolation calculation, the video processing device 1 can read a plurality of pixel values before conversion from a memory block that can be read for each pixel. As a result, the image processing device 1 can refer to the pixels to be referred to when performing the interpolation calculation in pixel units at a time, so that the wide-angle camera can refer to the pixels as compared to the method of reading one pixel in line units such as DRAM. The captured image can be converted into an equirectangular image at high speed.
The video processing device 1 can operate a computer (not shown) with a program (video processing program) for functioning as each of the above-mentioned means.

<Operation of video processing device>
Next, the operation of the video processing apparatus 1 according to the embodiment of the present invention will be described with reference to FIG. 6 (see FIG. 1 for the configuration as appropriate).

In step S1, the video input means 10 inputs a video having the fisheye image G1 as a frame, and sequentially writes and stores the video in the input image storage means 11.
In step S2, the memory block generation means 12 stores the fisheye image G1 stored in the input image storage means 11 in step S1 in units of memory blocks divided into predetermined number of pixels in the horizontal direction and the vertical direction, respectively. It is stored in the block storage means 13. At this time, the memory block generation means 12 reads the pixel value of the number of lines corresponding to the number of pixels in the vertical direction of the memory block from the input image storage means 11 for each line, and for each number of pixels in the horizontal direction of the memory block, Memory block Write as individual memory blocks of storage means 13.

In step S3, the pre-conversion coordinate calculation means 140 of the interpolation means 14 calculates the pixel position (coordinates) of the fisheye image G1 before the equirectangular conversion corresponding to the pixel position of the equirectangular image G2.
Here, the pre-conversion coordinate calculation means 140 calculates the pixel position (coordinates) of the fisheye image G1 by performing the inverse conversion of the equirectangular transformation with respect to the pixel position of the equirectangular image G2.

In step S4, the interpolation pixel value calculation means 141 of the interpolation means 14 stores the pixel values of the four peripheral pixels of the pixel position (coordinates) of the fisheye image G1 calculated in step S3 in the memory block storage means 13. Get from the corresponding memory block.
In step S5, the interpolation pixel value calculation means 141 uses the pixel positions (coordinates) of the four pixels acquired in step S4 and the respective pixel values by bilinear interpolation calculation to convert the equirectangular image G2 into a regular distance cylinder. Calculate the pixel value of.
In step S6, the interpolation means 14 writes and stores the pixel value of the equirectangular image G2 calculated in step S5 in the output image storage means 15.

Although not shown as a step, the interpolation means 14 sequentially instructs the memory block generation means 12 to generate the next memory block when the pixel referred to by the memory block reaches the final line. , Update the memory block.

In step S7, the interpolation means 14 determines whether or not the calculation of the pixel values for all the pixels of the equirectangular image G2 is completed.
Here, if the calculation of the pixel values for all the pixels has not been completed (No in step S7), the interpolation means 14 returns to step S3 and continues the operation.
On the other hand, when the calculation of the pixel values for all the pixels is completed (Yes in step S7), the video output means 16 uses the equirectangular image G2 stored in the output image storage means 15 as a frame in the video in step S8. Is output.

In step S9, the video input means 10 determines whether or not the next frame is input.
Here, when the next frame is input (Yes in step S9), the video processing apparatus 1 returns to step S1 and continues the operation.
On the other hand, when the next frame is not input (No in step S9), the video processing apparatus 1 ends the operation.

By the above operation, the image processing apparatus 1 can convert the image taken by the wide-angle camera into the image of the equirectangular image at high speed, as compared with the method of reading one pixel in one line unit such as DRAM. ..

<Modification example>
Although the configuration and operation of the video processing apparatus 1 according to the embodiment of the present invention have been described above, the present invention is not limited to this embodiment.

(Modification 1)
Here, the input image (fish-eye image G1) has been described as a configuration (for example, RGB) in which the pixel values correspond to the pixel positions.
However, even if the fisheye image G1 is an image in which the color difference component is thinned out from the luminance component (Y) and the color difference component (Cb, Cr) such as 4: 2: 0, 4: 2: 2 of YCbCr. I do not care.
In that case, the memory block generation means 12 may write the luminance Y and the color difference components Cb and Cr as pixel values in association with each pixel of the memory block.

For example, as shown in FIG. 7A, when the fisheye image G1 is YCbCr4: 2: 2, the memory block generating means 12 generates the memory block B0 shown in FIG. 7B. It should be noted that the illustration of the other memory blocks B1 and the like is omitted here.
In the case of the example of FIG. 7, the memory block generation means 12 sets the values of Y0, Cb0, and Cr0 of the fisheye image G1 in the pixel P0 of the memory block B0, and Y1 of the fisheye image G1 in the pixel P1. Set each value of Cb0 and Cr0. As a result, the pixel position of the memory block B0 is set to the pixel value for each pixel of the fisheye image G1.
When the equirectangular image G2 has the same image format as the fisheye image G1, the video output means 16 may convert the output image into a format such as 4: 2: 0, 4: 2: 2.

(Modification 2)
Here, the interpolation operation in the interpolation means 14 has been described with an example of a bilinear interpolation operation.
However, this interpolation operation may be another interpolation operation, for example, a bicubic interpolation operation.
When the bicubic interpolation calculation is used, the memory block generation means 12 may generate a memory block with a size in which at least 16 pixels (4 × 4 pixels) referred to for the interpolation calculation are included in one block.
Then, the interpolation pixel value calculation means 141 of the interpolation means 14 reads the pixel values of the peripheral 16 pixels of the pixel position before conversion from the memory block stored in the memory block storage means 13, and performs the pixel value by the bicubic interpolation calculation. Should be calculated.

(Modification 3)
Here, the image processing device 1 has been described as converting an image taken by a wide-angle camera (fish-eye camera) into an image in equirectangular projection format.
However, the image processing device 1 may target any image as long as it converts a frame image with interpolation calculation.
For example, the image processing device 1 assumes that the image is enlarged by performing pixel interpolation, and it is also possible to input a 4K image and output an 8K image.
In that case, the image processing apparatus 1 may input a 4K image as a frame image, perform enlargement by interpolation calculation, and then output the 8K image as a frame image.
At this time, the pre-conversion coordinate calculation means 140 of the interpolation means 14 shall calculate the coordinates before enlargement from the coordinates after enlargement.

(Modification example 4)
Here, the pre-conversion coordinate calculation means 140 of the interpolation means 14 sequentially calculates the pre-conversion pixel position corresponding to the converted pixel position.
However, when this conversion is fixed, instead of the pre-conversion coordinate calculation means 140, a storage means for storing a table in which pixel positions before and after the conversion are associated in advance is provided, and the interpolation pixel value calculation means 141 refers to the table. You may do it.

1 Video processing device 10 Video input means 11 Input image storage means 12 Memory block generation means 13 Memory block storage means 14 Interpolation means 140 Pre-conversion coordinate calculation means 141 Interpolation pixel value calculation means 15 Output image storage means 16 Video output means

Claims (9)

An image processing device that converts input video into output video by conversion processing that involves interpolation.
A video input means for inputting the input video and
An input image storage means for storing a frame image constituting the input video, and an input image storage means.
A memory block generation means for dividing the frame image into memory blocks having predetermined sizes in the horizontal direction and the vertical direction, which are larger than the number of pixels required for the interpolation calculation of one pixel after conversion by the conversion process.
A memory block storage means composed of a memory that can be read in pixel units for storing a memory block generated by the memory block generation means, and a memory block storage means.
An interpolation means that reads the pixel values of the peripheral pixels of the pixel position before conversion corresponding to the pixel position after conversion from the memory block and calculates the pixel value of the pixel position after conversion by interpolation calculation.
An output image storage means that stores the pixel value of the pixel position after the conversion as the pixel value of the corresponding pixel position of the frame image constituting the output image.
A video output means for outputting the output video having an image stored in the output image storage means as a frame image, and a video output means.
A video processing device characterized by being equipped with. The video processing apparatus according to claim 1, wherein the memory block generating means overlaps a predetermined number of pixels in the horizontal direction and the vertical direction, respectively, to generate the memory block. The memory block generation means reads out the pixels of the number of lines of the frame image corresponding to the number of pixels in the vertical direction of the memory block from the input image storage means, and divides the pixels into the number of pixels in the horizontal direction of the memory block. The image processing apparatus according to claim 1 or 2, wherein the memory block is generated. The frame image constituting the input image is an image in a format in which the color difference component is thinned out from the luminance component and the color difference component, and the memory block generation means has the luminance component and the brightness component for each pixel of the memory block. The image processing apparatus according to any one of claims 1 to 3, wherein the memory block is generated using the color difference component corresponding to the luminance component as a pixel value. The interpolation operation is a bilinear interpolation operation, and the memory block generation means generates the memory block with a larger number of pixels than the horizontal 2 pixels × vertical 2 pixels required for the interpolation operation. The video processing apparatus according to any one of claim 4. The interpolation operation is a bicubic interpolation operation, and the memory block generation means generates the memory block with a larger number of pixels than the horizontal 4 pixels × vertical 4 pixels required for the interpolation operation. The image processing apparatus according to any one of claims 4. Claims 1 to 6 are characterized in that the frame image constituting the input image is a fisheye image, and the conversion process is an equirectangular conversion that converts the fisheye image into a regular distance cylindrical image. The image processing apparatus according to any one of the above. The interpolation means is
Pre-conversion coordinate calculation to calculate the pixel position of the fish-eye image before conversion corresponding to the pixel position of the regular-distance cylindrical image by the inverse conversion of the equirectangular transformation that converts the fish-eye image to the regular-distance cylindrical image. Means and
Interpolated pixel value calculating means for calculating the pixel value of the pixel position of the equidistant cylindrical image after conversion by performing an interpolation calculation with the pixel values of the peripheral pixels of the pixel position before conversion calculated by the coordinate calculation means before conversion. When,
7. The video processing apparatus according to claim 7. A video processing program for causing a computer to function as the video processing device according to any one of claims 1 to 8.

Images