JP7047878B2 - Video encoding device - Google Patents

Description

The present invention relates to a moving image coding device.

In moving image coding, an image in a rectangular region is often divided into blocks, and motion compensation and frequency conversion are performed in block units. As a typical moving image coding method, ISO / IEC 23008.2 High Efficiency Video Coding (HEVC) can be mentioned.

In recent years, the functions of cameras that shoot images have been expanded, and there is a tendency to shift from shooting rectangular areas to wider-angle panoramic shooting. Further, it is becoming possible to shoot an omnidirectional panoramic image using a fisheye lens and a 360 ° panoramic image using a plurality of cameras (see, for example, Non-Patent Documents 1 to 3).

A method for estimating motion of a panoramic image including 360 ° omnidirectional image information is also known (see, for example, Patent Document 1). A method of determining a motion vector search area using a global vector between images and a method of dividing a panoramic image to generate a panoramic image in which a moving object can be easily monitored are also known (for example, Patent Documents 2 and 3). See).

Japanese Patent Publication No. 2008-510359 Japanese Unexamined Patent Publication No. 2010-109917 Japanese Unexamined Patent Publication No. 2013-218432

"Entaniya Fisheye Support Blog", [online], [Search on November 16, 2016], Internet <URL: https://www.entapano.com/blog/360-degree-panoramic-video-camera/> "THETA", [online], [Search on November 16, 2016], Internet <URL: https://theta360.com/ja/about/theta/> "Professional Plug & Play 360 ° Video Camera", [online], [Searched on November 16, 2016], Internet <URL: http://www.sphericam.com/sphericam2/>

When encoding a panoramic image with large motion, many motion vectors may be generated and the coding efficiency (compression rate) may decrease.
In one aspect, the present invention aims to improve the coding efficiency of panoramic images.

In one proposal, the moving image coding device includes a storage unit, a determination unit, a correction unit, and a coding unit.
The storage unit stores a reference image used for encoding a coded panoramic image included in a panoramic image in which a plurality of images captured by a plurality of image pickup devices are combined.

The determination unit represents the amount of deviation of the coding target area with respect to the reference image when the coding target area in the coding target panoramic image shifts with respect to the reference image due to the movement of the shooting range of each of the plurality of images. Determine the vector. The correction unit corrects the position of the coded target area in the coded target panoramic image based on the vector representing the deviation amount, thereby generating the corrected coded target area.

The coding unit encodes the corrected image of the coded target area using the reference image.

According to the embodiment, the coding efficiency of the panoramic image can be improved.

It is a figure which shows the 1st development panoramic image. It is a figure which shows the 2nd development panoramic image. It is a figure which shows the 3rd development panoramic image. It is a functional block diagram of the 1st moving image coding apparatus. It is a flowchart of the 1st moving image coding process. It is a functional block diagram of the 2nd moving image coding apparatus. It is a flowchart of the 2nd moving image coding process. It is a functional block diagram of a moving image decoding apparatus. It is a flowchart of a moving image decoding process. It is a block diagram of the panoramic image coding system. It is a functional block diagram which shows the specific example of a moving image coding apparatus. It is a figure which shows the correction process. It is a figure which shows the panoramic image of the whole circumference. It is a figure which shows the correction process for the all-around panoramic image. It is a flowchart which shows the specific example of the 1st moving image coding process. It is a flowchart of the correction process based on the number of occurrences of a motion vector. It is a flowchart of the correction process based on ME efficiency. It is a flowchart of the moving image coding process based on the coding efficiency. It is a flowchart of the correction process based on the coding efficiency. It is a functional block diagram which shows the specific example of the moving image decoding apparatus. It is a flowchart which shows the specific example of the moving image decoding process. It is a flowchart which shows the specific example of the 2nd moving image coding processing. It is a block diagram of an information processing apparatus.

Hereinafter, embodiments will be described in detail with reference to the drawings.
When the panoramic image at each time included in the panoramic image is encoded, the captured panoramic image is in a distorted state, and if it is encoded as it is, the coding efficiency is lowered. Therefore, it is effective to correct the distortion of the panoramic image, develop it into a rectangular panoramic image, divide the developed panoramic image into a predetermined area of the coding unit, and perform coding in the predetermined area unit. be.

The rectangular panoramic image at each time may be referred to as a frame or picture, and a predetermined area of coding unit may be referred to as a block.

When the reference position when dividing a rectangular panoramic image into blocks is a fixed position with respect to the captured panoramic image, panning, misalignment, or movement of the camera causes all blocks to be in a fixed direction between frames. An image that looks like it has moved to may be generated. If such a panoramic image is encoded while the reference position is fixed, an unnecessarily large number of motion vectors are generated and the coding efficiency is lowered.

The motion estimation method of Patent Document 1 utilizes the high correlation of the panoramic image including 360 ° omnidirectional image information, and the pixel of the left boundary portion of the screen is used as the pixel of the region on the right side of the right boundary of the screen. Loaded. Further, the pixels in the right boundary portion of the screen are read as the pixels in the region on the left side of the left boundary of the screen. Thereby, the image quality of the right side boundary portion and the left side boundary portion of the screen can be improved.

In this motion estimation method, it is premised that the position of the camera is fixed and does not move significantly for the panoramic image in all directions to be captured. That is, the target is a panoramic image that looks around 360 ° around a predetermined point.

However, in recent years, handheld or in-vehicle omnidirectional cameras as shown in Non-Patent Documents 1 to 3 have come to be used.

FIG. 1 shows an example of a first developed panoramic image generated by developing a panoramic image taken by the camera of Non-Patent Document 1. This camera can capture a circular panoramic image 101 using a single convex lens (fisheye lens).

For example, when the panoramic image 101 is photographed outdoors with the convex lens directed directly upward, the sky is captured in the central portion of the panoramic image 101, so the developed panoramic image 103 is generated using the region excluding the central portion. be able to. In this case, the developed panoramic image 103 is generated by cutting the panoramic image 101 at the boundary line 102 and converting it into a rectangular shape.

FIG. 2 shows an example of a second developed panoramic image generated by developing a panoramic image taken by a 360 ° camera of Non-Patent Document 2. This 360 ° camera can capture a circular panoramic image 201 and a circular panoramic image 202 by using two cameras arranged back to back.

For example, when a user in a vehicle shoots one camera toward the front in the traveling direction, the panoramic image 201 shows the scenery in front and the panoramic image 202 shows the scenery in the rear. In this case, by converting the panoramic image 201 and the panoramic image 202 into rectangular shapes, the front unfolded panorama image 203 and the rear unfolded panorama image 204 are generated. Further, it is also possible to generate the omnidirectional panoramic image 205 by connecting the expanded panoramic image 203 and the expanded panoramic image 204.

FIG. 3 shows an example of a third developed panoramic image generated by developing a panoramic image taken by a 360 ° camera of Non-Patent Document 3. This 360 ° camera can capture a panoramic image 301 of not only the entire horizontal circumference but also the entire space including the heavens and the earth by using six cameras arranged in a spherical shape.

For example, when this 360 ° camera is mounted on the roof of a vehicle and photographed, the landscape in front of the traveling direction is captured in the region 311 in the panoramic image 301, and the landscape in the rear left is captured in the region 312, and the region 313 is captured. And the landscape on the right rear is reflected in the area 314. In this case, by converting the images of the region 311 and the region 312 into rectangular shapes, the front expanded panoramic image 321 and the left rear expanded panoramic image 322 are generated. Further, by converting the images of the area 313 and the area 314 into a rectangular shape, the developed panoramic image 323 on the right rear side is generated.

Further, it is also possible to generate an omnidirectional panoramic image by connecting a plurality of developed panoramic images including the developed panoramic image 321 to the developed panoramic image 323.

Since the panoramic image 101 of FIG. 1, the panoramic image 201 of FIG. 2, the panoramic image 202, and the panoramic image 301 of FIG. 3 are images taken with a wide field of view, they are often distorted. On the other hand, the unfolded panorama image 103, the unfolded panorama image 203, the unfolded panorama image 204, the all-around panorama image 205, and the unfolded panorama image 321 to the unfolded panorama image 323 are converted into a rectangular shape and are images without distortion.

In moving image coding, it is more efficient to encode a panoramic image without distortion than to encode a panoramic image with distortion, such as motion estimation (ME). Therefore, it is expected that there will be many opportunities to use the developed panoramic image when encoding the panoramic image.

The ME efficiency is represented by, for example, the absolute value of the difference between the image of each block in the coded panoramic image and the image of the predicted block determined by the motion vector. In this case, the smaller the absolute value of the difference, the better the ME efficiency.

Since the moving image coding includes motion compensation and frequency conversion in block units, the generated information is mainly motion vector information and frequency-converted difference information. Therefore, when the ME efficiencies are almost the same, the amount of motion vector information has a great influence on the coding efficiency. Further, in the panoramic image with almost no movement and the panoramic image with large movement, the latter generates more motion vector information. That is, the larger the motion of the panoramic image, the larger the amount of code generated, and the lower the coding efficiency.

It is possible to increase the quantization scale in order to suppress the amount of code generated and keep the coding efficiency constant, but in that case, the image quality of the panoramic image restored from the coded panoramic image is deteriorated.

As described above, when an image in which all blocks move in a certain direction between frames is generated by panning, misalignment, or movement of the camera, in the motion estimation method of Patent Document 1, many motion vectors are generated. It occurs and the coding efficiency decreases.

FIG. 4 shows a functional configuration example of the first moving image coding device of the embodiment. The moving image coding device 401 of FIG. 4 includes a storage unit 411, a determination unit 412, a correction unit 413, and a coding unit 414. The storage unit 411 stores the reference panoramic image 421 used for encoding the coded panoramic image. The coded panoramic image is a panoramic image obtained by developing a panoramic image included in the panoramic image taken by the image pickup apparatus. The determination unit 412, the correction unit 413, and the coding unit 414 perform moving image coding processing using the reference panoramic image 421.

FIG. 5 is a flowchart showing an example of the first moving image coding process performed by the moving image coding device 401 of FIG. First, the determination unit 412 determines a vector representing the amount of deviation of the coded panoramic image with respect to the reference panoramic image 421 (step 501).

Next, the correction unit 413 generates a corrected coded target panoramic image by correcting the positions of each of the plurality of coded target areas in the coded target panoramic image based on a vector representing the amount of deviation. (Step 502). Then, the coding unit 414 encodes the image of each of the plurality of coding target regions in the corrected coded target panorama image by using the reference panorama image 421 (step 503).

According to such a moving image coding device 401, it is possible to improve the coding efficiency of the panoramic image.

FIG. 6 shows a functional configuration example of the second moving image coding device of the embodiment. The moving image coding device 601 of FIG. 6 includes a storage unit 611, a determination unit 612, a correction unit 613, and a coding unit 614. The storage unit 611 stores a reference image 621 used for encoding the coded panoramic image. The coded panoramic image is a panoramic image included in a panoramic image in which a plurality of images taken by a plurality of image pickup devices are combined. The determination unit 612, the correction unit 613, and the coding unit 614 perform a moving image coding process using the reference image 621.

FIG. 7 is a flowchart showing an example of the second moving image coding process performed by the moving image coding device 601 of FIG. First, when the coding target area in the coding target panoramic image is deviated from the reference image 621 due to the movement of the shooting range of each of the plurality of images, the determination unit 612 determines the coding target area for the reference image 621. A vector representing the amount of deviation of is determined (step 701).

Next, the correction unit 613 corrects the position of the coded target area in the coded panoramic image based on the vector representing the deviation amount, thereby generating the corrected coded target area (step 702). .. Then, the coding unit 614 encodes the corrected image of the coded target region using the reference image 621 (step 703).

According to such a moving image coding device 601, it is possible to improve the coding efficiency of the panoramic image.

FIG. 8 shows a functional configuration example of the moving image decoding device of the embodiment. The moving image decoding device 801 of FIG. 8 includes a storage unit 811, an extraction unit 812, a decoding unit 813, and a correction unit 814. The moving image coding device 401 of FIG. 4 generates a coded panoramic image by encoding the coded panoramic image. The moving image decoding device 801 decodes the coded panoramic image including the coded panoramic image generated by the moving image coding device 401.

The storage unit 811 stores the reference panoramic image 821 used for decoding the coded panoramic image. The extraction unit 812, the decoding unit 813, and the correction unit 814 perform a moving image decoding process using the reference panoramic image 821.

FIG. 9 is a flowchart showing an example of the moving image decoding process performed by the moving image decoding device 801 of FIG. First, the extraction unit 812 extracts a vector representing the amount of deviation of the coded panoramic image with respect to the reference panoramic image 821 from the coded panoramic image (step 901).

Next, the decoding unit 813 decodes each of the plurality of decoding target regions in the coded panoramic image using the reference panoramic image 821 to generate a decoded panoramic image (step 902). Then, the correction unit 814 restores the coded panoramic image by correcting the positions of the plurality of decoded target regions in the decoded panoramic image based on the vector representing the deviation amount (step 903).

According to such a moving image decoding device 801, it is possible to improve the coding efficiency of the panoramic image.

FIG. 10 shows a configuration example of a panoramic video coding system. The panoramic video coding system 1001 of FIG. 10 includes a photographing unit 1011, a compositing unit 1012, a developing unit 1013, and a moving image coding device 1014.

The photographing unit 1011 includes one or a plurality of imaging devices. As the photographing unit 1011 including one image pickup device, for example, the camera of Non-Patent Document 1 can be used. Further, as the photographing unit 1011 including a plurality of image pickup devices, for example, a 360 ° camera of Non-Patent Document 2 or a 360 ° camera of Non-Patent Document 3 can be used.

When the photographing unit 1011 includes one image pickup device, the synthesis unit 1012 outputs an omnidirectional panoramic image at each time included in the panoramic image captured by the image pickup device. The expansion unit 1013 divides the omnidirectional panoramic image into a plurality of regions and repeats the planar projective transformation in each region to generate the expansion panoramic image. In this case, the moving image coding device 1014 corresponds to the moving image coding device 401 of FIG. 4, and encodes the developed panoramic image at each time to generate a bit stream of the coded panoramic image.

On the other hand, when the photographing unit 1011 includes a plurality of imaging devices, the combining unit 1012 synthesizes the images included in the images captured by the respective imaging devices to generate an omnidirectional panoramic image, and the developing unit 1013 generates an omnidirectional panoramic image. Convert the omnidirectional panoramic image to generate a developed panoramic image. In this case, the moving image coding device 1014 corresponds to the moving image coding device 601 of FIG. 6 and encodes the developed panoramic image at each time to generate a bit stream of the coded panoramic image.

When the photographing unit 1011 is stationary, the same photographing range overlooking the surroundings by 360 ° with the photographing unit 1011 as the center is always photographed as the whole panoramic image of the entire circumference. However, there is a possibility that the position of the subject in the picture may shift due to the shift of the starting point of the panoramic image of the entire circumference at each time due to the panning of the photographing unit 1011 or the like. On the other hand, when the photographing unit 1011 is moving, it is assumed that the photographing range changes with the passage of time and the captured subject also changes.

FIG. 11 shows a specific example of the moving image coding device 1014 of FIG. The moving image coding device 1014 of FIG. 11 includes a change unit 1101, a determination unit 1102, a determination unit 1103, a change unit 1104, a subtraction unit 1105, a conversion and quantization unit (T / Q) 1106, and an entropy coding unit (ENT). ) 1107 is included. Further, the moving image coding device 1014 includes an inverse quantization and inverse conversion unit (IQ / IT) 1108, an addition unit 1109, a motion compensation unit 1110, a prediction image generation unit 1111 and a frame memory 1112.

First, the operation when the moving image coding device 1014 corresponds to the moving image coding device 401 of FIG. 4 will be described. In this case, the change unit 1101 and the determination unit 1102 correspond to the correction unit 413, and the determination unit 1103 corresponds to the determination unit 412. The subtraction unit 1105, T / Q1106, ENT1107, IQ / IT1108, addition unit 1109, motion compensation unit 1110, and prediction image generation unit 1111 correspond to the coding unit 414, and the frame memory 1112 corresponds to the storage unit 411. ..

The unfolded panoramic image generated by the unfolding unit 1013 is input to the moving image coding device 1014 as a coded target panoramic image. The moving image coding device 1014 encodes the coded panoramic image and outputs the coded panoramic image as a bit stream. The coded panoramic image is divided into a plurality of blocks, and each block is input to the subtraction unit 1105 and the motion compensation unit 1110 as the coded block. The coded block corresponds to the coded area.

The determination unit 1103 uses the motion vector output from the motion compensation unit 1110 to determine a global vector representing the amount of deviation of the coded panoramic image with respect to the reference panoramic image stored in the frame memory 1112. Then, the determination unit 1103 outputs the determined global vector to the determination unit 1102, the change unit 1104, and the ENT 1107.

The determination unit 1103 may use the global vector of the panoramic image that has already been encoded as the global vector of the panoramic image to be encoded, and is global based on the result of motion estimation using the reduced panoramic image with a small amount of processing. The vector may be determined.

The determination unit 1102 determines whether or not to change the position of each coded target block in the coded target panoramic image based on the global vector, and outputs the global vector and the determination result to the change unit 1101.

When the determination result indicates that the position of each coding target block is changed, the changing unit 1101 changes the position of each coding target block based on the global vector, and the changed coding target block is subtracted from the subtracting unit 1105 and It is output to the motion compensation unit 1110. On the other hand, when the determination result indicates that the position of each coded block is not changed, the changing unit 1101 outputs each coded block as it is to the subtraction unit 1105 and the motion compensation unit 1110.

The subtraction unit 1105 outputs a prediction error signal representing the difference between the coded block and the prediction block image output from the prediction image generation unit 1111 to the T / Q 1106. The T / Q1106 converts a prediction error signal into a frequency signal by orthogonal transformation, and quantizes the frequency signal to generate coefficient information. As the orthogonal transform, for example, a discrete cosine transform, a discrete wavelet transform, or the like is used. Then, the T / Q1106 outputs the generated coefficient information to the ENT1107 and the IQ / IT1108.

IQ / IT1108 inversely quantizes the coefficient information output from T / Q1106 to generate a frequency signal, and converts the frequency signal into a reconstruction prediction error signal by inverse orthogonal transformation. Then, IQ / IT1108 outputs the reconstruction prediction error signal to the addition unit 1109.

The addition unit 1109 generates a decoding block image by adding the prediction block image output from the prediction image generation unit 1111 and the reconstruction prediction error signal, and outputs the generated decoding block image to the frame memory 1112. .. When the position of the coded block is changed, the changing unit 1104 returns the position of the decoded block image to the position of the coded block before the change based on the global vector.

The frame memory 1112 accumulates the decoded block image and outputs the accumulated decoded block image as a reference image to the motion compensation unit 1110 and the predicted image generation unit 1111. The plurality of decoded block images generated from each of the plurality of coded blocks in the coded panoramic image correspond to the reference panoramic image.

The motion compensation unit 1110 generates a motion vector by performing motion estimation for the coded block using the reference image, and outputs the generated motion vector to the determination unit 1103. In motion estimation, for example, one-way prediction using one reference panoramic image or two-way prediction using two or more reference panoramic images is performed. Then, the motion compensation unit 1110 acquires the reference image indicated by the motion vector from the reference panoramic image and outputs it to the prediction image generation unit 1111.

The prediction image generation unit 1111 uses the reference image to generate an intra prediction block image of the coding target block from the pixel values of the peripheral pixels already encoded in the coding target panorama image. Further, the prediction image generation unit 1111 uses the reference image output from the motion compensation unit 1110 as the inter-prediction block image. Then, the prediction image generation unit 1111 selects either the intra prediction block image or the inter prediction block image, and outputs the selected prediction block image to the subtraction unit 1105 and the addition unit 1109.

The ENT1107 entropy-codes the coefficient information output from the T / Q1106, the information in the prediction mode of the intra-prediction or the inter-prediction, and the information of the global vector output from the determination unit 1103. In entropy coding, a variable length code is assigned according to the frequency of appearance of each symbol in the signal. Then, the ENT 1107 outputs a bit stream including a variable length code.

FIG. 12 shows an example of the correction process performed by the change unit 1101. In this example, numbers 0-9 are used to distinguish between a plurality of positions in the omnidirectional panoramic image. In the case of the omnidirectional panoramic image, as shown in FIG. 1, since the image at the leftmost position 0 and the image at the rightmost position 9 are continuous, the correction process can be performed by utilizing this continuity. ..

When the positions of the subjects in the coded panoramic image 1202 are all shifted to the right with respect to the position of each subject in the reference panoramic image 1201, the same motion vector 1211 is generated for the blocks from position 0 to position 7. Generated. However, as it is, a motion vector completely different from the motion vector 1211 is generated for the blocks at positions 8 and 9.

Therefore, the determination unit 1103 determines the motion vector 1211 as a global vector, and the change unit 1101 shifts the position of each block in the coded target panorama image 1202 to the left according to the motion vector 1211. Then, the changing unit 1101 generates the coded target panorama image 1203 by adding the image of the block at the position 8 and the position 9 protruding from the coded target panorama image 1202 to the right end of the coded target panorama image 1202. In this case, the starting point of the coded panoramic image 1203 is changed from the position 8 to the position 0, which coincides with the starting point of the reference panoramic image 1201.

As a result, the movement of each subject in the coded panoramic image 1203 with respect to each subject in the reference panoramic image 1201 becomes almost 0, so that the generated motion vector can be minimized.

FIG. 13 shows an example of an omnidirectional panoramic image including a person. The panoramic image 1301, the panoramic image 1302, and the panoramic image 1303 represent the coded panoramic images at time T, time T + 1, and time T + 2, respectively. The position of the person in these three panoramic images has not changed, but instead the position of the subject in the background has shifted to the right over time. In this case, since the area of the moving background is larger than the area of the stationary person, it is preferable to shift the starting point of each panoramic image to the left so that the background is stationary.

FIG. 14 shows an example of correction processing for the omnidirectional panoramic image of FIG. When the panoramic image 1301 is used as a reference panoramic image in the motion estimation of the panoramic image 1302, the determination unit 1103 determines a motion vector representing the motion of the background of the panoramic image 1302 with respect to the background of the panoramic image 1301 as a global vector.

Then, the changing unit 1101 shifts the position of each block in the panoramic image 1302 to the left according to the global vector, and adds the image of the block protruding from the panoramic image 1302 to the right end to generate the panoramic image 1401. If the panoramic image 1401 is encoded instead of the panoramic image 1302, a motion vector for a background having a large area is hardly generated, and a motion vector is generated concentrated on a person having a small area. Therefore, the generated motion vector information can be suppressed, and the coding efficiency is improved.

When the panoramic image 1302 is used as the reference panoramic image in the motion estimation of the panoramic image 1303, the same correction processing is performed. In this case, the determination unit 1103 determines a motion vector representing the movement of the background of the panoramic image 1303 with respect to the background of the panoramic image 1302 as a global vector.

Then, the changing unit 1101 shifts the position of each block in the panoramic image 1303 to the left according to the global vector, and adds the image of the block protruding from the panoramic image 1303 to the right end to generate the panoramic image 1402. By encoding the panoramic image 1401 instead of the panoramic image 1303, the coding efficiency is improved.

When decoding the coded panoramic image generated by the moving image coding device 1014, the panoramic image 1302 and the panoramic image 1303 are obtained by shifting the position of each block in the coded panoramic image in the opposite direction based on the global vector. Can be restored.

In this way, by changing the starting point of the panoramic image to be encoded based on the global vector, it is possible to adaptively change the position of the boundary line that cuts the panoramic image around the entire circumference according to the movement of the image. Become. It is preferable to change the starting point according to the movement of the forward prediction. For example, the starting point may be changed only for the forward prediction picture (P picture).

The ENT1107 can insert the global vector information into the bitstream as header information. In this case, the moving image decoding device that has received the bitstream can acquire the global vector by referring to the header information and return the position of each block in the coded panoramic image to the original position.

FIG. 15 is a flowchart showing a specific example of the moving image coding process performed by the moving image coding device 1014 of FIG. The moving image coding device 1014 encodes the coded target image block by block, using the image at each time included in the coded target image as the coded target image.

First, the change unit 1101 checks whether or not the coded image is an omnidirectional panoramic image (step 1501).

When the coded image is not an all-around panoramic image (steps 1501 and No), the changing unit 1101 outputs the coded object block to the subtracting unit 1105 and the motion compensating unit 1110 as it is without changing the position of the coded block. Then, the moving image coding device 1014 encodes the coded block (step 1502).

On the other hand, when the coded target image is an omnidirectional panoramic image (steps 1501 and Yes), the determination unit 1103 determines the global vector of the coded target panoramic image (step 1505).

For example, the determination unit 1103 can determine a motion vector that occurs more than other motion vectors among the motion vectors that occur in the coded panoramic image as a global vector. The motion vector that occurs more than the other motion vectors may be the motion vector that occurs most in the coded panoramic image. In this case, the motion compensation unit 1110 performs motion estimation for all the blocks in the coded panoramic image in advance, obtains the motion vector of each block, and outputs the motion vector to the determination unit 1103. Then, the determination unit 1103 selects the motion vector having the largest number of occurrences and determines it as a global vector.

By selecting the motion vector with the largest number of occurrences, the vector representing the motion of the entire panoramic image to be encoded can be determined as the global vector. For example, when the area of the background is larger than the area of the foreground and a certain movement is seen in the background, the vector representing the movement of the background is adopted as the global vector. Further, when the foreground is close-up and the area of the foreground is larger than the area of the background, the vector representing the movement of the foreground is adopted as the global vector.

Next, the change unit 1101 checks whether or not the coded block is a block of the forward prediction picture (step 1506). When the coded block is not a block of the forward prediction picture (steps 1506, No), the moving image coding device 1014 performs the processing after step 1502.

On the other hand, when the coded block is a block of the forward prediction picture (steps 1506, Yes), the changing unit 1101 and the determination unit 1102 perform correction processing based on the global vector (step 1507).

When the determination unit 1102 determines that the coded block is not corrected in this correction process, the moving image coding device 1014 performs the processes after step 1502. On the other hand, when the determination unit 1102 determines that the coded target block is to be corrected, the changing unit 1101 shifts the position of the coded target block in the direction of the global vector by the size of the global vector to correct the coding. Generate the target block.

Next, the ENT 1107 generates header information including the global vector information (step 1508), and the moving image coding device 1014 encodes the corrected coded block (step 1509). Then, the change unit 1104 returns the position of the decoded block image to the original position based on the global vector (step 1510). At this time, the change unit 1104 generates a corrected decoded block image by shifting the position of the decoded block image by the size of the global vector in the direction opposite to the global vector, and writes the corrected block image to the frame memory 1112.

Next, the change unit 1101 checks whether or not all the blocks in the image to be encoded have been encoded (step 1503). When an unencoded block remains (steps 1503, No), the moving image coding apparatus 1014 repeats the processing after step 1501 with the next block as the coding target block.

When all the blocks are encoded (steps 1503, Yes), the changing unit 1101 checks whether or not all the images included in the coded target video are encoded (step 1504). When an unencoded image remains (steps 1504, No), the moving image coding apparatus 1014 repeats the processes after step 1501 with the next image as the image to be encoded. Then, when all the images are encoded (steps 1504, Yes), the moving image encoding device 1014 ends the process.

For example, when the moving image coding method is HEVC, in step 1508, the ENT 1107 can transmit the global vector information by using the user data included in the header information.

Specifically, user_data_unregistered (payloadSize) of payloadType = 5 of Supplemental Enhancement Information (SEI) can be used. In this case, as the number of bits of the X component and the Y component of the global vector, the number of bits indicated by log2_max_mv_length_horizontal and log2_max_mv_length_vertical may be secured, respectively. Alternatively, 16 bits, which is the maximum value of the syntax element, may be secured.

Considering that the user data is used for other purposes, it is also preferable to use SEI (user data) in which predetermined identification information indicating the global vector is inserted before the global vector. As the predetermined identification information, for example, an American Standard Code for Information Interchange (ASCII) code corresponding to a predetermined character string representing a global vector can be used.

According to the moving image coding process of FIG. 15, when coding the forward prediction picture included in the omnidirectional panoramic image, the starting point of the coded panoramic image is according to the movement of the entire coded panoramic image. Can be changed. As a result, it becomes possible to suppress the motion vector information generated for the coded target block, and it is possible to perform appropriate motion estimation for all the blocks in the coded target panoramic image. Therefore, ME efficiency and coding efficiency are improved.

FIG. 16 is a flowchart showing an example of a correction process in which, in step 1507 of FIG. 15, it is determined whether or not to correct the coded block by using the number of motion vector generations. First, the determination unit 1102 compares the ratio R of the number of occurrences of global vectors Num_GMV to the total number of motion vectors Num_AllMV in the coded panoramic image with the threshold value TH1 (step 1601).

When R is TH1 or more (steps 1601, Yes), the determination unit 1102 determines that the coded block is corrected. Then, the changing unit 1101 generates the corrected coded object block by shifting the position of the coded object block using the global vector (step 1602), and the moving image coding device 1014 performs the motion image coding device 1014 and subsequent steps. Perform processing.

On the other hand, when R is less than TH1 (steps 1601 and No), the determination unit 1102 determines that the coding target block is not corrected, and the moving image coding device 1014 performs the processing after step 1502.

According to the correction process of FIG. 16, when the global vector determined by the determination unit 1103 is generated at a ratio of a predetermined value or more in the coded target panoramic image, the position of the coded target block is changed. As a result, the position of the coded target block can be shifted according to the constant movement occurring in the entire coded target panoramic image.

FIG. 17 is a flowchart showing an example of a correction process in which it is determined by using the ME efficiency whether or not to correct the coded block in step 1507 of FIG.

First, the determination unit 1102 requests the motion compensation unit 1110 to estimate the motion when the coded panoramic image is not corrected (step 1701). The motion compensation unit 1110 performs motion estimation for all blocks in the coded panoramic image and obtains a motion vector for each block. Then, the motion compensation unit 1110 calculates the difference absolute value sum (SAD) between the reference panoramic image and the coded target panoramic image, and outputs the calculated SAD as SAD1 to the determination unit 1102.

Next, the determination unit 1102 requests the motion compensation unit 1110 to estimate the motion when the coded panoramic image is corrected (step 1702). The motion compensation unit 1110 generates a corrected coded object block by shifting the position of the coded object block using a global vector. Next, the motion compensation unit 1110 performs motion estimation for all the blocks in the corrected panoramic image to be coded, and obtains a motion vector for each block. Then, the motion compensation unit 1110 calculates the SAD between the reference panoramic image and the corrected panoramic image to be encoded, and outputs the calculated SAD as SAD2 to the determination unit 1102.

Next, the determination unit 1102 compares SAD1 and SAD2 (step 1703). When SAD2 is smaller than SAD1 (steps 1703, Yes), the determination unit 1102 determines that the coded block is to be corrected. Then, the changing unit 1101 generates the corrected coded object block by shifting the position of the coded object block using the global vector (step 1704), and the moving image coding device 1014 performs after step 1508. Perform processing.

On the other hand, when SAD2 is SAD1 or more (steps 1703 and No), the determination unit 1102 determines that the coded block is not corrected, and the moving image coding device 1014 performs the processing after step 1502.

According to the correction process of FIG. 17, the position of the coded target block is changed when the ME efficiency is improved by shifting the position of the coded target block based on the global vector determined by the determination unit 1103. This makes it possible to improve the ME efficiency of the panoramic image to be encoded.

FIG. 18 is a flowchart showing an example of a moving image coding process for determining whether or not to correct a panoramic image to be coded by using the coding efficiency. The processing of steps 1801 to 1806 is the same as the processing of steps 1501 to 1506 of FIG.

When the coded block is a block of a forward prediction picture (step 1806, Yes), the moving image coding device 1014 performs the process of step 1807 and the process of steps 1808 to 1811. The process of step 1807 is the same as the process of step 1802 when the coded block is not corrected.

In step 1808, the change unit 1101 generates the corrected coded object block by shifting the position of the coded object block using the global vector. The processing of steps 1809 to 1811 is the same as the processing of steps 1508 to 1510 when the coded block is corrected in FIG.

Next, the change unit 1101 checks whether or not all the blocks in the image to be encoded have been encoded (step 1812). When an unencoded block remains (step 1812, No), the moving image coding apparatus 1014 repeats the processing after step 1801 with the next block as the coding target block.

When all the blocks are encoded (step 1812, Yes), the changing unit 1101 and the determination unit 1102 perform a correction process based on the coding efficiency (step 1813). Then, the moving image coding device 1014 performs the processing after step 1804.

FIG. 19 is a flowchart showing an example of the correction process in step 1813 of FIG. The determination unit 1102 calculates the coding efficiency using the quantization scale used by the T / Q1106 to quantize the frequency signal and the amount of information of the bitstream generated by the ENT1107. For example, the product of the average value of the quantization scale for all the blocks in the coded panorama image and the total number of bits of the code generated from the coded panorama image is encoded in the coded panorama image. It can be used as an efficiency.

First, the determination unit 1102 calculates the coding efficiency CE1 when the coding target panoramic image is not corrected from the processing result of step 1807 for all the blocks in the coding target panoramic image (step 1901). The coding efficiency CE1 is obtained as the product of the average value QP1 of the quantization scale and the total number of bits I1 of the code when the panoramic image to be coded is not corrected.

Next, the determination unit 1102 calculates the coding efficiency CE2 when the coding target panorama image is corrected from the processing results of steps 1808 to 1811 for all the blocks in the coding target panorama image (step 1902). ). The coding efficiency CE2 is obtained as the product of the average value QP2 of the quantization scale when the panoramic image to be coded is corrected and the total number of bits I2 of the code.

Next, the determination unit 1102 compares CE1 and CE2 (step 1903). When CE2 is smaller than CE1 (step 1903, Yes), the determination unit 1102 determines that the coded panoramic image is corrected. Then, the moving image coding device 1014 adopts the result of the processing of steps 1808 to 1811 (step 1905).

On the other hand, when CE2 is CE1 or more (step 1903, No), the determination unit 1102 determines that the coded panoramic image is not corrected, and the moving image coding device 1014 adopts the result of the process of step 1807. (Step 1904).

According to the correction process of FIG. 19, the position of the coding target block is changed when the coding efficiency is improved by shifting the position of the coding target block based on the global vector determined by the determination unit 1103. .. This makes it possible to improve the coding efficiency of the panoramic image to be coded.

FIG. 20 shows a specific example of the moving image decoding device 801 of FIG. The moving image decoding device 2001 of FIG. 20 includes an extraction unit 2011, a change unit 2012, a block decoding unit 2013, an addition unit 2014, a motion compensation unit 2015, a prediction image generation unit 2016, and a frame memory 2017.

The extraction unit 2011 corresponds to the extraction unit 812 of FIG. 8, the change unit 2012 corresponds to the correction unit 814, and the block decoding unit 2013, the addition unit 2014, the motion compensation unit 2015, and the prediction image generation unit 2016 are decoded. The frame memory 2017 corresponds to the storage unit 813 and corresponds to the storage unit 811.

The bit stream output by the moving image coding device 1014 of FIG. 11 is input to the moving image decoding device 2001 as a coded panoramic image. The moving image decoding device 2001 decodes the coded panoramic image at each time included in the coded panoramic image and restores the coded panoramic image.

The extraction unit 2011 extracts the global vector information from the header information included in the coded panoramic image and outputs it to the change unit 2012.

The block decoding unit 2013 reverse-quantifies the coefficient information of each decoding target block in the coded panoramic image to generate a frequency signal, and converts the frequency signal into a reconstruction prediction error signal by inverse orthogonal transformation. Then, the block decoding unit 2013 outputs the reconstruction prediction error signal to the addition unit 2014.

The addition unit 2014 generates a decoding block image by adding the prediction block image output from the prediction image generation unit 2016 and the reconstruction prediction error signal, and outputs the generated decoding block image to the frame memory 2017. .. When the global vector for the coded panoramic image is output from the extraction unit 2011, the change unit 2012 returns the position of the decoded block image to the position of the coded block before the change based on the global vector.

The frame memory 2017 accumulates the decoded block image, and outputs the accumulated decoded block image as a reference image to the motion compensation unit 2015 and the prediction image generation unit 2016. The plurality of decoded block images generated from each of the plurality of decoded blocks in the coded panoramic image correspond to the decoded panoramic image. The plurality of decoded block images after the position is changed by the change unit 2012 represent the restored panoramic image to be encoded and correspond to the reference panoramic image.

The motion compensation unit 2015 acquires the reference image indicated by the motion vector from the reference panoramic image and outputs it to the prediction image generation unit 2016.

The prediction image generation unit 2016 generates an intra prediction block image of the decoding target block from the pixel values of the peripheral pixels already decoded in the coded panorama image using the reference image. Further, the prediction image generation unit 2016 uses the reference image output from the motion compensation unit 2015 as the inter-prediction block image. Then, the prediction image generation unit 2016 selects either the intra prediction block image or the inter prediction block image, and outputs the selected prediction block image to the addition unit 2014.

FIG. 21 is a flowchart showing a specific example of the moving image decoding process performed by the moving image decoding device 2001 of FIG. 20. The moving image decoding device 2001 encodes the image to be decoded for each block, using the image at each time included in the encoded video as the image to be decoded.

First, the moving image decoding device 2001 checks whether or not the video to be decoded is an omnidirectional panoramic video (step 2101). When the video to be decoded is not an all-around panoramic video (steps 2101 and No), the moving image decoding apparatus 2001 decodes the decoding target block as it is (step 2102).

On the other hand, when the video to be decoded is an all-around panoramic video (steps 2101 and Yes), the extraction unit 2011 extracts global vector information from the header information (step 2105).

Next, the moving image decoding device 2001 checks whether or not the decoding target block is a block of the forward prediction picture (step 2106). When the block to be decoded is not a block of the forward prediction picture (steps 2106, No), the moving image decoding apparatus 2001 performs the processes after step 2102.

On the other hand, when the decoding target block is a block of the forward prediction picture (step 2106, Yes), the moving image decoding apparatus 2001 decodes the decoding target block (step 2107). Then, the change unit 2012 returns the position of the decoded block image to the position of the coded block before the change based on the global vector (step 2108). At this time, the change unit 2012 generates a corrected decoded block image by shifting the position of the decoded block image by the size of the global vector in the direction opposite to the global vector, and writes it in the frame memory 2017.

Next, the moving image decoding device 2001 checks whether or not all the blocks in the image to be decoded have been decoded (step 2103). When a block that has not been decoded remains (steps 2103 and No), the moving image decoding apparatus 2001 repeats the processes after step 2101 with the next block as the decoding target block.

When all the blocks are decoded (step 2103, Yes), the moving image decoding apparatus 2001 checks whether or not all the images included in the video to be decoded have been decoded (step 2104). When the undecrypted image remains (step 2104, No), the moving image decoding device 2001 repeats the processes after step 2101 with the next image as the image to be decoded. Then, when all the images are decoded (steps 2104, Yes), the moving image decoding apparatus 2001 ends the process.

Next, the operation when the moving image coding device 1014 of FIG. 11 corresponds to the moving image coding device 601 of FIG. 6 will be described. In this case, the change unit 1101 and the determination unit 1102 correspond to the correction unit 613, and the determination unit 1103 corresponds to the determination unit 612. The subtraction unit 1105, T / Q1106, ENT1107, IQ / IT1108, addition unit 1109, motion compensation unit 1110, and prediction image generation unit 1111 correspond to the coding unit 614, and the frame memory 1112 corresponds to the storage unit 611. ..

When the photographing unit 1011 of FIG. 10 is moving, it is assumed that the photographing range of the panoramic image of the entire circumference changes with the passage of time, and the subject in the image also changes. In this case, even if the entire panoramic image included in the omnidirectional panoramic image is converted into a developed panoramic image, it is difficult to capture it as one moving image. Therefore, as shown in FIG. 3, it is effective to extract a partial region from each panoramic image and convert it into a rectangular shape to generate a moving image of the extracted partial region.

In this case, the compositing unit 1012 can generate a coded panoramic image by combining a plurality of images captured by a plurality of image pickup devices included in the photographing unit 1011. The unfolded panoramic image generated by the unfolding unit 1013 is input to the moving image coding device 1014 as a coded target panoramic image.

The determination unit 1103 uses the motion vector output from the motion compensation unit 1110 to determine a global vector representing the amount of deviation of the coded block with respect to the reference image stored in the frame memory 1112. Then, the determination unit 1103 outputs the determined global vector to the change unit 1101.

For example, the determination unit 1103 can determine the motion vector of a specific block in the coded panoramic image as a global vector. In this case, the motion compensation unit 1110 performs motion estimation for all the blocks in the coded panoramic image in advance, obtains the motion vector of each block, and outputs the motion vector to the determination unit 1103. Then, the determination unit 1103 selects a motion vector of a specific block and determines it as a global vector.

By using the motion vector of a specific block as a global vector, the position of the partial region extracted from the coded panoramic image can be shifted according to the motion of the block. As a result, the image of the partial area displayed on the screen is corrected so that the subject in the specific block always appears in the same position on the screen, so that it is possible to generate an image that follows the specific subject. It will be possible.

For example, when the moving image coding device 1014 includes a display device equipped with a touch panel, the determination unit 1103 obtains the address of the block to which the position touched by the user in the screen belongs, and determines a specific block using the address. can do. When the moving image coding device 1014 does not include the touch panel, the address of the block can be calculated from the coordinates indicating the position in the screen, and the specific block can be specified by using the calculated address.

For example, if the block size is 16 × 16 and the coordinates of the pixels specified in the screen are (x, y), the block address is [x / 16, y / 16]. If the block size is 64 × 64 and the coordinates of the pixels specified in the screen are (x, y), the block address is [x / 64, y / 64].

FIG. 22 is a flowchart showing a specific example of the moving image coding process when the moving image coding device 1014 corresponds to the moving image coding device 601. The moving image coding device 1014 encodes the coded target image block by block, using the image at each time included in the coded target image as the coded target image.

When the coded image is an omnidirectional panoramic image, the moving image coding apparatus 1014 targets one or a plurality of partial areas extracted from the coded panoramic image, and displays the image of the partial area block by block. Encode.

First, the change unit 1101 checks whether or not the coded image is an omnidirectional panoramic image (step 2201).

When the coded image is not an all-around panoramic image (steps 2201 and No), the changing unit 1101 outputs the coded block as it is to the subtraction unit 1105 and the motion compensation unit 1110. Then, the moving image coding device 1014 encodes the coded block (step 2202).

On the other hand, when the coded target image is an omnidirectional panoramic image (step 2201, Yes), the determination unit 1103 determines the global vector of the coded target panoramic image (step 2205).

Next, the changing unit 1101 sets a new coding target block by shifting the position of the coding target block in the direction opposite to the global vector by the size of the global vector in the coded panoramic image. (Step 2206). Then, the moving image coding device 1014 encodes the set new coded target block (step 2207). By shifting the position of each block in the subregion in the direction opposite to the global vector, the starting point of the subregion can be changed to generate the corrected subregion.

Next, the change unit 1101 checks whether or not all the blocks in the image to be encoded or the partial region are encoded (step 2203). When the unencoded block remains (step 2203, No), the moving image coding apparatus 1014 repeats the processing after step 2201 with the next block as the coding target block.

When all the blocks are encoded (step 2203, Yes), the changing unit 1101 checks whether or not all the images included in the coded target video are encoded (step 2204). When an unencoded image remains (step 2204, No), the moving image coding apparatus 1014 repeats the processes after step 2201 with the next image as the image to be encoded. Then, when all the images are encoded (step 2204, Yes), the moving image encoding device 1014 ends the process.

In the moving image coding process of FIG. 22, unlike the moving image coding process of FIG. 15, the process of returning the position of the decoded block image to the original position based on the global vector is not performed. Further, even in the moving image decoding device, the processing of returning the position of the decoded block image to the original position is not performed, so that the moving image coding device 1014 does not need to transmit the global vector information to the moving image decoding device. ..

According to the moving image coding process of FIG. 22, even when the shooting range of the image of each of the plurality of imaging devices changes due to the movement of the shooting unit 1011, the coding is performed according to the movement of a specific block. The starting point of the partial area extracted from the target panoramic image can be changed. This makes it possible to suppress the motion vector information generated for each block in the partial region, and it is possible to perform appropriate motion estimation for each block. Therefore, ME efficiency and coding efficiency are improved.

The configuration of the moving image coding device of FIGS. 4, 6 and 11 is only an example, and some components may be omitted or changed depending on the use or conditions of the moving image coding device. For example, in the moving image coding device 1014 of FIG. 11, when entropy coding is not performed, ENT1107 can be omitted.

The configurations of the moving image decoding apparatus shown in FIGS. 8 and 20 are merely examples, and some components may be omitted or changed depending on the use or conditions of the moving image decoding apparatus. The configuration of the panoramic video coding system of FIG. 10 is only an example, and some components may be omitted or changed depending on the use or conditions of the panoramic video coding system. For example, when the photographing unit 1011 includes only one image pickup device, the compositing unit 1012 can be omitted.

The flowcharts shown in FIGS. 5, 7, 9, 15 to 19, 21, and 22 are merely examples, and depending on the configuration or conditions of the moving image coding device or the moving image decoding device, one The processing of the part may be omitted or changed. For example, in the moving image coding process of FIG. 15, when the input coded image is limited to the omnidirectional panoramic image, the processes of steps 1501 and 1502 can be omitted. The determination unit 1103 does not need to perform the process of step 1505 for each block, and may perform the process of step 1505 only once for one coded panoramic image.

In the correction process of FIG. 17, the motion compensation unit 1110 calculates another index representing the difference between the reference panoramic image and the coded target panoramic image instead of the SAD, and the determination unit 1102 uses the index to code. It may be determined whether or not to correct the conversion target block.

In the moving image coding process of FIG. 18, when the input coded target image is limited to the omnidirectional panoramic image, the processes of steps 1801 and 1802 can be omitted. The determination unit 1103 does not need to perform the process of step 1805 for each block, and may perform the process of step 1805 only once for one coded panoramic image.

In the correction process of FIG. 19, the determination unit 1102 may calculate the coding efficiency by using another statistical value such as a median value and a maximum value instead of the average value of the quantization scale.

In the moving image decoding process of FIG. 21, when the input decoding target image is limited to the omnidirectional panoramic image, the processes of step 2101 and step 2102 can be omitted. The extraction unit 2011 does not need to perform the process of step 2105 for each block, and may perform the process of step 2105 only once for one image to be decoded.

In the moving image coding process of FIG. 22, when the input coded target image is limited to the omnidirectional panoramic image, the processes of step 2201 and step 2202 can be omitted. The determination unit 1103 does not need to perform the process of step 2205 for each block, and may perform the process of step 2205 only once for one coded panoramic image.

The developed panoramic images of FIGS. 1 to 3 are merely examples, and another developed panoramic image may be used depending on the configuration or conditions of the panoramic video coding system.

The omnidirectional panoramic image of FIGS. 12 to 14 is only an example, and the omnidirectional panoramic image changes depending on the subject existing in the shooting range. The moving direction of the subject in the coded panoramic image is not limited to the horizontal direction, but may be a vertical direction or an oblique direction. Even if the movement of the entire panoramic image to be encoded is vertical or diagonal, the moving image coding processing of FIGS. 5, 7, 15, 18 and 22 and the moving image decoding of FIGS. 9 and 21 are performed. Processing can be applied. Instead of the coded panoramic image at the time immediately before the coded panoramic image, the coded panoramic image at another time may be used as the reference panoramic image.

FIG. 23 shows a configuration example of an information processing device (computer) used as the moving image coding device of FIG. 4, FIG. 6, or FIG. 11, or the moving image decoding device of FIG. 8 or FIG. The information processing device of FIG. 23 includes a Central Processing Unit (CPU) 2301, a memory 2302, an input device 2303, an output device 2304, an auxiliary storage device 2305, a medium drive device 2306, and a network connection device 2307. These components are connected to each other by bus 2308.

The memory 2302 is, for example, a semiconductor memory such as a Read Only Memory (ROM), a Random Access Memory (RAM), or a flash memory, and stores a program and data used for a moving image coding process or a moving image decoding process. The memory 2302 can be used as the storage unit 411 of FIG. 4, the storage unit 611 of FIG. 6, the storage unit 811 of FIG. 8, the frame memory 1112 of FIG. 11, and the frame memory 2017 of FIG.

The CPU 2301 (processor) operates as a determination unit 412, a correction unit 413, and a coding unit 414 in FIG. 4 by executing a program using, for example, the memory 2302. The CPU 2301 also operates as the determination unit 612, the correction unit 613, and the coding unit 614 in FIG. 6 by executing the program using the memory 2302. The CPU 2301 also operates as the extraction unit 812, the decoding unit 813, and the correction unit 814 in FIG. 8 by executing the program using the memory 2302.

The CPU 2301 also operates as the change unit 1101, the determination unit 1102, the determination unit 1103, the change unit 1104, the subtraction unit 1105, the T / Q1106, and the ENT 1107 by executing the program using the memory 2302. The CPU 2301 also operates as an IQ / IT1108, an addition unit 1109, a motion compensation unit 1110, and a prediction image generation unit 1111.

The CPU 2301 also operates as the extraction unit 2011, the change unit 2012, the block decoding unit 2013, the addition unit 2014, the motion compensation unit 2015, and the prediction image generation unit 2016 in FIG. 20 by executing the program using the memory 2302. do.

The input device 2303 is, for example, a keyboard, a pointing device, or the like, and is used for inputting instructions or information from a user or an operator. The output device 2304 is, for example, a display device, a printer, a speaker, or the like, and is used for making an inquiry to a user or an operator and outputting a processing result. When the information processing device is a moving image decoding device, the processing result may be the restored panoramic image to be encoded.

The auxiliary storage device 2305 is, for example, a magnetic disk device, an optical disk device, a magneto-optical disk device, a tape device, or the like. The auxiliary storage device 2305 may be a hard disk drive or a flash memory. The information processing device can store programs and data in the auxiliary storage device 2305 and load them into the memory 2302 for use.

The medium driving device 2306 drives the portable recording medium 2309 to access the recorded contents. The portable recording medium 2309 is a memory device, a flexible disk, an optical disk, a magneto-optical disk, or the like. The portable recording medium 2309 may be a Compact Disk Read Only Memory (CD-ROM), a Digital Versatile Disk (DVD), or a Universal Serial Bus (USB) memory. The user or the operator can store the programs and data in the portable recording medium 2309 and load them into the memory 2302 for use.

As described above, computer-readable recording media that store programs and data used for processing include physical (non-temporary) recording media such as memory 2302, auxiliary storage device 2305, and portable recording medium 2309. A recording medium is included.

The network connection device 2307 is a communication interface that is connected to a communication network such as a Local Area Network (LAN) or the Internet and performs data conversion associated with the communication. When the information processing device is a moving image coding device, the network connection device 2307 can transmit a bit stream of the coded panoramic video to the moving image decoding device. When the information processing device is a moving image decoding device, the network connection device 2307 can receive a bit stream of the coded panoramic image from the moving image coding device.

The information processing device can also receive programs and data from an external device via the network connection device 2307 and load them into the memory 2302 for use.

It is not necessary for the information processing apparatus to include all the components shown in FIG. 23, and some components may be omitted depending on the intended use or conditions. For example, if the interface with the user or the operator is unnecessary, the input device 2303 and the output device 2304 may be omitted. If the information processing device does not access the portable recording medium 2309, the medium drive device 2306 may be omitted.

Although the embodiments of the disclosure and their advantages have been described in detail, those skilled in the art will be able to make various changes, additions and omissions without departing from the scope of the invention expressly described in the claims. Let's do it.

Further, the following appendices will be disclosed with respect to the embodiments described with reference to FIGS. 1 to 23.
(Appendix 1)
A storage unit that stores a reference panoramic image used for encoding a coded panoramic image obtained by expanding the panoramic image included in the panoramic image captured by the image pickup device.
A determination unit that determines a vector representing the amount of deviation of the coded panoramic image with respect to the reference panoramic image.
A correction unit that generates a corrected panoramic image to be coded by correcting the position of each of a plurality of areas to be coded in the panoramic image to be coded based on a vector representing the amount of deviation.
A coding unit that encodes an image of each of the plurality of coded target areas in the corrected panoramic image to be coded by using the reference panoramic image.
A moving image encoding device comprising.
(Appendix 2)
The appendix is characterized in that, among the motion vectors of each of the plurality of coded target regions before correction, the motion vector generated more than the other motion vectors is determined as the vector representing the deviation amount. 1 The motion image coding device according to 1.
(Appendix 3)
When the ratio of the number of vectors representing the deviation amount to the total number of motion vectors in the coded panoramic image before correction is larger than a predetermined value, the correction unit may perform the plurality of coded targets before correction. The moving image coding device according to Appendix 2, wherein the position of each region is corrected based on a vector representing the deviation amount.
(Appendix 4)
The correction unit obtains a first difference based on a first motion estimation using the reference panoramic image and the coded panoramic image before correction, and obtains the reference panoramic image and the corrected panoramic image to be coded. When the second difference is obtained based on the second motion estimation using and, and the second difference is smaller than the first difference, the position of each of the plurality of coded target regions before the correction represents the deviation amount. The moving image coding device according to Appendix 2, characterized in that correction is performed based on a vector.
(Appendix 5)
The correction unit obtains a quantization scale based on the first motion estimation using the reference panoramic image and the coded target panoramic image before correction, and the information amount of the code based on the first motion estimation. The quantization scale based on the second motion estimation using the reference panoramic image and the corrected panoramic image to be encoded, and the information amount of the code based on the second motion estimation are obtained, and the second motion estimation is performed. The product of the quantization scale based on the above and the amount of code information based on the second motion estimation is smaller than the product of the quantization scale based on the first motion estimation and the amount of code information based on the first motion estimation. In this case, the moving image coding apparatus according to Appendix 2, wherein the position of each of the plurality of coded target regions before the correction is corrected based on a vector representing the deviation amount.
(Appendix 6)
A storage unit that stores a reference image used for encoding a coded panoramic image included in a panoramic image that is a combination of a plurality of images taken by a plurality of image pickup devices.
When the coding target area in the coding target panoramic image shifts with respect to the reference image due to the movement of the shooting range of each of the plurality of images, the amount of deviation of the coding target area with respect to the reference image is calculated. The decision part that determines the vector to be represented, and
A correction unit that generates a corrected coded target area by correcting the position of the coded target area in the coded target panoramic image based on a vector representing the deviation amount.
A coding unit that encodes the corrected image of the coded target area using the reference image, and a coding unit.
A moving image encoding device comprising.
(Appendix 7)
A moving image decoding device that decodes a coded panoramic image including a coded panoramic image generated by encoding a coded panoramic image obtained by expanding the panoramic image included in the panoramic image taken by the image pickup device. ,
A storage unit for storing a reference panoramic image used for decoding the coded panoramic image, and a storage unit.
An extraction unit that extracts a vector representing the amount of deviation of the coded panoramic image with respect to the reference panoramic image from the coded panoramic image, and
A decoding unit that generates a decoded panoramic image by decoding each of a plurality of decoding target areas in the coded panoramic image using the reference panoramic image.
A correction unit that restores the coded panoramic image by correcting the position of each of the plurality of decoded panoramic images in the decoded panoramic image based on a vector representing the amount of deviation.
A moving image decoding device.
(Appendix 8)
Using the reference panoramic image used to encode the coded panoramic image obtained by developing the panoramic image included in the panoramic image taken by the image pickup apparatus, the amount of deviation of the coded target panoramic image with respect to the reference panoramic image is determined. Determine the vector to represent,
By correcting the position of each of the plurality of coded target areas in the coded target panoramic image based on the vector representing the deviation amount, the corrected coded target panoramic image is generated.
Each of the plurality of coded areas in the corrected panoramic image to be coded is encoded by using the reference panoramic image.
A video encoding program that causes a computer to perform processing.
(Appendix 9)
The computer is characterized in that, among the motion vectors of each of the plurality of coded target regions before correction, a motion vector that is generated more than the other motion vectors is determined as a vector representing the deviation amount. The described video coding program.
(Appendix 10)
With respect to the reference image used for encoding the coded panoramic image included in the panoramic image in which the plurality of images are combined by moving the shooting range of each of the plurality of images captured by the plurality of image pickup devices. When the coded area in the coded panoramic image is deviated, a vector representing the amount of deviating of the coded area with respect to the reference image is determined.
By correcting the position of the coded target area in the coded target panoramic image based on the vector representing the deviation amount, the corrected coded target area is generated.
The image of the corrected coded area is encoded by using the reference image.
A video encoding program that causes a computer to perform processing.
(Appendix 11)
A computer that decodes a coded panoramic image, including a coded panoramic image generated by encoding a coded panoramic image obtained by expanding the panoramic image included in the panoramic image taken by the image pickup device.
A vector representing the amount of deviation of the coded panoramic image with respect to the reference panoramic image used for decoding the coded panoramic image is extracted from the coded panoramic image.
Each of the plurality of decoding target areas in the coded panoramic image is decoded using the reference panoramic image to generate a decoded panoramic image.
The coded panoramic image is restored by correcting the position of each of the plurality of decoded panoramic images in the decoded panoramic image based on the vector representing the deviation amount.
A moving image decoding program that executes processing.
(Appendix 12)
The computer
Using the reference panoramic image used to encode the coded panoramic image obtained by developing the panoramic image included in the panoramic image taken by the image pickup apparatus, the amount of deviation of the coded target panoramic image with respect to the reference panoramic image is determined. Determine the vector to represent,
By correcting the position of each of the plurality of coded target areas in the coded target panoramic image based on the vector representing the deviation amount, the corrected coded target panoramic image is generated.
Each of the plurality of coded areas in the corrected panoramic image to be coded is encoded by using the reference panoramic image.
A moving image coding method characterized by the above.
(Appendix 13)
The computer is characterized in that, among the motion vectors of each of the plurality of coded target regions before correction, a motion vector that is generated more than the other motion vectors is determined as a vector representing the deviation amount. The described moving image coding method.
(Appendix 14)
The computer
With respect to the reference image used for encoding the coded panoramic image included in the panoramic image in which the plurality of images are combined by moving the shooting range of each of the plurality of images captured by the plurality of image pickup devices. When the coded area in the coded panoramic image is deviated, a vector representing the amount of deviating of the coded area with respect to the reference image is determined.
By correcting the position of the coded target area in the coded target panoramic image based on the vector representing the deviation amount, the corrected coded target area is generated.
The image of the corrected coded area is encoded by using the reference image.
A moving image coding method characterized by the above.
(Appendix 15)
A computer that decodes a coded panoramic image, including a coded panoramic image generated by encoding a coded panoramic image obtained by expanding the panoramic image included in the panoramic image taken by the image pickup device,
A vector representing the amount of deviation of the coded panoramic image with respect to the reference panoramic image used for decoding the coded panoramic image is extracted from the coded panoramic image.
Each of the plurality of decoding target areas in the coded panoramic image is decoded using the reference panoramic image to generate a decoded panoramic image.
The coded panoramic image is restored by correcting the position of each of the plurality of decoded panoramic images in the decoded panoramic image based on the vector representing the deviation amount.
A moving image decoding method characterized by this.

101, 201, 202, 301, 1301 to 1303, 1401, 1402 Panorama image 102 Borderline 103, 203, 204, 321 to 323 Expanded panoramic image 205 All-around panoramic image 311 to 314 Regions 401, 601 and 1014 Dynamic image coding Equipment 411, 611, 811 Storage unit 412, 612, 1103 Determination unit 413, 613, 814 Correction unit 414, 614 Coding unit 421, 821, 1201 Reference panoramic image 621 Reference image 801, 2001 Video decoding device 812, 2011 Extraction Part 813 Decoding part 1001 Panorama video coding system 1011 Imaging part 1012 Synthesis part 1013 Expansion part 1101, 1104, 2012 Change part 1102 Judgment part 1105 Subtraction part 1109, 2014 Addition part 1110, 2015 Motion compensation part 1111, 2016 Predictive image generation part 1112, 2017 Frame memory 1202, 1203 Codified panoramic image 1211 Motion vector 2013 Block decoding unit 2301 CPU
2302 Memory 2303 Input device 2304 Output device 2305 Auxiliary storage device 2306 Media drive device 2307 Network connection device 2308 Bus 2309 Portable recording medium

Claims (6)

A storage unit that stores a reference image used for encoding a coded panoramic image included in a panoramic image that is a combination of a plurality of images taken by a plurality of image pickup devices.
When the coded panoramic image deviates from the reference image due to the movement of the shooting range of each of the plurality of images , the motion vector of each of the plurality of coded areas in the coded panoramic image. A determination unit that selects a motion vector of a specific coding target region from among the motion vectors and determines the motion vector of the specific coding target region as a vector representing the amount of deviation of the plurality of coding target regions with respect to the reference image. When,
A correction unit that generates a corrected coded target area by correcting the position of each of the plurality of coded target areas in the coded target panoramic image based on a vector representing the deviation amount.
A coding unit that encodes the corrected image of the coded target area using the reference image, and a coding unit.
A moving image encoding device comprising. The moving image coding device according to claim 1, wherein the specific coding target area is a coding target area corresponding to a position designated by a user among the plurality of coding target areas. With respect to the reference image used for encoding the coded panoramic image included in the panoramic image in which the plurality of images are combined by moving the shooting range of each of the plurality of images captured by the plurality of image pickup devices. When the coded panoramic image is displaced , a motion vector of a specific coded area is selected from the motion vectors of each of the plurality of coded areas in the coded panoramic image.
The motion vector of the specific coding target region is determined as a vector representing the deviation amount of the plurality of coding target regions with respect to the reference image.
By correcting the position of each of the plurality of coded target areas in the coded target panoramic image based on the vector representing the deviation amount, the corrected coded target area is generated.
The image of the corrected coded area is encoded by using the reference image.
A video encoding program that causes a computer to perform processing. The moving image coding program according to claim 3, wherein the specific coding target area is a coding target area corresponding to a position designated by a user among the plurality of coding target areas. The computer
With respect to the reference image used for encoding the coded panoramic image included in the panoramic image in which the plurality of images are combined by moving the shooting range of each of the plurality of images captured by the plurality of image pickup devices. When the coded panoramic image is displaced , a motion vector of a specific coded area is selected from the motion vectors of each of the plurality of coded areas in the coded panoramic image.
The motion vector of the specific coded area is determined as a vector representing the amount of deviation of the plurality of coded areas with respect to the reference image.
By correcting the position of each of the plurality of coded target areas in the coded target panoramic image based on the vector representing the deviation amount, the corrected coded target area is generated.
The image of the corrected coded area is encoded by using the reference image.
A moving image coding method characterized by this. The moving image coding method according to claim 5, wherein the specific coding target area is a coding target area corresponding to a position designated by a user among the plurality of coding target areas.

Images