JPH06269025A - Coding system for multi-eye stereoscopic video image - Google Patents

Abstract

PURPOSE:To improve the coding efficiency by calculating a video signal from other camera based on a video signal from a predetermined camera in a multi- eye stereoscopic video signal and correcting the video signal from each camera based on the correction value. CONSTITUTION:A multi-eye camera picks up an object and a block matching section 1 takes block matching to picture data F0-Fn outputted from each camera based on a camera KO. Then a correction value calculation section 2 receiving block matching result information calculates a correction value representing relative position deviation between the reference camera KO and each of the cameras K1-Kn. Then correction sections 3-0 to 3-n correct picture data F1-Fn except reference picture data FO and a coding section 4 makes coding and a multiplexer section 5 applies multiplex processing to the data and the result is sent to a transmission line.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a coding system, and more particularly to a system for coding multi-view stereoscopic video data with high efficiency.

It is necessary to correct the position distortion of an image caused by uneven positions of cameras for outputting stereoscopic images of a binocular or more multi-lens type.

[0003]

2. Description of the Related Art FIG. 20 shows a conceptual configuration of a system for producing a multi-view stereoscopic image which was already published at NHK Technical Research Institute in 1992. First, a stationary or moving subject (octopus) is shown. ) 100 is photographed by a plurality of camera groups 110 whose positions are slightly shifted vertically and horizontally. Next, the image data obtained from the plurality of camera groups 110 is subjected to high-efficiency coding by the encoder 120, multiplexed by the multiplexing unit 130, and then transmitted via the transmission line 140 or the like, and the separation unit 140 on the receiving side. Demultiplexing is performed and then decoding is performed, and the result is displayed on the display 160.

As the output-side display 160, for example, a lenticular lens (when there is parallax only in the horizontal direction) or a fly's eye lens (when there is parallax in the vertical and horizontal directions) is used.

As an example, FIG. 21 shows an example in which the output from each of the five vertical and five horizontal cameras when the octopus is the subject is displayed in an easy-to-understand manner. In this example,
There is also parallax in the vertical direction.

As described above, the use of a plurality of cameras whose positions are slightly shifted vertically and horizontally allows the stereoscopic vision to be performed by forming the binocular parallax using the output from one camera as the input to one eye. In addition, when a large number of cameras are used, a person who looks at the display 160 in the output system can obtain a natural stereoscopic view even if the person shakes his / her head.

As described above, the multi-view stereoscopic image data is expected to be used in various ways as it gives a natural stereoscopic image. Therefore, the encoding system is a highly efficient encoding system which requires a small transmission capacity. There is a need to.

[0008]

As described above, when shooting is performed using the multi-lens type stereoscopic system, when using a two-lens to multi-lens type integrated camera, a single camera (monocular) is used.
When used in combination, there are two.

Even in the case of the integrated type, there are many cases where a single camera is fixed by screws, wires or the like to form the integrated type.

As described above, in the case where a single camera is combined or fixed to form an integral type, when considering one camera as a reference, the other camera body has a tilt different from the parallax. In many cases, stereoscopic viewing was difficult.

For example, when two cameras are combined to capture a stereoscopic image of a twin-lens type, it is difficult to correct the position, and the lines may be vertically displaced by several lines or may be rotated. It happened often.

In such a case, fine adjustment is troublesome, and even if a few lines are vertically displaced or a slight rotation is applied, not only is it difficult to visually perceive stereoscopic vision, but eye strain is also caused. It is liable to occur, and when the high-efficiency encoding is performed by correlating the input images from the cameras with each other, the encoding characteristic is deteriorated.

Therefore, an object of the present invention is to realize a system capable of highly efficiently encoding multi-view stereoscopic image data in which positional deviation between cameras is corrected.

[0014]

Means and Actions for Solving the Problem As means for solving the above problems, a certain camera is used as a reference and block matching with another camera is performed. From the block matching result, the correction amount indicating the positional deviation between the cameras is calculated. The image data from each camera is corrected according to the result. It is possible.

Therefore, in the multi-view stereoscopic video coding system according to the present invention, as shown in principle in FIG. 1, a predetermined one of the multi-view stereoscopic video signals F1 to Fn from a plurality of cameras is selected. A block matching unit 1 that performs block matching with a video signal from another camera on the basis of a video signal from a camera, and a correction value indicating a shift between the video signals is calculated from the matching result information in the block matching unit 1. A correction value calculation unit 2, a correction unit 3 that corrects a video signal from each camera based on the correction value, an encoding unit 4 that encodes an output signal of each correction unit 3, and each of the encoding units 4. And a multiplexing unit 5 that multiplexes output signals and sends them to a transmission line.

In operation, first, a subject is photographed by a multi-lens camera, and image data F0 to F output from each camera.
For n, the block matching unit 1 performs block matching with a certain camera K0 as a reference.

Then, the correction value calculation unit 2 which has received the block matching result information from the block matching unit 1 calculates a correction value indicating a relative positional deviation between the reference camera K0 and each of the cameras K1 to Kn.

Thereafter, the correction units 3-0 to 3-n correct the respective image data F1 to Fn except the reference image data F0, and thereafter, the coding unit 4 codes the data and the multiplexing unit 5 performs the multiplexing process. And send it to the transmission line.

In the above-described present invention, the correction value calculation unit 2 can calculate the correction value in the vertical direction by using the vector value in the vertical direction of the matching result information.

Further, the correction value calculation unit 2 can refer to the frequency of occurrence of the vector value in the vertical direction and use the vector value generated most as the correction value.

Further, the correction value calculation unit 2 may calculate the average value of the vector values in the vertical direction and use it as the correction value.

Further, the correction value calculation unit 2 detects that only the reference camera K0 is deviated by determining that the difference between the correction values is within the threshold value, and the correction for the reference camera K0 is performed. The value is calculated and each correction unit 3-0 to 3-
The image data may be corrected with n.

Further, the correction value calculation unit 2 is configured to detect the reference camera K0.
The average value of all correction values may be taken as the correction value for.

Further, the correction value calculation unit 2 may calculate the same correction value not only in the vertical direction as described above but also in the horizontal direction.

Further, the correction value calculation unit 2 can detect the rotation angle of each pixel of the vector of the matching result information and calculate the correction value by coordinate conversion.

[0026]

DESCRIPTION OF THE PREFERRED EMBODIMENTS The "correction value calculation" in the multi-view stereoscopic video encoding system according to the present invention shown in FIG. 1 will be described in detail with reference to the first to fourth embodiments described below. 2
The case of the multi-view type can be regarded as a multi-view type subset in terms of method, and thus the multi-view type will be described.

(1) First Embodiment: FIGS. 2 to 11 [Principle] In this embodiment, when another camera group is vertically offset with respect to a certain predetermined camera To realize the correction of. In addition, since the left and right directions often move with parallax, there is not much point in correcting,
Not done here.

2 and 3 show an algorithm in which the block matching unit 1 and the correction value calculation unit 2 according to the first embodiment are combined. As shown in FIG. 4, the cameras are arranged in a grid pattern ( It is assumed that they are arranged (not necessarily at equal intervals).

Output image data from a reference camera (for example, K0) is defined as F0 (704, 480), and output image data from a camera to be corrected is defined as Fn (704, 480). However, n exists as many as the number of cameras based on the image data F0, and “704” and “4”
“80” indicates an effective pixel area of one screen as shown in the image format of FIG. 5 (1).

Further, a histogram as will be described later and shown in FIG. 6 is set to H (-15; 15) (initial value is all "0"), and the vectors obtained by block matching are Vnx (44,30) and Vny (44, 30) (step S1 in FIG. 2).

Using the image data F0 as a reference, block matching with image data Fn of another camera is performed by a known method (at step S2).

This block matching algorithm is shown in the flowchart of FIG. 7. In this example, two cameras are used, the output image data of the camera K0 is F0, and the output image data of the camera Kn is Fn. ,
As shown in FIG. 8, the screen is divided into blocks of 16 × 16 pixels (44 in the x direction and 30 in the y direction), and the coordinates of the upper left pixel of the block for which block matching is performed are (x
₀ , y ₀ ).

Then, the search is performed in ± in both the x direction and the y direction.
It is performed in the range (VI, VJ) for 15 pixels (steps S31 to S33 in FIG. 7). As an initial value, A = 999999
(Maximum possible value), Vx = Vy = 16 is given.

A value Ao obtained by accumulating the absolute value of the difference between the two images as an absolute value error (or squared error) of the evaluation function for the entire block is determined (at step S34), and the cumulative value is the smallest and the vector size is the same. A vector having a small (sum of squares) is selected (at steps S35 to S41), and finally remaining vectors Vx and Vy are obtained (at step S42).

In this way, by performing block matching between one block for all blocks in one screen, the vector Vnx (44, 30) in the x direction and the vector in the y direction in the camera Kn with respect to the reference camera K0. Vny (44,30) can be calculated (at step S42).
And step S2 of FIG. 2).

Now, in order to correct the vertical shift,
Only the vertical vector Vny is used. The number of vectors (-15 to 15) obtained within the search range of ± 15 pixels of the vertical vector Vny is recorded in the frame as a histogram H as shown in FIG. (S3 to S10). The total number of vectors that take this histogram is 30 (vertical) × 44 (horizontal) = 1
It will be 320 pieces.

Then, the value of the most generated vector in the histogram is set as Uy, and this Uy is set as the correction amount (S11 to S14).

That is, a correction value Uy is given to each of the correction units 3-0 to 3-n so as to move the image data Fn to be corrected from the camera Kn vertically by Uy as a whole (at the same step S).
15).

When the correction vector thus obtained is newly used as a parallax compensation vector, the elements of the array of Vny (44, 30) (in the y direction of the entire screen are arranged for each block by a known method). Uy may be subtracted from the vector to obtain the parallax compensation vector (S16 to S16).
22).

As described above, the normal image data signal has the format shown in FIG. 5 (1),
Only the effective pixel area is displayed on the normal screen, and the matching is performed only in the effective pixel area.

Therefore, as shown in FIG. 5B after the correction, even if there is a portion where the image data information does not exist,
Inserting a dummy signal will not damage the image data.

Circuit block example of the first embodiment: FIG . 9 The circuit block of FIG . 9 shows the correction value calculation unit 2 and the correction unit 3 (collective names of the correction units 3-0 to 3-n). Then, the vector in the vertical direction (y direction) obtained from the block matching unit 1 is read by the ROM or the switch 11 from −15 to −15.
It is determined which of the 15 vectors corresponds to, the counts are input to the respective counters 12-1 to 12-30 to count, and the most one of these count values is the comparator 1
After being selected in 3, the decoder 14 converts it into a correction value Uy.

Then, the difference between this correction value Uy and the above vector is obtained to be a parallax compensation vector, and the correction value Uy is given to the variable delay unit 3 as a correction unit so that the camera image data Fn is the correction value Uy only. Variable delay.

Block Matching Circuit Example: FIG. 10 The block matching algorithm shown in the flow chart of FIG. 7 can also be realized by the circuit shown in FIG. 10, and the subtracter 15 and the adders 16, 23, 27 are used.
, Delay device 17, comparators 18 and 24, selector 1
9, 28, 29, decoder 20, and squarers 21, 2
2, 25, 26.

Modification of the First Embodiment: FIG. 11 In this modification, instead of taking the histogram H and using the maximum y-direction vector, the average value of the vector Vny in the y (vertical) direction of the entire screen is calculated ( Steps S23 and S2
4), which is corrected as a correction value Uy.

(2) Second Embodiment: FIGS. 12 and 13 [Principle] In the first embodiment, the image data F0 from a certain camera K0 is used as a reference, and the other cameras K1 to Kn are used.
I tried to correct the image data F1 to Fn from
Actually, only the reference image data F0 is the image data F1 to F1.
There may be a vertical shift as compared with n, and this shift is corrected in the second embodiment.

In FIG. 12, reference image data F0 (704, 480) of the camera K0, image data F1 (704, 480) of the camera K1 to be corrected, and image data F2 (704) of the camera K2 to be corrected. , 480),
The image data of the camera K3 to be corrected is set to F3 (704, 4
80).

Further, the vector obtained by block matching between these cameras is represented by {V1x (44,3
0), V1y (44,30)}, {V2x (44,3)
0), V2y (44,30)}, {V3x (44,3)
0), V3y (44, 30)} (step S51 in FIG. 12).

First, it is assumed that the block matching (FIGS. 7 and 10) in the first embodiment is performed (this is omitted), and the vector obtained by this is used as the image data F.
In the case of 0 and F1 = vector V1x (44,30), V1
y (44,30), image data F0 and F2 = vector V2x (44,30), V2y (44,30), image data F0 and F3 = vector V3x (44,3)
0), V3y (44, 30) (S52).

Next, also by the steps S3 to S15 in the above-mentioned first embodiment, the above-mentioned vector V1y is obtained.
The correction value U1y and the vector V2y (4
4, 30) to the correction value U2y and the vector V3y (44,
The correction value U3y is obtained from 30) (at step S53).

Then, here, the correction values U1y, U2y, U
Check if 3y are similar to each other. That is, the absolute value of the difference between the correction values U1y and U2y is smaller than the threshold value TH1,
Further, it is checked whether or not the absolute value of the difference between U1y and U3y is smaller than the threshold value TH2 (at step S54). Is determined to be inclined,
The correction value U0y of the reference camera K0 is calculated (S
55).

The correction value U0y is the correction values U1y and U2.
y and U3y are calculated as an average value, and the correction value U1 is calculated.
y, U2y, U3y are the correction values U1 obtained as described above.
The correction values U0y are subtracted from y, U2y, and U3y, respectively, and corrected (S55).

The correction value thus obtained is sent to each of the correction units 3-0 to 3-n to correct the image data from each camera (at step S57).

If the absolute value of the difference between the correction value U1y and the correction value U2y is larger than the threshold value TH1 or the absolute value of the difference between the correction value U1y and the correction value U3y is larger than the threshold value TH2,
With the correction value U0y = 0, the output image data of each camera is corrected as in the first embodiment (S56).

Circuit block example of the second embodiment: FIG. 13 A circuit block diagram of the second embodiment is shown in FIG. 13, and the circuit block shown in FIG. 9 and FIG. Three are prepared as -1 to 2-3, absolute values of the correction values U1y, U2y, and U3y obtained from these are calculated by absolute value calculation units 31 and 32, and absolute values of the differences are calculated. TH2 is compared with each of the comparators 33 and 34, the determination corresponding to step S54 is performed by the decoder 35, and the average value obtained by the average value calculation unit 36 or "0" is obtained.
Is selected by the selector 37, a simple circuit for outputting the correction value U0y is added.

MODIFIED EXAMPLE Although the first and second embodiments described above detect and correct the vertical displacement, it is also possible to detect and correct the lateral (lateral) displacement as a modified example. is there. In this case, in the flowchart, the vector in the y direction is detected and the correction value is calculated in the first and second embodiments, whereas the vector in the x direction is detected and the correction value is calculated. It is only described differently and will not be described.

(3) Third Embodiment: FIG. 14 to FIG. 19 [Principle] In this embodiment, a shift occurs when a force such as a twist is applied to the output image of the camera to be corrected and the image is rotated. The present invention deals with the detection / correction method of (1), and shows an embodiment of a method of detecting a rotation angle and performing correction by coordinate conversion using a matrix.

The flowcharts shown in FIGS. 14 to 16 show an embodiment in which the correction value calculation unit 2 shown in FIG. 1 and the correction units 3-0 to 3-n are combined. First, the reference camera is used. The image data of K0 is F0 (704,480), and the image data of the camera Kn to be corrected is Fn (704,48).
0), the image data F0, and the vector obtained by the block matching of the image data Fn are Vnx (44,3
0), Vny (44, 30), and rotation parameters corresponding to each vector are V'11x to V'230x, V'11y to V'244y,
Threshold values THx, THy, check array Check (70
4, 480) (initial values are all "0") (Fig. 1
4 step S61).

First, according to the first embodiment described above, the vectors Vnx (44, 30) and Vny (44, 3) are obtained by block matching between the image data F0 and the image data Fn.
0) is obtained (at step S62).

Next, at step S63, this image data F
Rotation parameters V'11x to V'230x and V'11y to indicate how much n is rotated with respect to the reference image data F0.
Calculate the value of V'244y. These rotation parameters are
This is an element for obtaining the approximate rotation angle (tan θ) of the vector when the center of the screen is the origin.

That is, the rotation parameters V'11x to V'230x,
V′11y to V′244y are all vectors of the central portions in the x direction and the y direction (blocks 22 to 23 in the x direction and blocks 15 to 16 in the y direction) indicated by diagonal lines in FIG. 17 showing one screen, respectively. It is normalized by the coordinates away from the center.

Next, such a rotation parameter V'11x-
Average values (rotation parameters per block) Vax and Vay of V'230x and V'11y to V'244y are obtained (step S64 in FIG. 15).

Further, each rotation parameter V'11x-
The average value (dispersion value) V _VAR y and V _VAR x of the absolute value error between V ′ 230x, V ′ 11y to V ′ 244y and the average value Vax and Vay is calculated (the same S65), and these average values V _VAR y and V _VAR
It is determined whether x is smaller than the threshold values THx and THy (S66 of the same), and V _VAR y> THx or V _VAR x>.
If THy, the variance is large and it is determined that the rotation parameters are not in the same direction (not rotating) and the routine exits (S67). However, V _VAR y <THx and V
_{If VAR} x <THy, it is determined that the dispersion is small and the rotation parameters are in the same direction (rotating), and the next correction is performed.

That is, with the average value of Vax and Vay as the overall average value Va, the rotation angle θ = tan ⁻¹ V shown in FIG.
a is obtained (at step S68), according to a well-known formula (at step S69),
The coordinates Fn (x, y) of the image data before correction are converted to the new coordinates (X, Y), and the image data after correction is converted to F.
(704, 480) (at step S70).

At this time, it should be noted that
The above-mentioned coordinate conversion causes pixels having no converted numerical value to appear.

Although the dummy value is inserted in FIG. 5 (2), in this embodiment, the coordinates A, B, C, D (where the coordinates after the coordinate conversion have no numerical values as shown in FIG. It is determined whether or not it belongs to the area outside the effective pixel area) (see FIG.
Steps S71 to S74), if the values do not belong to the parts A to D, the value is output as it is (S75 to S82, S).
84 to S87) and belong to portions A to D, the average value of the pixel values from above, below, left and right is forcibly set as the pixel value (S83).

Circuit block example of the third embodiment: FIG. 18 In this circuit example, the output signals Vx and Vy of the block matching circuit 1 shown in FIG. 10 are supplied to counters 41 and 42, accumulators 43 and 44, and divider 45. , 46 to obtain the average values Vax, Vay, and the decoder 49 controls the correction circuit 50 based on the comparison result in the comparators 47, 48 to output the correction value F (x, y) which is coordinate-converted. To do.

FIG. 19 shows a circuit example of the above-mentioned correction circuit 50, in which the average values Vax and Vay are input and tan is inputted.
The steps S68 to S87 of FIGS. 15 and 16 are executed by using trigonometric functions such as ⁻¹ , sin, and cos.

Note that once these values are obtained, there is no need to recalculate them in the subsequent frame.

Further, in each of the above embodiments, if the position of the camera does not change frequently, once the correction value is calculated, only the correction processing need be executed.

[0071]

As described above, according to the multi-view stereoscopic video encoding method of the present invention, the video signal from a predetermined camera among the multi-view stereoscopic video signals from a plurality of cameras is used as a reference. As a result, a correction value is calculated by performing block matching with the video signal from another camera, and the video signal from each camera is corrected based on the correction value, so that a visually natural stereoscopic vision can be achieved. Moreover, it is possible to improve the coding efficiency even when high-efficiency coding is performed using the correlation between the left and right eyes.

[Brief description of drawings]

FIG. 1 is a block diagram showing the principle of a multi-view stereoscopic image encoding method according to the present invention.

FIG. 2 is a flowchart diagram (No. 1) showing the first embodiment of the block matching unit and the correction value calculating unit in the multi-view stereoscopic image encoding system according to the present invention.

FIG. 3 is a flowchart diagram (No. 2) showing the first embodiment of the block matching unit and the correction value calculation unit in the multi-view stereoscopic video encoding system according to the present invention.

FIG. 4 is a layout view of a camera that outputs a multi-view stereoscopic image.

FIG. 5 is a diagram for explaining an effective pixel area.

FIG. 6 is a graph diagram for explaining a histogram used in the first embodiment of the multi-view stereoscopic image encoding method according to the present invention.

FIG. 7 is a flowchart showing block matching used in each embodiment of the multi-view stereoscopic video encoding method according to the present invention.

FIG. 8 is a schematic diagram for explaining block matching.

FIG. 9 is a circuit block diagram of a first embodiment of a multi-view stereoscopic image encoding system according to the present invention.

FIG. 10 is a block diagram showing an embodiment of a block matching circuit.

FIG. 11 is a flowchart showing a modified example of the first embodiment of the multi-view stereoscopic image encoding system according to the present invention.

FIG. 12 is a flowchart showing a second embodiment of the block matching unit and the correction value calculating unit in the multi-view stereoscopic image encoding system according to the present invention.

FIG. 13 is a circuit block diagram of a second embodiment of a multi-view stereoscopic image encoding system according to the present invention.

FIG. 14 is a flowchart diagram (No. 1) showing a third embodiment of the block matching unit and the correction value calculation unit in the multi-view stereoscopic image encoding system according to the present invention.

FIG. 15 is a flowchart diagram (No. 2) showing the third embodiment of the block matching unit and the correction value calculating unit in the multi-view stereoscopic image encoding system according to the present invention.

FIG. 16 is a flowchart diagram (No. 3) showing the third embodiment of the block matching unit and the correction value calculation unit in the multiview stereoscopic image encoding system according to the present invention.

FIG. 17 is a graph diagram for explaining rotation detection according to a third embodiment of a multi-view stereoscopic image encoding method according to the present invention.

FIG. 18 is a circuit block diagram of a third embodiment of a multi-view stereoscopic video encoding system according to the present invention.

FIG. 19 is a circuit block diagram of a correction circuit in a third embodiment of the multi-view stereoscopic image encoding system according to the present invention.

[Fig. 20] Fig. 20 is a block diagram illustrating a general configuration of a multi-view stereoscopic video encoding method.

[Fig. 21] Fig. 21 is a diagram illustrating an example of camera output of a multi-view system (5 eyes x 5 eyes).

[Explanation of symbols]

 1 block matching unit 2 correction value calculation unit 3 (3-0 to 3-n) correction unit 4 encoding unit 5 multiplexing unit K0 to Kn camera F0 to Fn output image data Show.

Claims (8)

[Claims] 1. A multi-view stereoscopic video signal from a plurality of cameras
A block matching unit (1) that performs block matching with a video signal from another camera based on a video signal from a predetermined camera among (F1 to Fn), and a matching result in the block matching unit (1) A correction value calculation unit (2) that calculates a correction value indicating the positional deviation between the cameras from the information, and a correction unit (3-0 to 3-n) that corrects the video signal from each camera based on the correction value. Encoding unit that encodes the output signal of each correction unit (3-0 to 3-n)
(4) and a multiplexing unit (5) that multiplexes each output signal of the encoding unit (4) and sends out to a transmission line, and a multi-view stereoscopic video encoding system characterized by the following: . 2. The multi-lens according to claim 1, wherein the correction value calculation unit (2) calculates the correction value in the vertical direction by using the vector value in the vertical direction of the matching result information. A stereoscopic video coding method. 3. The correction value calculation unit (2) refers to the frequency of occurrence of the vector values in the vertical direction, and sets the vector value generated most as the correction value. Multi-view stereoscopic video coding method. 4. The multi-view stereoscopic image according to claim 2, wherein the correction value calculation unit (2) calculates an average value of the vector values in the vertical direction and sets the average value as the correction value. Encoding method. 5. The correction value calculation unit (2) detects that only the reference camera is displaced by determining that the difference between the correction values is within a threshold value, and the reference camera 2. The multi-view stereoscopic video encoding method according to claim 1, wherein a correction value for is calculated. 6. The encoding system for multi-view stereoscopic video according to claim 5, wherein the correction value calculation unit (2) takes an average value of all correction values as a correction value for the reference camera. . 7. The correction value calculation unit (2) calculates the same correction value in the horizontal direction as well.
7. The encoding method for multi-view stereoscopic video according to any one of 1 to 6. 8. The correction value calculation unit (2) according to claim 1, wherein the correction value calculation unit (2) detects a rotation angle of each pixel of the vector of the matching result information and calculates a correction value by coordinate conversion. A multi-lens stereoscopic video coding method.