JPH1040420A - Method for controlling sense of depth - Google Patents

Abstract

PROBLEM TO BE SOLVED: To evade a sudden change in parallax angle at the time of processing a three-dimensional(3D) moving image by controlling sense of depth at the time of displaying a subject as a 3D moving image by executing conversion processing for easing a change in the depth sense factor of the subject in the time direction. SOLUTION: A 3D moving image generating device 30 is provided with a frame data storing part 32 having FIFO structure for successively storing a right eye picture inputted as frame data, a depth conversion part 38 for inputting depth information, a scene change detection part 40, and so on. In the structure, a right eye picture and depth information are received from a stereoscopic camera and a parallax angle based on the depth information is applied to generate a right eye picture. A moment sharply changing the depth is detected by the scene change detection part 40 and a change in the depth is moderated by the depth conversion part 38. Since the depth change is moderated, a change in the parallax angle is also moderated.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

The present invention relates to a depth control method. In particular, the present invention relates to a method for controlling depth, which is a means for giving a three-dimensional effect to a stereoscopic moving image when shooting, generating, displaying, or the like.

[0002]

2. Description of the Related Art One of VR (virtual reality) techniques is a three-dimensional display of an image. This is an image processing technique for producing a pseudo three-dimensional effect by manipulating images provided to the left and right eyes of a viewer. In order to generate such a stereoscopic image, a method of photographing a subject with a stereo camera is known.

FIG. 6 is a diagram showing the principle of generating a stereoscopic image by a stereo camera. In the figure, two cameras 2, 4
Constitute a stereo camera, which captures a subject 6, a car 8, and a mountain 10 as subjects. The angle generated when the two cameras 2 and 4 are connected to the subject is called a parallax angle, and the parallax angle when the person 6 is viewed is α in FIG. β and γ.
When the parallax angle is defined in this manner, the parallax angle becomes larger for a subject closer to the viewer, and theoretically the parallax angle becomes zero for a subject at infinity. This parallax angle gives a sense of depth to the subject.

Here, the sense of depth refers to the sense of perspective of a subject viewed from the eyes of a viewer. If the sense of depth is different for each subject or each part thereof, various perspectives are generated on the image, and a three-dimensional effect is generated. The factor that generates the depth sensation (hereinafter, referred to as “depth sensation factor”) is the above-described parallax angle, and there is a simple relationship between this and the distance from the viewer to the subject, that is, the depth. Let z be the depth, θ be the parallax angle, and d be the distance between the stereo cameras. If θ is small, then z = d / θ. Therefore, the depth z can be considered as a factor of the depth feeling. In the case of FIG. 6, the depths of the person 6, the car 8, and the mountain 10 are Z1, Z2, and Z3, respectively.

When a stereoscopic image is actually displayed, for example, the image of the left camera 2 is given to the left eye of the viewer, and the image of the right camera 4 is given to the right eye. Depending on the system, an image of one eye, for example, a right-eye image and a parallax angle or depth are set and sent to a display device, and a left-eye image is generated on the display device side based on the parallax angle or depth, and a right-eye image is generated. It may be displayed together with the image for use.

As described above, the viewer's eyes sense the perspective of the subject based on the depth perception factor such as the parallax angle. If the parallax angle suddenly changes many times, visual fatigue is promoted. It has been known. In particular, when a video scene is switched, a relatively large change in the parallax angle is likely to occur. For example, when moving to a distant scene after a vase scene just in front of the camera, the parallax angle suddenly decreases at the scene joint. In order to avoid a sudden change in the parallax angle, the following considerations have recently been made when creating a three-dimensional moving image.

[0007] 1. The cameraman continuously shoots subjects having similar parallax angles, that is, subjects having substantially equal depths. When the scene is changed from the vase to the distant view, the subject is once photographed so as to sandwich the scene of the subject at an intermediate distance.

[0008] 2. Instead of shooting freely, the cameraman rearranges the scenes in an order that does not increase the parallax angle during the subsequent video editing.

The necessity of considering the parallax angle at the time of a scene change is described in, for example, page 83 of “Basics of 3D video” (edited by NHK Science and Technical Research Laboratories edited by Takehiro Izumi, Ohmsha). It indicates the necessity to prevent a sudden change in depth. "

[0010]

However, the above-mentioned 1.2. Such a method depends largely on the know-how or experience of photographers and editors. Therefore, if there is no person having such know-how, there will be a problem in creating a stereoscopic image. Even if there is such a person, attention must be paid to shooting or editing, which requires labor and time. As a result, it has been recognized that there is a troublesome restriction on processing such as creation and display of a stereoscopic image in spite of a relatively simple photographing principle.

The present invention has been made in view of the above problems, and an object of the present invention is to provide a depth sense control method for avoiding a sudden change in parallax angle when processing a three-dimensional moving image. Also, the present invention is suitable not only when the change in the parallax angle occurs over the entire screen such as a scene change, but also when the change occurs on a part of the screen such as when a person suddenly enters the front of the screen. A depth control method.

[0012]

[Means for Solving the Problems]

(1) The depth sense control method of the present invention controls a sense of depth when a subject is displayed as a three-dimensional moving image by performing a conversion process for reducing a change in a factor of the depth sense of the subject in a time direction. It is.

[0013] Here, the "subject" means everything that falls within the image frame, and does not necessarily need to be photographed by a camera. In this specification, for example, it is assumed that a subject includes a computer graphics character and the like. The depth sensation factor is either the parallax angle or the depth, but may be considered to be the on-screen parallax described later. The “change in the depth sense factor of the subject” refers to, for example, when the scene changes from the person 6 to the mountain 10 in FIG.
To change. On the other hand, “the sense of depth when the subject is displayed as a three-dimensional moving image” refers to the sense of depth perceived by a person who has viewed the image. “Reducing the change in the time direction” means reducing the change that occurs in a certain period. For example, when the depth changes at the time of a scene change, the scene change itself occurs at a certain moment, and the temporal change rate of the depth becomes theoretically infinite. In the present invention, by extending the duration of the change to some extent, the depth change of the image is controlled while the change in the depth is reduced in the time direction.

Since the present invention controls the depth, it can be interpreted as "a method of producing a stereoscopic moving image in which the depth is controlled".

(2) In one aspect of the present invention, in the case of (1), the conversion process is performed in consideration of a time change rate of a depth sensation factor when the subject is displayed as a three-dimensional moving image. That is, consideration is given to, for example, performing a relaxation process after setting a certain upper limit so that the rate of change does not increase.

(3) In another aspect of the present invention, in the case of (1), the conversion processing is performed in consideration of the duration of the conversion processing. That is, if the conversion process takes a relatively long time, the effect of the relaxation becomes large, and thus such consideration is made.

(4) In one aspect of the present invention, near the switching point of the scene included in the three-dimensional moving image, the depth sense factor of one scene is gradually changed over the plurality of display unit images to the depth sense of the other scene. Perform a conversion process that approaches the factor.

Here, the "scene" is a unit of video in which the object to be photographed is fixed to some extent, and a scene or a cut is an example. The “display unit image” refers to an image serving as a display unit, such as a frame or a picture. For example, when the depth relating to the scene before switching is Z, and the depth relating to the scene after switching is Z ′, Z is displayed over a plurality of display unit images near the switching point, such as before switching, after switching, or during a period between switching. That is, it gradually approaches Z 'over a certain period of time.

(5) According to another aspect of the present invention, when the depth sensation factor of a partial region of an image changes, the temporal change of the depth sensation factor of the partial region is reduced, and the depth of each region of the image is reduced. In order to reconstruct a sense relationship, a predetermined conversion process is also performed on a depth sense factor in an area other than the partial area.

Here, as an example in which a factor of a sense of depth in a partial area of an image changes, a new subject may move and appear at a position close to the camera. In this case, since the depth changes abruptly in the area of the subject (hereinafter, referred to as “attention area”), the depth of the attention area is not changed at once, but is changed over time to some extent. In this case, for example, the original depth of the attention area is “100”,
Assuming that the depth of a newly appearing subject (hereinafter, referred to as a “target body”) is “60”, the depth of this area is gradually changed, such as 100 → 80 → 60. This avoids a sudden change in the parallax angle for the attention area.

On the other hand, the depth of the object of interest is actually 60. Therefore, a region other than the region of interest (hereinafter, “normal region”)
), A subject having a depth of “70” should be displayed farther than the target object. However, while the depth of the attention area takes a number of 100 or 80, the subject having the depth “70” is displayed closer than the object of interest. Therefore, in the present invention, in order to reconstruct the relationship of the sense of depth of each region of the image,
The depth sense factor of the normal area is also changed.

(6) In the case of (5), in one aspect of the present invention, in order to suppress both the temporal change of the depth sensation factor of the attention area and the temporal change of the depth sensation factor of the normal area,
The sense of depth is controlled in consideration of both of these. In the case of the above-described example, it is assumed that, for example, among subjects included in the normal region, the depth of a subject having a depth of “70” is changed to “100”. The amount of this change is 30, which also changes the parallax angle of this subject area. Therefore, in the present invention, the depth is controlled in consideration of not only the temporal change of the depth of the attention area but also the temporal change of the depth of the normal area.

(7) In the case of (6), in one embodiment of the present invention, the energy for comprehensively evaluating the temporal change of the depth sensation factor of the attention area and the temporal change of the depth sensation factor of the normal area is calculated. An expression is introduced to control the sense of depth.
The “energy formula” means a formula having an evaluation significance at an extreme value, and is used to reduce the temporal change of the depth of the attention area and the temporal change of the depth of the normal area. To find out.

(8) In the case of (6), according to an aspect of the present invention, the maximum rate for determining at least one of the upper limit of the temporal change rate of the depth sensation factor of the attention area or the temporal change rate of the depth sensation factor of the normal area is set. Controlling the sense of depth by introducing a rate-of-change limiting formula.

[0025]

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiment 1 FIG . In the first embodiment, the depth of the subject is selected as the depth feeling factor to be controlled. FIG. 1 is a configuration diagram of a three-dimensional moving image generation device 30 that implements the depth feeling control method of the present embodiment. This device can be provided inside a stereoscopic movie or stereoscopic television, for example.

As shown in the figure, the apparatus inputs a right-eye image together with depth information from images taken by a stereo camera or the like, and generates a left-eye image. The generated left-eye image and right-eye image are output after being synchronized. The output images for the left and right eyes are respectively sent to a display device (not shown), and stereoscopic display is performed.

This apparatus has a frame data storage unit 32 having a FIFO structure for sequentially storing right-eye images input as frame data, a depth conversion unit 38 to which depth information is input, and a scene change detection unit 40.
The frame data storage unit 32 is used as a temporary buffer for temporarily storing frame data during the calculation when the depth is gradually changed over a plurality of frames as described later.

The scene change detecting section 40 detects the location of a scene change based on the input depth information of each frame. At this time, for example, assuming that the depth of each area of the image of the frame s is in the range of Zsmin to Zsmax and that of the frame t immediately after that is in the range of Ztmin to Ztmax, the absolute value D of the average depth difference of each frame, If D = | (Zsmin + Zsmax) / 2− (Ztmin + Ztmax) / 2 | (Equation 1) exceeds a certain threshold value, it is determined that a scene change has occurred. However, this detection may be based on a difference in variance or the like, instead of a difference in average value of depth.

When a scene change is detected, the depth converter 38 converts the input depth so that the depth gradually changes over a plurality of frames thereafter. In the case of Equation 1, the depth suddenly changes from Ztmin to Zt at frame t.
Rather than becoming max, Zsmin gradually becomes Ztmin and Zsmax gradually becomes Ztm over several frames after frame t.
The depth corresponding to each frame after the frame t is converted so as to approach ax. However, while there is no scene change, the depth conversion unit 38 outputs the input depth information as it is.

The output of the depth converter 38 is provided to the left-eye image generator 36. If the depth output from the depth conversion unit 38 is, for example, related to the frame s, the left-eye image generation unit 36 reads the data of the frame s from the frame data storage unit 32, and reads the right-eye image data and the input depth. Is used to generate a left-eye image. The generated left-eye image is provided to a display device (not shown).

The timing adjustment section 34 is a synchronization circuit for adjusting the output timings of the generated left-eye image and the corresponding right-eye image. From this, the right-eye image is output in a state where the timing matches the left-eye image, and is provided to the display device.

FIG. 2 is a diagram showing depth control by the three-dimensional moving image generator 30. As shown in FIG. FIG. 3A shows a conventional case in which depth control is not performed, and FIG. 4B shows a case in which control is performed by the present apparatus.

In the figure, frames 1 to 13 of the right-eye image are shown.
Are continuously input to the device. Here is frame 1
4 is a scene S1 and a frame 5 and subsequent frames are another scene S2, and the depth of each image region of each frame in the scene S1 is Z1min to Z1max (the average is Z1a
ve) and that of scene S2 are all Z2min ~ Z2ma
x (mean Z2ave). That is, in this case, the depth changes suddenly between frames 4 and 5 where a scene change occurs, and if the left-eye image is generated as it is from this depth, the parallax angle greatly changes at this switching point as shown in FIG. I do.

On the other hand, in FIG. 3B, the depth gradually changes after the scene change. Although the depth should actually take a discrete value for each frame, it is schematically depicted here as a continuous value. When processing is performed by the present apparatus, first, the scene change detection unit 40 in FIG. 1 reads the depth of each frame and calculates the difference between the depths of consecutive frames. Here, when | Z1ave−Z2ave | is calculated between frames 4 and 5, this value is determined to be large, and a scene change is detected.

Subsequently, the depth conversion unit 38 outputs the frame 5
Manipulate the depth to be given to subsequent frames. For example, the following two methods can be considered as rules for the depth operation.

1. 1. Determine the number of frames to be operated for depth. Determine the upper limit of the rate of change of depth. May mitigate the change in depth by taking into account the duration of the mitigation process. Specifically, 1. In the case of, for example, a rule that “the depth is gradually changed using n frames including the frame immediately after the scene change” can be defined. If it is considered that FIG. 2B complies with this rule, the above n corresponds to 4, and the depth is manipulated in four frames 5 to 8. When adopting this rule, if a scene change occurs in a relatively short time, set the above n to a small value,
Otherwise, it is possible to take a large control.

On the other hand, 2. If 1. In contrast, the number of frames whose depth is manipulated does not matter in principle.
For example, in the case of FIG. 2B, the depth is controlled so that the absolute value of the inclination of the straight line L connecting the maximum value Z1max of the depth included in the frame 4 to the maximum value Z2max of the frame 5 does not exceed a certain value. Therefore, if the difference between Z1max and Z2max is doubled, the number of frames for which the depth operation is performed is also doubled. When this rule is adopted, it is easy to suppress the rate of change of the parallax angle to a certain value or less.

When the depth conversion is completed in this way, the converted depth is input to the left-eye image generation unit 36, where the left-eye image is converted based on each frame of the right-eye image and the depth corresponding to the frame. Generate. The generated left-eye image is output to the subsequent display device together with the right-eye image that has passed through the timing adjustment unit 34.

The above is the contents of the present embodiment. According to the present embodiment, when the viewer sees the finally displayed stereoscopic moving image, the number of scenes in which the parallax angle changes rapidly decreases,
Visual fatigue can be reduced. Note that the present embodiment may have the following modifications.

(1) Here, the depth operation is performed starting from the frame immediately after the scene change, but this operation may be started from another frame. For example, the depths of four frames 3 to 6 in FIG. 2 may be operated so as to sandwich the timing of the scene change, or the frame 1 before the scene change may be operated.
4 to 4 may be used.

(2) In FIG. 2B, the depth is linearly converted between the two scenes. However, another conversion method may be used. For example, the depth of frame 4 and frame 9
May be used as a non-linear transformation using a tanh function or the like that smoothly connects the depths of.

(3) In this embodiment, the depth conversion is performed after detecting a scene change, but in principle, the depth difference between arbitrary frames is gradually changed regardless of the presence or absence of a scene change. You can go. In that case,
The scene change detection unit 40 in FIG. 1 does not need to be provided.

(4) Three-dimensional moving image generating apparatus 3 of the present embodiment
0 can be provided inside a three-dimensional movie or a three-dimensional television, but can also be applied to the photographing side and the editing side device, not to the reproducing side device side. For example, the present invention can be incorporated into a part of a stereo camera system that converts the depth of a captured video on the spot, or an image editing apparatus such as a broadcasting station.

(5) In the present embodiment, the control of the depth of the subject photographed by the stereo camera has been described.
The present invention is also applicable to, for example, moving images of computer graphics. In such a case, an image may be generated or displayed so that the depth gradually changes when switching frames or scenes.

(6) After controlling the depth according to the present invention, the image can be transmitted over a network by combining the depth and the image that is the basis of the three-dimensional display, or can be recorded on an image medium and distributed. . That is, the present invention provides a method for communicating an image generated in a state where the change in the actual depth of a subject is reduced in the time direction, and an image generated in a state in which the change in the actual depth of the subject is reduced in the time direction. Can be applied in the form of a medium on which is recorded.

(7) In the present embodiment, a right-eye image and depth information are input to generate a left-eye image. Other than that, A right-eye image, a left-eye image, and depth information are input, and a left-eye image and a right-eye image obtained by correcting the input are output.

2. An intermediate image and depth information of the right-eye image and the left-eye image are input, and a left-eye image and a right-eye image obtained by combining the intermediate images are output.

The following is conceivable. Especially this 1. In the case of (1), the amount of correction of each image is small, and good image quality can be realized.

(8) In the present embodiment, the depth z is taken as an object to be controlled, but this may be a parallax angle or a parallax on a screen. In this case, the information input to the depth conversion unit 38 of the three-dimensional moving image generating device in FIG. 1 is disparity angle information or on-screen disparity information instead of depth information. Note that the on-screen parallax refers to the parallax that the subject has at the screen position of the display device. The meaning is as shown in FIG. 7, and in this case, the on-screen parallax is given by fe / z.

Embodiment 2 The first embodiment mainly deals with a sudden change in the depth due to a scene change. However, actually, even in the same scene, the depth of a partial area of an image, that is, the attention area may suddenly change. The present embodiment relates to a depth sensation control method considering such a case.

The technology on which the present embodiment is based is a part of a depth or parallax angle conversion method proposed by the present applicant in Japanese Patent Application No. 7-257141 (hereinafter referred to as "prerequisite technology"). The prerequisite technique is to limit the depth or parallax angle of a captured image to a predetermined range by linear or non-linear conversion, and aims to eliminate visual fatigue and the like as in the present application. In the base technology, when a region of interest appears in a certain frame, the depth of all regions including the depth of this region is converted into a predetermined range. In the present embodiment, the speed and degree of the conversion operation are adjusted. It is.

FIG. 3 is a view for explaining the correspondence between two adjacent frames (t-1) and the frame t, and the attention area 60 appearing in the frame t. The figure shows an image in which a mountain is in the background and a car is in front of the mountain. This ball forms the attention area 60.

If the depths of a car, a mountain, and a ball are Zcar, Zmnt, and Zball, respectively, and these are to be uniformly set within a desired range ZMIN to ZMAX (hereinafter, referred to as an “ideal range”) by linear transformation, In the frame (t-1), a conversion rule is determined by calculating a and b (hereinafter, a and b are also referred to as "depth conversion coefficients") from a conversion formula of ZMIN = aZcar + bZMAX = aZmnt + b. On the other hand, in the frame t, new depth conversion coefficients a and b satisfying ZMIN = aZball + bZMAX = aZmnt + b are determined by the appearance of the ball. As a result, the depth of the vehicle changes from frame (t-1) to frame t.
The parallax angle of the background other than the ball changes, for example, it suddenly becomes larger at the moment of transition to. If the amount of change is large, there arises a problem that not only the visual fatigue but also the image becomes unnatural.

FIG. 4 shows a three-dimensional moving image generating apparatus 5 according to this embodiment.
FIG. In the figure, parts corresponding to those in FIG. 1 are given the same reference numerals, and different parts will be mainly described below.

The three-dimensional moving image generator 50 includes an energy calculator 52 instead of the scene change detector 40 shown in FIG.
have. The energy calculation unit 52 is provided with corresponding point information between frames, that is, information for specifying the position of the corresponding point, from outside the apparatus. In the present embodiment, it is assumed that no corresponding point information is provided for a point having no corresponding point, for example, a point inside the ball in FIG. Conversely, it is assumed that corresponding point information is always given to an area other than the attention area 60, that is, a normal area.

The energy calculator 52 is also provided with depth information. It is assumed that the depth information is given for all regions or all pixels of each frame, regardless of whether or not the region of interest 60. In the present embodiment, the attention area 6
In order to suppress both the temporal change of the depth of 0, that is, the change of the depth from the frame (t-1) to the frame t and the temporal change of the depth of the normal region, the depth is controlled in consideration of both of them. . In the energy calculation unit 52,
The magnitude of the temporal change in the depth of the attention area 60 and the temporal change in the depth of the normal area are comprehensively evaluated by an energy equation, and a coordination point that reduces both of them is found. Here, as shown in FIG. 3, the point Pi of the frame t is located at the frame (t-1).
) (Hereinafter referred to as “corresponding point”). Hereinafter, the actual depth of the subject is represented by Zxx, and the depth for display is represented by ZAxx, and the parameters are represented as follows.

Z (i, t): actual depth of point Pi at time t ZA (i, t): display depth of point Pi at time t Zmin (t): actuality of the entire video at time t Zmax (t): Maximum value of the actual depth of the entire image at time t. ZAmin (t): Minimum value of the display depth of the entire image at time t. ZAmax (t): At time t. The maximum value of the display depth of the entire video • ZMIN, ZMAX: The minimum and maximum values of the ideal range of the display depth. Here, if t is replaced with (t-1), the parameters relating to the frame (t-1) are respectively used. Becomes Zmin (t), ZAmi
It should be noted that the depth of the ball is also reflected in n (t) since the whole image is considered. At this time, the formula of the energy calculation performed by the energy calculation unit 52 is as follows.

E = Σ (ZA (i, t−1) − (aZ (i, t) + b)) ² + k (ZMIN− (aZmin (t) + b)) ² + k (ZMAX− (aZmax (t) + b) )) ² (Equation 2) Σ in the first term is the sum of all the corresponding points, and this term acts to suppress the fluctuation of the depth of the normal region. The first term gives an evaluation relating to the normal area, and this calculation does not include points inside the attention area 60.

On the other hand, the second and third terms are equivalent to the conversion method of the base technology, and operate to keep the depth of each frame within an ideal range. The attention area 60 is also considered in the calculation of these terms. K applied to these terms is an appropriate coupling coefficient. The larger the value of k, the closer to the control of the base technology, and the smaller the value of k, the closer the depth after conversion at time t, that is, the depth for display, to the depth for display at time (t-1). Here, the energy calculation is as follows: dE / da = 0, dE / db =
This is a calculation for finding a and b that become 0. With this calculation, a coordination point that suppresses both the rate of change of the depth in the attention area and the change in the depth in the normal area is found. The energy calculation unit 52 performs this energy calculation for each frame, obtains the optimum depth conversion coefficients a and b at that time, and notifies the depth conversion coefficients to the depth conversion unit 38. The depth conversion unit 38 performs a linear conversion on the depth by using the notified depth conversion coefficient, and the left-eye image generation unit 36 generates a left-eye image based on the conversion result.

FIG. 5 is a diagram showing how the depth conversion unit 38 converts the depth as a result of the energy calculation by the energy calculation unit 52. In the figure, the line of sight of the viewer is the z-axis. During a stable period in which the attention area 60 does not appear and the effect of the attention area that has appeared before has already disappeared (hereinafter referred to as “stable period”), the display depth of each frame falls within this range.

FIG. 9 shows a range of depth extending to four levels, of which the range 76 of the fourth level is the frame (t-1).
) Is the real depth range. This is converted to the third range 74 in the previous processing, and the frame (t-1)
It is assumed that the display depth range 74 has already been determined. In the figure, the frame (t-1) is in the stable period, and its display depth matches the ideal range.

In this state, the processing of frame t is performed. First, it is assumed that the frame t is photographed, and the range of the actual depth of the entire video is in the range 72 as shown in the second row in FIG. Here, this range 72 protrudes in a direction larger than the ideal range.

On the other hand, the first range 70 is the depth conversion unit 3.
8, the original range 72 is a display depth range obtained as a result of the conversion. Here, as a result of considering the display depth range and the ideal range of the immediately preceding frame by the above-described energy calculation, a range between the two is obtained. As for the first term of Expression 2, as shown in the figure, the depth ZA (i, t-1) at the point Pi of the frame (t-1)
And the actual depth Zi of the point Pi of the frame t is considered, and the depth of this point is converted to ZA (i, t).

In the case where the attention area 60 remains in the frame after the frame t in FIG. 3, according to the present embodiment, all the depths finally fall within the ideal range again. The grounds will be described.

[0065] First, the time _{(t-1) a t-} 1 the depth transform coefficients determined in, b _t-1, those obtained at time t and a _t, b _t. On the other hand, E in Equation 2 is divided into a first term, a second term, and a third term, and are expressed as follows.

E = E1 + kE2 (Equation 3) where E1 = Σ (ZA (i, t−1) − (aZ (i, t) + b)) ² E2 = (ZMIN− (aZmin (t) + b)) ² + (ZMAX- (aZmax
(t) + b)) Whether or not the depth range of the frame after ^two hours has elapsed matches the ideal range can be evaluated by E2.
That is, if this E2 decreases uniformly over time,
At any point, the depth range of the frame matches the ideal range. Here, Expression 3 at time (t-1) and t is described as follows.

E (t−1) = E1 (t−1) + kE2 (t−1) (Expression 4) E (t) = E1 (t) + kE2 (t) (Expression 5) in _{4 a t-1, b t} -1, in formula 5 a _t, a b _t.

Here, even at time t, time (t-1)
Considering the situation using the depth conversion coefficients a _t−1 and b _t−1 at the time of (5), the equation in this case is expressed as E (t) ′ = E1 (t) ′ + kE2 (t) ′ based on Equation 5. (Equation 6). At this time, since the depth conversion coefficient at the time (t-1) is used, the depths of the normal regions of the two frames match, and E1 (t) 'becomes zero. Therefore, Equation 6 can be modified as follows.

E (t) ′ = kE2 (t) ′ = kE2 (t−1) (Equation 7) On the other hand, the depth conversion coefficients a _t and b _t obtained as the optimal numerical values at the time _t are E (t) Is minimized, so that E (t) ≦ E (t) ′ (Equation 8) holds. Here, from Equations 5 and 7, E2 (t) −E2 (t−1) = (E (t) −E1 (t) −E (t) ′)
/ K is obtained. Here, considering that Equation 8 and E1 (t) are positive, E2 (t) −E2 (t−1) ≦ 0, and the uniform decreasing property of E2 is proved. Therefore, even when the depth of a partial area of an image changes suddenly as in the present embodiment, it is possible to finally reduce the depth to an ideal range while relaxing the change.

FIG. 8 is a diagram showing the relationship between the actual depth of the subject and the display depth converted by the energy calculation formula. In the figure, the thick solid line indicates the display depth,
Broken line indicates the actual depth, a new object suddenly closer has appeared at time T _0. As shown by arrows 80 and 82 in the figure, at T ₀ , the display depth once protrudes above and below the ideal range, and thereafter falls within the ideal range with the passage of time.

The second embodiment has the same modifications as the first embodiment, and the following modifications are also conceivable.

(1) In the present embodiment, an image is generated in accordance with the display depth obtained according to the energy calculation equation (2). However, when the calculation results in a depth exceeding the ideal range, the portion beyond the ideal range may be displayed after being corrected to uniformly change the upper limit or the lower limit closer to the ideal range.

(2) In this embodiment, the energy calculation is performed in order to consider both the attention area and the normal area.
This may be performed by the maximum change rate limit calculation. The maximum change rate restriction calculation refers to a calculation performed by setting at least one upper limit of the time change rate of the depth of the attention area or the time change rate of the depth of the normal area. An example of this calculation will be described. Here, the maximum change rate, that is, the upper limit of the change rate is written as UL,
The rate of change is defined as "change in display depth between adjacent frames". Under these conditions, first, KMIN = MIN (ZA (i, t-1))-ZMIN KMAX = MAX (ZA (i, t-1))-ZMAX, thereby performing the following processes 1 and 2. .

1. if (| KMIN | ≦ UL) then ZMINTMP =
ZMIN; else if (KMIN> UL) then ZMINTMP = MIN (ZA (i, t-
1))-UL; else if (KMIN <-UL) then ZMINTMP = MIN (ZA (i, t-
1)) + UL; if (| KMIN | ≦ UL) then ZMINTMP = ZMIN; else if (KMIN> UL) then ZMINTMP = MIN (ZA (i, t-
1))-UL; else if (KMIN <-UL) then ZMINTMP = MIN (ZA (i, t-
1)) + UL; Then, the conversion coefficients a and b are determined from ZMINTMP = aZmin (t-1) + bZMAXTMP = aZmax (t-1) + b.

In this calculation, when the depth is changed linearly, attention is paid to the fact that the point having the largest change rate among the corresponding points (hereinafter referred to as the maximum change rate point) originally has either the maximum or the minimum depth. ing. Then, a and b are determined so that the change rate at the maximum change rate point is equal to or less than an upper limit. Specifically, 1. In the first row of the frame (t-1
), Even if the minimum display depth is less than ZMIN,
If the difference is equal to or less than UL, the minimum value of the display depth is directly changed to ZMIN in frame t. This is because the rate of change between these two frames falls below UL. On the other hand, 1. In the second row, the minimum display depth of the frame (t-1) is smaller than ZMIN and if the difference exceeds UL, the minimum display depth of the frame (t-1) is directly
Instead of changing to ZMIN, it is approaching ZMIN by UL. Therefore, the rate of change between these two frames is exactly UL. After that, it will be finally stored in ZMIN in several frames.

Hereinafter, the same can be considered, and in any case, the rate of change does not exceed UL. As long as a and b are determined in this way, the change rates at points other than the maximum change rate point are also uniformly compressed by the linear conversion, and therefore always fall below the upper limit of the change rate.

FIG. 9 is a diagram showing the relationship between the actual depth of the subject and the display depth converted therefrom by the maximum rate-of-change limiting method. The notation in the figure complies with FIG. Since the upper limit of the rate of change is limited as indicated by arrows 84 and 86 in the figure, the behavior differs from that in FIG. As a result of the object appearing at a close distance at time T ₀ , the minimum value of the display depth of the previous frame is converted to Zs in the figure. The amount of change between frames, that is, the change rate, is UL. On the other hand, the time T ₀ after the frame, the minimum value of those of the display depth approaches the similarly ZMIN by UL. Therefore, the inclinations of the arrow 84 and the arrow 86 are equal. FIG.
In the case of (1), the convergence to the ideal range occurs uniformly, so that the time required for convergence is short.

[0078]

According to the depth sense control method of the present invention, the change in the actual depth of the subject included in the three-dimensional moving image is reduced in the time direction, so that the change in the parallax angle can be reduced. As a result, a good three-dimensional moving image can be easily obtained without relying on the know-how and manual work of a cameraman or an image editor as in the related art.

When the above-described relaxation is performed in consideration of the rate of change of the depth of the image, for example, an upper limit can be set for the rate of change of the parallax angle, which is preferable from the viewpoint of eliminating visual fatigue. The same applies when considering the duration of the relaxation process.

In the vicinity of the scene switching point, the depth of one scene is gradually changed over a plurality of display unit images,
When gradually approaching the depth of the other scene, it is possible to properly cope with a scene in which the parallax angle tends to change relatively abruptly, such as a so-called scene change.

When both the temporal change of the depth of the attention area and the temporal change of the depth of the normal area are considered, not only the visual fatigue due to the change of the depth of both areas is reduced, but also the unnaturalness of the stereoscopic display. Can also be reduced. When these two temporal changes are evaluated by the energy equation or the maximum rate-of-change limiting equation, it is guaranteed that not only the objectivity of the evaluation but also that the depth of the image finally falls within the ideal range.

[Brief description of the drawings]

FIG. 1 is a configuration diagram of a three-dimensional moving image generation device that implements a depth feeling control method according to a first embodiment.

FIG. 2 is a diagram illustrating depth control by the three-dimensional moving image generation device according to the first embodiment.

FIG. 3 is a diagram illustrating a relationship between two adjacent frames (t−1) and a frame t according to the second embodiment;
FIG. 6 is a diagram for explaining an attention area that has appeared in FIG.

FIG. 4 is a configuration diagram of a three-dimensional moving image generation device according to a second embodiment.

FIG. 5 is a diagram illustrating a state in which depth is converted by a depth conversion unit as a result of energy calculation by the energy calculation unit according to the second embodiment.

FIG. 6 is a diagram showing a principle of generating a stereoscopic image by a stereo camera.

FIG. 7 is a diagram illustrating the meaning of on-screen parallax.

FIG. 8 is a diagram showing the relationship between the actual depth of the subject and the display depth converted from the actual depth by an energy calculation formula.

FIG. 9 is a diagram showing the relationship between the actual depth of a subject and the display depth converted therefrom by the maximum rate-of-change limiting method.

[Explanation of symbols]

30, 50 stereoscopic video generation device, 32 frame data storage unit, 34 timing adjustment unit, 36 left eye image generation unit, 38 depth conversion unit, 40 scene change detection unit, 52 energy calculation unit, 60 attention area, 7
Range of 0, 72, 74 depth.

Claims (8)

[Claims] 1. A depth sensation control that performs a conversion process for mitigating a change in a depth sensation factor of a subject in a time direction, thereby controlling a depth sensation when the subject is displayed as a three-dimensional moving image. Method. 2. The depth sensation control method according to claim 1, wherein the conversion processing is performed in consideration of a time change rate of a depth sensation factor when the subject is displayed as a three-dimensional moving image. 3. The depth sense control method according to claim 1, wherein the conversion processing is performed in consideration of a continuation time of the conversion processing. 4. A conversion process in which a depth sensation factor of one scene gradually approaches a depth sensation factor of another scene over a plurality of display unit images near a switching point of a scene included in a three-dimensional moving image. A method of controlling a sense of depth, which is performed. 5. When the depth sensation factor of a partial area of an image changes, to temporally change the depth sensation factor of the partial area, and to reconstruct the relationship of the depth sensation of each area of the image. A depth feeling control method, wherein a predetermined conversion process is also performed on a depth feeling factor in an area other than the partial area. 6. The depth sensation control method according to claim 5, wherein a temporal change in a depth sensation factor in the partial area and a temporal change in a depth sensation factor in an area other than the partial area are both suppressed. A depth sensation control method characterized by controlling the depth sensation in consideration of both of these. 7. The depth sensation control method according to claim 6, wherein a temporal change of the depth sensation factor of the partial area and a temporal change of the depth sensation factor of an area other than the partial area are comprehensively calculated. A depth sense control method that controls the sense of depth by introducing an energy formula to be evaluated. 8. The depth sense control method according to claim 6, wherein at least one of a time change rate of the depth sense factor of the partial area and a time change rate of the depth sense factor of the area other than the partial area. Depth control method that controls the depth sensation by introducing a maximum rate-of-change limiting formula that defines the upper limit of the depth.