WO2012039340A1 - Dispositif et procédé de traitement d'image et programme - Google Patents

Dispositif et procédé de traitement d'image et programme Download PDF

Info

Publication number
WO2012039340A1
WO2012039340A1 PCT/JP2011/071079 JP2011071079W WO2012039340A1 WO 2012039340 A1 WO2012039340 A1 WO 2012039340A1 JP 2011071079 W JP2011071079 W JP 2011071079W WO 2012039340 A1 WO2012039340 A1 WO 2012039340A1
Authority
WO
WIPO (PCT)
Prior art keywords
parallax
frame
frames
image processing
adjustment
Prior art date
Application number
PCT/JP2011/071079
Other languages
English (en)
Japanese (ja)
Inventor
岳彦 指田
Original Assignee
コニカミノルタホールディングス株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by コニカミノルタホールディングス株式会社 filed Critical コニカミノルタホールディングス株式会社
Priority to JP2012535012A priority Critical patent/JP5601375B2/ja
Publication of WO2012039340A1 publication Critical patent/WO2012039340A1/fr

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/122—Improving the 3D impression of stereoscopic images by modifying image signal contents, e.g. by filtering or adding monoscopic depth cues
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/207—Image signal generators using stereoscopic image cameras using a single 2D image sensor
    • H04N13/221—Image signal generators using stereoscopic image cameras using a single 2D image sensor using the relative movement between cameras and objects

Definitions

  • the present invention relates to an image processing technique.
  • 3D televisions that use stereoscopically visible video (also called 3D video) are in the spotlight.
  • an image also referred to as a 3D image
  • the focus adjustment function of the user's eyes is used to realize a sense of depth of the image.
  • Patent Document 1 proposes a technique that can reduce the fatigue of a user who performs stereoscopic viewing by gradually changing the sense of depth at the scene switching position.
  • 3D image technology has been adopted in various video fields.
  • an endoscopic device that enables stereoscopic viewing of an image in a wide range by adjusting parallax so as to fall within a human fusion range.
  • Patent Document 2 proposes an endoscopic device that enables stereoscopic viewing of an image in a wide range by adjusting parallax so as to fall within a human fusion range.
  • an image also referred to as a virtual viewpoint image
  • a 3D image is generated.
  • a moving image also referred to as a virtual viewpoint moving image
  • a 3D moving image can be generated. It becomes.
  • an area corresponding to a portion of the subject that is not captured in the original captured image is an area where the pixel value is unknown (also referred to as an occlusion area). Due to the occurrence of the occlusion area, there is no video that should originally exist in the 3D moving image, which makes the user feel uncomfortable. Furthermore, the higher the proportion of the occlusion area in the image, the greater the sense of discomfort experienced by the user. In particular, in a scene where a foreground appears suddenly in an image, an increase in discomfort experienced by the user becomes significant.
  • Patent Documents 1 and 2 For such a problem, none of the techniques of Patent Documents 1 and 2 described above can reduce the sense of incongruity experienced by the user in correspondence with an occlusion region in which different sizes can occur. In order to solve such a problem, it is required that the virtual viewpoint moving image that is the basis of the 3D moving image is visually uncomfortable.
  • the present invention has been made in view of the above problems, and an object of the present invention is to provide a technique for reducing visual discomfort given to a user when generating a pseudo moving image viewed from a virtual viewpoint. .
  • an image processing apparatus includes an image acquisition unit that acquires a moving image including a plurality of frames, and a predetermined rule for each pixel for each of the frames.
  • a target parallax setting unit that sets a target parallax
  • an initial parallax setting unit that sets an initial parallax for each pixel for the first frame of the plurality of frames, and the one of the plurality of frames
  • a parallax adjustment unit that adjusts the parallax of each pixel from the initial parallax toward the target parallax for a plurality of frames related to a first period and a second period that are time-sequential following the first frame;
  • the first change rate obtained by dividing the amount of change in parallax in one period by the length of the first period is obtained by dividing the amount of change in parallax in the second period by the length of the second period. Second rate of change Remote small.
  • the image processing device is the image processing device according to the first aspect, and stores a plurality of types of adjustment patterns related to a change in parallax from the initial parallax toward the target parallax,
  • the parallax adjustment unit adjusts the parallax of each pixel along an adjustment pattern according to a difference between the initial parallax and the target parallax among the plurality of types of adjustment patterns.
  • An image processing device is the image processing device according to the first aspect, wherein the parallax adjustment unit is adjusted to the Nth after being adjusted by the parallax adjustment unit when N is a natural number.
  • the parallax of the (N + 1) th and subsequent frames is adjusted based on the parallax of the current frame and the target parallax.
  • An image processing device is the image processing device according to the first aspect, wherein when the parallax adjustment unit sets Q as a natural number, the target parallax for each pixel in the N + Q-th frame and The parallax of the (N + 1) th and subsequent frames is adjusted based on the parallax of the Nth frame after being adjusted by the parallax adjusting unit.
  • An image processing device is the image processing device according to the first or second aspect, wherein the parallax adjustment unit calculates the parallax of each pixel for the number of frames corresponding to a predetermined time. The parallax is adjusted to the target parallax.
  • An image processing device is the image processing device according to any one of the first to fourth aspects, wherein the parallax adjustment unit has a parallax adjustment amount between temporally adjacent frames. The parallax of each pixel is adjusted so as not to exceed the upper limit value.
  • An image processing apparatus is the image processing apparatus according to any one of the first to sixth aspects, wherein the parallax adjustment unit has a parallax of a first subject captured by the moving image. Accordingly, the parallax of each pixel is adjusted for a second subject different from the first subject.
  • An image processing device is the image processing device according to any one of the first to sixth aspects, wherein the parallax adjustment unit is in accordance with the position of the subject captured in each frame. Then, the parallax of each pixel is adjusted.
  • An image processing device is the image processing device according to any one of the first to eighth aspects, wherein the target parallax is a separation between a viewpoint related to each frame and a virtual viewpoint. This corresponds to the distance and the distance from the imaging device to the subject when each of the frames is captured.
  • An image processing apparatus is the image processing apparatus according to any one of the first to ninth aspects, wherein N is a N number among the plurality of frames when N and p are natural numbers.
  • a detection unit that detects an adjusted region in the Nth frame by comparing the Nth frame and the Npth frame that is temporally prior to the Nth frame, and the parallax adjustment The unit adjusts the parallax of each pixel related to the adjusted region from the initial parallax toward the target parallax.
  • An image processing method includes (a) a step of acquiring a moving image including a plurality of frames, and (b) target parallax for each pixel based on a predetermined rule for each frame. (C) setting an initial parallax for each pixel for the first frame of the plurality of frames, and (d) setting the first frame of the plurality of frames to the first frame.
  • the first rate of change obtained by dividing the amount by the length of the first period is greater than the second rate of change obtained by dividing the amount of change in parallax in the second period by the length of the second period.
  • a program according to a twelfth aspect is executed by a control unit included in the information processing apparatus to cause the information processing apparatus to function as the image processing apparatus according to any one of the first to tenth aspects. It is.
  • the image processing apparatus According to the image processing apparatus according to any one of the first to tenth aspects, it is possible to reduce the visual discomfort given to the user when generating a pseudo moving image viewed from a virtual viewpoint.
  • the image processing apparatus According to the image processing apparatus according to the fourth aspect, it is possible to easily adjust the parallax corresponding to the change in the target parallax.
  • the image processing apparatus since the variation in time required for parallax adjustment is suppressed, the visual discomfort given to the user can be appropriately suppressed.
  • the image processing apparatus since a significant change in parallax is suppressed, the visual discomfort given to the user can be appropriately suppressed.
  • the image processing apparatus since the consistency of parallax between objects is improved, the visual discomfort given to the user can be appropriately suppressed.
  • the image processing apparatus According to the image processing apparatus according to the eighth aspect, it is possible to adjust the parallax according to the degree of attention of the user.
  • the image processing apparatus it is possible to set a target parallax suitable for generating an image that can be stereoscopically viewed.
  • the parallax can be adjusted as necessary.
  • Any of the image processing method according to the eleventh aspect and the program according to the twelfth aspect can reduce the visual discomfort given to the user when generating a pseudo moving image viewed from a virtual viewpoint. it can.
  • FIG. 1 is a diagram for explaining the characteristics of a process according to an embodiment.
  • FIG. 2 is a diagram illustrating a schematic configuration of an image processing system according to an embodiment.
  • FIG. 3 is a block diagram illustrating a functional configuration according to the image processing apparatus.
  • FIG. 4 is a diagram for explaining how to obtain the target parallax using the principle of triangulation.
  • FIG. 5 is a diagram schematically illustrating an example of an Nth right eye frame.
  • FIG. 6 is a diagram schematically illustrating an example of an Nth right eye frame.
  • FIG. 7 is a graph schematically illustrating the relationship between time and adjusted parallax.
  • FIG. 8 is a graph schematically illustrating the relationship between time and adjusted parallax.
  • FIG. 1 is a diagram for explaining the characteristics of a process according to an embodiment.
  • FIG. 2 is a diagram illustrating a schematic configuration of an image processing system according to an embodiment.
  • FIG. 3 is a block diagram illustrating a functional configuration according
  • FIG. 9 is a flowchart showing an operation flow of the image processing apparatus.
  • FIG. 10 is a flowchart showing an operation flow of the image processing apparatus.
  • FIG. 11 is a schematic view illustrating one aspect of parallax adjustment related to a virtual viewpoint image.
  • FIG. 12 is a graph schematically illustrating the relationship between time and adjusted parallax.
  • FIG. 13 is a graph schematically illustrating the relationship between time and adjusted parallax.
  • FIG. 14 is a schematic view illustrating one aspect of parallax adjustment related to a virtual viewpoint image.
  • FIG. 15 is a graph schematically illustrating the relationship between time and adjusted parallax.
  • FIG. 16 is a graph schematically illustrating the relationship between time and adjusted parallax.
  • FIG. 17 is a schematic view illustrating an aspect of parallax adjustment related to a virtual viewpoint image.
  • FIG. 18 is a graph schematically illustrating the relationship between time and adjusted parallax.
  • FIG. 19 is a graph schematically illustrating the relationship between time and adjusted parallax.
  • FIG. 20 is a graph schematically illustrating the relationship between time and adjusted parallax.
  • FIG. 1 is a diagram for explaining the characteristics of a process according to an embodiment.
  • an Nth (N is a natural number less than n) frame (also referred to as an Nth frame) GN of a moving image composed of n frames (n is a natural number greater than or equal to 2) is N , With a rectangular outer edge.
  • the upper left pixel of the N frame G N is the origin
  • the direction (in this case the transverse direction) along the long sides of the N-th frame G N is the X-axis direction
  • the short side of the N-th frame G N The direction along the vertical axis (here, the vertical direction) is taken as the Y-axis direction.
  • the right direction of the N-th frame G N is the + X direction
  • the downward direction of the N-th frame G N is the -Y direction.
  • an Nth frame (also referred to as an Nth frame) of N frames (N is a natural number of n or less) of n frames (n is a natural number of 2 or more).
  • N is a natural number of n or less
  • n is a natural number of 2 or more
  • the N-th frame G N is (also referred to as a first N left eye frame) as the N-th frame for the left eye is used as a G N L
  • the N-th frame for the right eye from the N frame G N (No. N is also referred to as the right-eye frame) is assumed that G N R is generated.
  • the position of the camera (specifically, the image sensor) when the Nth frame G N (the Nth left eye frame G N L) is captured is the Nth viewpoint, and the Nth right eye frame.
  • position of the virtual camera for capturing G N R is the N-th virtual viewpoint.
  • the Nth viewpoint and the Nth virtual viewpoint are in a positional relationship between a human standard left eye and right eye.
  • a displacement (that is, parallax) of the pixel positions where the same subject is captured occurs between the N-th left eye frame G N L and the N-th right eye frame G N R.
  • the N-th right eye frame G N R includes not only the area corresponding to the object (here, a person) OB N and the background captured in the N-th left eye frame G N L, but also the N-th left eye.
  • the eye frame G N L there is an area OC N (also referred to as a non-imaging area or an occlusion area) corresponding to a portion of the subject that is not captured because it is located behind the person OB N.
  • This occlusion region OC N is the actual pixel value is not known. Therefore, in the scene to switch to one scene and the object is completely different at the beginning of the video, the occlusion region OC N is suddenly displayed on the screen, there is a possibility of giving an uncomfortable feeling to the user.
  • the amount of change in parallax is appropriately adjusted according to various conditions so that the occlusion area OC N is not as noticeable as possible. , the amount of the occlusion region OC N is adjusted.
  • FIG. 2 is a diagram showing a schematic configuration of the information processing system 1 according to the embodiment of the present invention.
  • the information processing system 1 includes a camera 2, a distance measuring device 3, and an information processing device 4.
  • the information processing device 4 is connected to the camera 2 and the distance measuring device 3 so as to be able to transmit and receive data.
  • the camera 2 is, for example, an imaging device configured by a digital camera.
  • a digital camera has an image sensor such as a CCD.
  • the camera 2 receives light from the subject and performs an operation (so-called photographing) in which information indicating a distribution relating to the luminance of the subject is acquired as image data by photoelectric conversion.
  • This camera 2 acquires n pieces of image data by performing a plurality of times of shooting at a predetermined timing.
  • the n pieces of image data correspond to n frames constituting the moving image.
  • the Nth frame is obtained by the Nth shooting.
  • Data relating to n frames obtained by the camera 2 is transmitted to the information processing device 4 via the data line CB.
  • the distance measuring device 3 is a device that is attached to the camera 2 while maintaining the positional relationship and posture with respect to the camera 2 and measures the distance to the subject using a laser, for example.
  • the distance measuring device 3 measures the distance from the camera 2 (specifically, the image sensor) to each part of the subject, thereby measuring the distance from the camera 2 to each part of the subject at the time of each photographing.
  • the Nth distance information includes information indicating the distance (also referred to as subject distance) between the portion of the subject captured by each pixel constituting the Nth frame and the Nth viewpoint.
  • the n pieces of distance information obtained by the distance measuring device 3 corresponding to the n frames are transmitted to the information processing device 4 through the data line CB.
  • the information processing apparatus 4 is configured by a personal computer (personal computer), for example, and includes an operation unit 41, a display unit 42, and an interface (I / F) unit 43.
  • the operation unit 41 includes a mouse and a keyboard
  • the display unit 42 includes a liquid crystal display and the like
  • the I / F unit 43 receives data from the camera 2 and the distance measuring device 3.
  • the information processing apparatus 4 includes a storage unit 44, an input / output unit 45, and a control unit 46.
  • the storage unit 44 is configured by, for example, a hard disk and stores each image obtained by the camera 2.
  • the storage unit 44 stores a program PG for realizing various operations in the information processing apparatus 4. These various operations include an operation of creating a stereoscopically viewable moving image.
  • the input / output unit 45 includes, for example, a disk drive, receives the storage medium 9 such as an optical disk, and exchanges data with the control unit 46.
  • the control unit 46 includes a CPU 46a that functions as a processor and a memory 46b that temporarily stores information, and comprehensively controls each unit of the information processing apparatus 4. Further, the control unit 46 reads and executes the program PG in the storage unit 44, thereby realizing various functions and various information processing. Data temporarily generated in this information processing is appropriately stored in the memory 46b. Then, under the control of the control unit 46, the information processing apparatus 4 functions as an image processing apparatus that creates a stereoscopically viewable moving image. Note that the program stored in the storage medium 9 can be stored in the memory 46 b via the input / output unit 45.
  • FIG. 3 is a block diagram illustrating a functional configuration of the image processing apparatus realized by the control unit 46.
  • the information processing apparatus 4 includes an image acquisition unit 461, a distance information acquisition unit 462, an area detection unit 463, a target parallax setting unit 464, an initial parallax setting unit 465, a parallax adjustment unit 466, a virtual viewpoint image generation unit 467, and a stereoscopic image A generation unit 468 is provided.
  • the image acquisition unit 461 acquires a moving image including a plurality of frames stored in the storage unit 44 or the like.
  • the moving image is configured to include a plurality of frames acquired by the camera 2 performing a plurality of shootings at a predetermined timing.
  • the plurality of frames corresponds to, for example, n frames (n is a natural number of 2 or more) constituting a moving image.
  • the distance information acquisition unit 462 is distance information indicating the distance from the camera 2 to the subject when the Nth frame constituting the moving image stored in the storage unit 44 or the like is captured (also referred to as Nth distance information). To get.
  • the distance from the camera 2 to the subject is the distance from the camera 2 (specifically, the image sensor) to the portion of the subject captured by each pixel of the Nth frame when the Nth frame is captured. .
  • the area detection unit 463 is configured such that the Nth frame (Nth frame) of n frames and the Npth frame (p is a natural number) of the same n frames (Npth frame). ). By this comparison, the region detection unit 463 detects a region (also referred to as an adjusted region) where the parallax should be adjusted in the Nth frame. Note that p is preferably 1 from the viewpoint of recognizing a detailed subject change.
  • the target parallax setting unit 464 sets a target parallax (also referred to as target parallax) based on a predetermined rule for each pixel for the Nth frame included in the n frames.
  • the target parallax is predicted to be obtained from the Nth virtual viewpoint (also referred to as the Nth virtual viewpoint) with the Nth frame captured from the Nth viewpoint as a reference. This corresponds to the shift amount of each pixel for generating (also referred to as an Nth virtual viewpoint image or an Nth virtual viewpoint frame).
  • the relationship between the Nth viewpoint and the Nth virtual viewpoint is such that if the position of the Nth viewpoint is the position of the left eye of the person, the position of the Nth virtual viewpoint is the position of the right eye of the same person.
  • a condition can be considered in which a virtual straight line (also referred to as a virtual straight line) connecting the Nth viewpoint and the Nth virtual viewpoint is orthogonal to the shooting direction of the camera 2.
  • a virtual straight line also referred to as a virtual straight line
  • the Nth viewpoint and the Nth virtual viewpoint may be arranged on a virtual circumference surrounding the subject, for example, or other arrangements may be adopted.
  • the initial parallax setting unit 465 sets an initial parallax for each pixel with respect to a frame (also referred to as an appearance frame) in which the adjustment area appears among the n frames for each adjustment area.
  • the initial parallax only needs to correspond to a predetermined number (for example, 0) of pixels in the ⁇ X direction (or + X direction). For example, when the distance from the Nth viewpoint and the Nth virtual viewpoint to the subject is short, the subject viewed with the right eye shifts to the left side with respect to the subject viewed with the left eye, so the initial parallax is 0 or more in the ⁇ X direction. It is preferable to set the number of pixels.
  • the initial parallax is 0 in the + X direction in order to strongly give the user a sense that the distance to the subject is far in the 3D image.
  • a mode in which the target parallax is set in the + X direction with the number of pixels set as described above is also conceivable.
  • the parallax adjustment unit 466 adjusts the parallax of each pixel from the initial parallax toward the target parallax for a plurality of frames following the appearance frame of the n frames for each adjusted region.
  • the parallax adjustment from the initial parallax toward the target parallax varies depending on various conditions. Various conditions include the size of the target parallax in the appearance frame, the difference between the adjusted parallax and the target parallax, the position of the adjusted area in the frame, the distance from the camera 2 to the object captured in the adjusted area, and the like. Can be mentioned.
  • the virtual viewpoint image generation unit 467 generates n virtual viewpoint frames for the n frames based on the parallax information adjusted by the parallax adjustment unit 466. That is, n right-eye frames are generated from n left-eye frames, respectively.
  • the stereoscopic image generation unit 468 is a moving image (3D moving image) that can be stereoscopically viewed based on the n left-eye frames and the n right-eye frames generated by the virtual viewpoint image generation unit 467. Is generated.
  • the generated 3D moving image data is output to, for example, the storage unit 44 or the input / output unit 45 and stored in the storage unit 44 or the storage medium 9.
  • the region detection unit 463 searches for a corresponding point between the N-th frame and the N-th frame to search for the N-th frame.
  • An area also referred to as an appearance area
  • processing using a phase-only correlation method (POC method), template matching, or the like can be cited.
  • the appearance area in the Nth frame is distinguished for each object based on the Nth distance information, so that the area indicating the object that did not exist in the Npth frame is detected as the adjusted area. Thereby, the parallax can be adjusted as necessary.
  • a method of classifying the appearance region for each object for example, when the subject distance is within a predetermined threshold (for example, 10 cm) for pixels adjacent to each other, the pixels adjacent to each other represent the same object. There is a method of recognizing the captured pixel.
  • the region detection unit 463 recognizes an appearance frame in which the adjusted region appears for each adjusted region among the n frames.
  • Target parallax setting method a target parallax setting method by the target parallax setting unit 464 will be described.
  • a rule using the principle of triangulation can be cited. This rule is based on the separation distance between the Nth viewpoint and the Nth virtual viewpoint at the time of shooting the Nth frame, and the distance (subject distance) from the camera 2 to each part of the subject when the Nth frame is shot. From this, the target parallax is set.
  • the distance from the camera 2 (that is, the Nth viewpoint) to the subject 20B is D
  • the separation distance between the Nth viewpoint and the Nth virtual viewpoint is B
  • the lens of the camera 2 is
  • the focal length is f
  • the shift amount (target parallax) of the pixels capturing the same portion of the subject between the Nth frame and the Nth virtual viewpoint frame is ⁇ d
  • the separation distance B is uniquely determined according to, for example, the separation distance between the standard human right eye and left eye.
  • the focal length f is determined by the design of the camera 2.
  • the distance D is given by the Nth distance information related to the Nth frame. For this reason, the target parallax ⁇ d is obtained for each pixel of the Nth frame.
  • parallax adjustment performed by the parallax adjustment unit 466 will be described with a specific example.
  • FIG. 5 schematically shows a specific example of the N-th right eye frame G N R generated according to the target parallax as it is when the target parallax of the adjusted area in the appearance frame in which the adjusted area appears is large.
  • the N-th right-eye frames G N R shown in Figure 5 (also referred to as an object region) is a region captured person corresponding to the adjusted region OB N and, occlusion area OC N corresponding to the area OB N And are included.
  • a relatively large occlusion region OC N is shown.
  • the user has an increase in the occlusion area between a case where a frame in which no occlusion area exists is used as a reference and a case in which a frame in which an occlusion area already exists is used as a reference. Impression is different. Therefore, when the target parallax is large, the parallax adjustment amount between temporally adjacent frames (also referred to as parallax adjustment amount per frame) is gradually increased as the occlusion area increases. The uncomfortable feeling given to the user is reduced. Then, in the number of frames corresponding to a relatively long time, it is preferable that the parallax related to the adjusted region is adjusted from the initial parallax toward the target parallax.
  • FIG. 6 schematically shows a specific example of the N-th right eye frame G N R generated according to the target parallax as it is when the target parallax of the adjusted area in the appearance frame in which the adjusted area appears is small.
  • FIG. 6 in the N-th right eye frame G N R shown in FIG. 6, similarly to the case shown in FIG. 5, a region (also referred to as an object region) OB N in which a person corresponding to the adjusted region is captured, An occlusion area OC N corresponding to the area OB N is included.
  • the parallax related to the adjusted area is adjusted from the initial parallax toward the target parallax in a number of frames corresponding to a relatively short time. Thereby, it is possible to suppress both a sense of incongruity and the time required for parallax adjustment.
  • the parallax adjustment amount per frame is suppressed to be low in at least one of the initial stage and the final stage when the parallax is adjusted. . That is, it is preferable that the parallax is adjusted relatively slowly in the initial and final stages.
  • ⁇ (6-1-2) Determination of target parallax size For example, the distance used to set the target parallax, that is, the distance from the camera 2 to the object captured in the adjusted area is a predetermined threshold (for example, If the value is less than 1 m), a method for determining that the target parallax is large is conceivable.
  • the method being determined is employed on the basis of the ratio between the shift amount of the pixel corresponding to the target parallax May be. In this method, for example, if the value obtained by dividing the shift amount by the object width is equal to or greater than a predetermined threshold (for example, 0.1), it is determined that the target parallax is large.
  • a predetermined threshold for example, 0.1
  • the area of the object region OB N the method being determined based on a ratio of the area of the occlusion area OC N may be employed.
  • FIG. 7 is a graph schematically illustrating the relationship between the passage of time and the adjusted parallax when the target parallax in the adjusted region of the appearing frame is large.
  • the horizontal axis indicates the passage of time
  • the vertical axis indicates the parallax.
  • the passage of time indicates the passage of time with reference to the time of the appearance frame in the moving image.
  • the target parallax da ob of the adjusted area in the appearance frame is indicated by a straight line L10 drawn by a thick broken line, and the relationship between the passage of time and the adjusted parallax is shown by a curved line L11 drawn by a thick solid line. Yes.
  • a predetermined threshold d TH for determining whether or not the target parallax is large is shown.
  • the evaluation value V i may be, for example, the time that has elapsed since the start of parallax adjustment, the number of frames since the start of parallax, or the parallax after adjustment. It may be.
  • the threshold values ⁇ and ⁇ may be fixed values determined in advance, respectively, the time that has elapsed since the start of parallax adjustment, the number of frames since the start of parallax, and the post-adjustment It may be calculated from at least one value of parallax.
  • the evaluation value V i is the adjusted parallax and the threshold values ⁇ and ⁇ are a predetermined number% of the target parallax. As a specific example, a case where the threshold value ⁇ is 20% of the target parallax and the threshold value ⁇ is 80% of the target parallax can be considered.
  • the first period A1 of the evaluation value V i is 0 up to the threshold value alpha
  • the evaluation value V i exceeds the threshold alpha up to the threshold ⁇
  • Two periods A2 and a third period A3 after the evaluation value V i exceeds the threshold value ⁇ are included.
  • the parallax adjustment amount per frame in the first period A1 is a constant value a1
  • the parallax adjustment amount per frame in the second period A2 is a constant value a2
  • a case where the amount of parallax adjustment per frame is a constant value a3 can be considered.
  • the parallax adjustment amount per frame is gradually changed in a certain period including before and after the evaluation value V i reaches the threshold values ⁇ and ⁇ from the viewpoint of giving the user as much discomfort as possible.
  • the parallax change in a curve by gradually changing the parallax adjustment amount.
  • the parallax gradually reaches the target parallax after increasing in an accelerated manner.
  • the parallax reaches from the initial parallax d0 to d1a in the first period A1 in which the elapsed time is T0 to T1a, and the parallax from d1a in the second period A2 in which the elapsed time is T1a to T2a. reaches d2a, disparity in the third period A3 elapsed time is T2a ⁇ T3a is shown an example to get from d2a to da ob.
  • the parallax of each pixel is directed from the initial parallax to the target parallax for a plurality of frames related to the first period A1, the second period A2, and the third period A3 following the appearance frame of the n frames. Adjusted.
  • the parallax change rate (also referred to as first change rate) obtained by dividing the parallax change amount in the first period A1 by the length of the first period A1 is the parallax change rate in the second period A2. It is smaller than the parallax change rate (also referred to as second change rate) obtained by dividing the change amount by the length of the second period A2.
  • the second change rate is larger than the parallax change rate (also referred to as the third change rate) obtained by dividing the parallax change amount in the third period A3 by the length of the third period A3.
  • the second rate of change is preferably several times to several tens of times the first rate of change and the third rate of change, for example.
  • the parabolic change of the parallax as shown by the curve L11 can be realized by various methods.
  • the various methods include a method using a pattern in which the parallax changes in a curve (also referred to as an adjustment pattern) that is prepared in advance, and a method of changing the parallax according to a predetermined function.
  • each type of adjustment pattern may be indicated by data in which the parallax changes in a curve from the initial parallax toward the target parallax over time, and may be stored in advance in the storage unit 44.
  • this adjustment pattern should just be expanded-contracted in the direction of passage of time according to the period required for the parallax adjustment determined according to the difference between the initial parallax and the target parallax, for example.
  • the parallax adjustment according to such an adjustment pattern makes it possible to easily adjust the parallax accurately.
  • the number of frames per second in a moving image is 60, that is, when the frame rate is 60 frames per second
  • the number of frames required for parallax adjustment is, for example, a relatively large number such as about 120 to 180. Become.
  • a mode in which the parallax of the (N + 1) th and subsequent frames is adjusted based on the initial parallax and the target parallax can be considered.
  • a mode in which the parallax of the (N + 1) th and subsequent frames is adjusted based on the adjusted parallax and the target parallax in the Nth Nth frame among the n frames is conceivable.
  • PID control based on the difference between the initial parallax and the target parallax and the difference between the parallax of the adjusted Nth frame and the target parallax is possible. Conceivable.
  • the integral term coefficient, the differential term coefficient, and the proportional term coefficient may be changed. As a result, the parallax is adjusted so as not to reach the target parallax immediately, but to reach the target parallax from the initial parallax with a certain delay time.
  • threshold values such as threshold values ⁇ and ⁇ are not used, and the parallax may be adjusted according to a predetermined parameter.
  • the predetermined parameter include the time already required for parallax adjustment, the time used for parallax adjustment, the relationship between the adjusted parallax and the target parallax, and the like.
  • a mode in which the amount of parallax adjustment is calculated by substituting the time already required for parallax adjustment into a function is conceivable.
  • a mode in which a time used for parallax adjustment is set according to the size of the target parallax, and a curved adjustment pattern indicating the relationship between the passage of time and the parallax is expanded and contracted according to the time is considered.
  • FIG. 8 is a graph schematically illustrating the relationship between the passage of time and the adjusted parallax when the target parallax in the adjusted region of the appearing frame is small.
  • the horizontal axis indicates the passage of time and the vertical axis indicates the parallax
  • the passage of time indicates the passage of time with reference to the time of the appearing frame in the moving image.
  • the target parallax db ob of the adjusted area in the appearance frame is indicated by a straight line L20 drawn by a thick broken line, and the relationship between the passage of time and the adjusted parallax is shown by a curved line L21 drawn by a thick solid line.
  • a predetermined threshold value d TH for determining the magnitude of the target parallax is shown.
  • the evaluation value V i and the threshold values ⁇ and ⁇ may be values of the types described above in the description of parallax adjustment when the target parallax is large.
  • the evaluation value V i is the adjusted parallax
  • the threshold ⁇ is 5% of the target parallax
  • the threshold ⁇ is 80% of the target parallax
  • the first period B1 from evaluation value V i is 0 up to the threshold value alpha
  • the evaluation value V i exceeds the threshold alpha up to the threshold ⁇
  • Two periods B2 and a third period B3 after the evaluation value V i exceeds the threshold value ⁇ are included.
  • the amount of parallax adjustment per frame may be, for example, a constant amount b1 in the first period B1, a constant amount b2 in the second period B2, and a constant amount b3 in the third period B3.
  • the parallax per frame in a certain period including before and after the evaluation value V i reaches the threshold values ⁇ and ⁇ . It is preferable that the adjustment amount is gradually changed.
  • the parallax reaches from the initial parallax d0 to d1b in the first period B1 in which the elapsed time is T0 to T1b, and the parallax from the d1b in the second period B2 in which the elapsed time is T1b to T2b. reaches d2b, disparity in the third period B3 elapsed time is T2b ⁇ T3b is shown an example that reaches d3 ob from d2b.
  • the parallax curve change with respect to time as shown by the curve L21 is, for example, a method for realizing the parallax curve change with respect to time as shown by the curve L11 in FIG.
  • FIG. 9 and FIG. 10 are flowcharts illustrating an operation flow relating to parallax adjustment according to the target parallax size. This operation flow is realized by reading and executing the program PG in the storage unit 44 by the control unit 46. For example, the generation of a 3D moving image in the information processing apparatus 4 is requested in accordance with the operation of the operation unit 41 by the user, and this operation flow is started.
  • the moving image including n frames stored in the storage unit 44 and the like is acquired by the image acquisition unit 461.
  • step S2 the distance information acquisition unit 462 acquires distance information corresponding to each of the n frames acquired in step S1 from the storage unit 44 or the like.
  • This distance information is information indicating the distance from the camera 2 to the subject when each of n frames constituting the moving image is photographed.
  • an example is shown in which the process of step S1 and the process of step S2 are executed in parallel, but the process of step S1 and the process of step S2 may be executed in this order. These steps may be executed in the reverse order.
  • step S3 the region detection unit 463 compares the Nth frame of the n frames with the Np frame (p is a natural number) of the same n frames, so that A region to be adjusted in the Nth frame is detected.
  • N is set in order from 1 to n, and among the Nth frames, areas indicating objects that did not exist in the Npth frame are detected as adjusted areas.
  • N 1, an area indicating all objects in the first frame is detected as an adjusted area.
  • identification information here, an identification number
  • step S4 the target parallax setting unit 464 sets the target parallax for each pixel based on a predetermined rule for each frame included in the n frames.
  • step S5 the identification number M of the adjustment target area to be processed is set to 1 by the initial parallax setting unit 465.
  • step S6 the initial parallax setting unit 465 designates the M-th adjusted area among all the adjusted areas detected in step S3, and the process proceeds to step S11 in FIG.
  • step S11 the initial parallax setting unit 465 sets the initial parallax for each pixel of the appearing frame in which the adjusted area appears for the adjusted area specified in step S6.
  • step S12 the parallax adjustment unit 466 determines whether or not the target parallax for the adjusted area specified in step S6 is equal to or greater than a predetermined threshold value dTH .
  • a predetermined threshold value dTH it is determined whether or not the minimum value of the target parallax set in step S4 for the adjusted region specified in step S6 is equal to or greater than a predetermined threshold value dTH . If the target parallax is greater than or equal to the predetermined threshold d TH , the process proceeds to step S13, and if the target parallax is less than the predetermined threshold d TH , the process proceeds to step S14.
  • the determination is performed using the minimum value of the target parallax.
  • the determination is not limited to this. For example, the maximum value of the target parallax or the average value of the target parallax may be used. The determination may be made using the calculated parameter.
  • step S13 the parallax adjustment unit 466 performs parallax adjustment when the target parallax is large.
  • step S14 the parallax adjustment unit 466 performs parallax adjustment when the target parallax is small.
  • step S15 the initial parallax setting unit 465 determines whether all the adjustment areas detected in step S3 have already been specified in step S6. If all the adjusted areas have already been specified, the process proceeds to step S16. If all the adjusted areas have not been specified yet, the process proceeds to step S17.
  • step S16 the initial parallax setting unit 465 sets the identification number M of the adjustment target area to be processed to one larger value, and the process proceeds to step S6 in FIG.
  • n virtual viewpoint frames are generated by the virtual viewpoint image generation unit 467 based on the parallax information adjusted in steps S13 and S14 for n frames constituting the moving image.
  • step S18 the stereoscopic image generating unit 468 can perform stereoscopic viewing (3D moving image) based on the n frames acquired in step S1 and the n virtual viewpoint frames generated in step S17. Is generated. Thereby, this operation flow is completed.
  • FIG. 11 is a schematic view illustrating an aspect of parallax adjustment related to a virtual viewpoint image. If the amount of parallax adjustment per frame is large, for example, the first to third right eye frames corresponding to the first to third virtual viewpoint frames, as shown by the left route in FIG. In G 1 C to G 3 C, the occlusion area increases rapidly. Thereafter, the parallax in the N-th right eye frame G N R corresponding to the N-th virtual viewpoint frame becomes the target parallax. In such a case, the amount of parallax adjustment per frame becomes too large, increasing the burden on the user's eyes.
  • the parallax adjustment amount per frame is suppressed to a certain amount or less for the purpose of giving the user's eyes a gentle and uncomfortable feeling.
  • an upper limit value may be provided for the parallax adjustment amount per frame.
  • the occlusion area gradually increases in the first to third right eye frames G 1 CR to G 3 R corresponding to the first to third virtual viewpoint frames. .
  • the parallax adjustment unit 466 adjusts the parallax of each pixel so that the parallax adjustment amount per frame does not exceed the upper limit value. Therefore, since the remarkable change of parallax is suppressed, the visual discomfort given to a user is suppressed appropriately.
  • FIG. 12 is a graph schematically illustrating the relationship between the passage of time and the adjusted parallax when an upper limit is provided for the parallax adjustment amount per frame.
  • the horizontal axis indicates the passage of time and the vertical axis indicates the parallax
  • the passage of time is the passage of time with reference to the time of the appearing frame in the moving image.
  • the target parallax dc ob of the adjusted area in the appearance frame is indicated by a straight line L30 drawn by a thick broken line, and the relationship between the passage of time and the adjusted parallax is shown by a curved line L31 drawn by a thick solid line.
  • the relationship between the passage of time and the adjusted parallax when the upper limit value is not set for the parallax adjustment amount per frame is indicated by a curved line L32 drawn by a one-dot chain line. Yes.
  • the upper limit is set for the parallax adjustment amount per frame even when the parallax increases rapidly as shown by the curve L32. Is provided, the parallax does not increase abruptly as indicated by the curve L31.
  • Such control can be realized by simple control in which, for example, when the parallax adjustment amount exceeds the upper limit value, the adjustment amount derived based on the function or the adjustment pattern is replaced with the upper limit value. .
  • an upper limit is set for the amount of parallax adjustment per frame, for example, if the frame rate is 60 frames per second, the number of frames required for parallax adjustment is relatively large, for example, about 120 to 180 Number.
  • a method of controlling the parallax adjustment amount per frame to be equal to or less than the upper limit value by expanding and contracting a curvilinear adjustment pattern indicating the relationship between the passage of time and the parallax is also conceivable.
  • the parallax adjustment unit 466 uses the target parallax of each pixel in the N + Q-th frame (N and Q are natural numbers) out of n frames constituting the moving image, and the parallax adjustment unit 466
  • the parallax of the (N + 1) th and subsequent frames may be adjusted based on the parallax of the Nth frame (Nth frame) that has already been adjusted.
  • the frame rate is 60 frames per second, for example, a configuration in which Q is set to about 1 to 60 is conceivable. Thereby, it is possible to easily adjust the parallax corresponding to the change in the target parallax.
  • the amount of parallax adjustment per frame is increased.
  • the difference is less than the threshold value, a configuration in which the amount of parallax adjustment per frame is reduced can be considered.
  • the target parallax in the N + Q frame is Occ N + Q
  • the adjusted parallax in the Nth frame is Occ N
  • the threshold is ⁇ th
  • parallax adjustment is preferably performed after, for example, the above-described parallax adjustment when the target parallax is large or parallax adjustment when the target parallax is small. Thereby, the visual discomfort given to the user when generating a pseudo moving image viewed from a virtual viewpoint is reduced.
  • FIG. 13 is a graph schematically illustrating the relationship between the passage of time and the adjusted parallax when the parallax is adjusted according to the target parallax that varies with the passage of time.
  • the target parallax that fluctuates with the passage of time is indicated by a curve L40 drawn with a thick broken line
  • the relationship between the elapsed time and the adjusted parallax is shown by a curve L41 drawn with a thick solid line.
  • the parallax adjustment in the direction in which the parallax increases and the parallax adjustment in the direction in which the parallax decrease are performed, respectively.
  • the adjustment of the parallax in the direction in which the parallax decreases may be performed according to the amount of change in the difference between the target parallax and the adjusted parallax, for example.
  • the adjustment of the parallax according to the variation of the target parallax is realized by, for example, PID control based on the parallax after the adjustment for the Nth frame and the target parallax in the N + Q frame. More specifically, the parallax is adjusted according to a function including an integral term, a differential term, a proportional term, and the like according to the difference between the adjusted parallax for the Nth frame and the target parallax in the N + Q frame.
  • a function including an integral term, a differential term, a proportional term, and the like according to the difference between the adjusted parallax for the Nth frame and the target parallax in the N + Q frame.
  • an arithmetic value such as an average value and a center value for the target parallax related to the (N + Q) th frame and the target parallax related to one or more frames that are temporally adjacent to the (N + Q) th frame
  • the parallax may be adjusted according to the difference from the adjusted parallax.
  • the parallax of each pixel may be adjusted according to the position of the subject captured in each frame. Thereby, the parallax can be adjusted according to the degree of attention of the user.
  • parallax adjustment is performed so that the parallax gradually increases without increasing the parallax for each pixel in the adjusted region located near the center of each frame. .
  • parallax adjustment is performed so that the parallax increases relatively quickly and approaches the target parallax for each pixel in the adjusted region located near the end of each frame.
  • FIG. 14 is a schematic view illustrating one aspect of parallax adjustment related to a virtual viewpoint image.
  • the parallax gradually increases in the adjusted region OB1 N where the object is captured near the center, and the object is near the end.
  • OB2 N For the captured adjustment area OB2 N , a mode in which parallax adjustment is performed such that parallax increases relatively quickly can be considered.
  • 15 and 16 are graphs schematically illustrating the relationship between the time and the adjusted parallax when the parallax is adjusted according to the position of the adjusted region in the frame.
  • the target parallax is indicated by a straight line L10 drawn with a thick broken line
  • the relationship between the elapsed time and the adjusted parallax for the adjusted region near the center of the frame is indicated by a curved line L11 drawn with a thick line.
  • the target parallax is indicated by a straight line L10 drawn with a thick broken line, and a curve L13 in which the relationship between the elapsed time and the adjusted parallax for the adjusted area near the end of the frame is drawn with a thick line. It is shown in
  • Such adjustment of parallax is performed according to the position of the adjusted area in the appearance frame in which the adjusted area appears, for example. More specifically, the frame may be divided into a plurality of regions from the vicinity of the center to the vicinity of the end portion, and different parallax adjustment may be performed for each pixel in the adjusted region for each of the divided regions. In addition, it is preferable that the number of areas to be divided is large because the user does not feel uncomfortable. Further, the parallax adjustment may be switched for each divided area, but the parallax adjustment amount may be calculated according to the position information (for example, address) of the pixel on the frame.
  • the amount of parallax adjustment is not only position information but also information on at least one of adjusted parallax, changed target parallax, time already used for parallax adjustment, time required for parallax adjustment, etc.
  • a mode determined according to the above is also conceivable.
  • an adjustment pattern in which the parallax changes in a curve as time passes is prepared for each area divided for the frame, the adjustment pattern is selected according to the position information of the to-be-adjusted adjustment area, and the selected adjustment pattern
  • the parallax may be adjusted according to the above.
  • the adjustment pattern may be expanded or contracted in at least one of the time direction and the magnitude direction of the parallax, depending on the time required for parallax adjustment, the size of the target parallax, and the like.
  • the position of the adjusted area in the frame changes with the passage of time.
  • the parallax adjustment content may be changed in accordance with the change in the position of the adjustment region in the frame. For example, when a region to be adjusted positioned near the center of the frame moves to the vicinity of the end of the frame, the parallax may be adjusted according to the vicinity of the end.
  • the parallax adjustment performed according to the position of the adjusted region in such a frame depends on the parallax adjustment when the target parallax is small, the parallax adjustment when the target parallax is large, and the variation of the target parallax. It may be appropriately combined with one or more parallax adjustments among the parallax adjustments.
  • the parallax of each pixel may be adjusted for another subject different from the one subject in accordance with the parallax related to the one subject captured in the moving image. Thereby, since the consistency of parallax between objects improves, the visual discomfort given to a user is suppressed appropriately.
  • substantially the same parallax may be set for the adjusted areas related to the plurality of objects.
  • substantially the same parallax may be set for the adjusted areas related to the plurality of objects.
  • the user is visually uncomfortable. Therefore, based on the distance information acquired by the distance information acquisition unit 462, a plurality of adjusted areas related to a plurality of objects having substantially the same distance from the camera 2 are grouped, and the same parallax adjustment is performed. It is preferable.
  • FIG. 17 is a schematic diagram illustrating an aspect of parallax adjustment related to a virtual viewpoint image in a case where parallax is adjusted by grouping a plurality of adjusted areas according to distance information.
  • first to third adjusted areas OB1 N to 3 in which three people having the same distance from the camera 2 in the N-th right eye frame G N R are respectively captured.
  • OB3 N conceivable manner such as the adjustment of the same parallax is performed.
  • FIG. 18 is a graph schematically illustrating the relationship between time and adjusted parallax when a plurality of adjusted areas are not grouped for comparison.
  • FIG. 19 is a graph schematically illustrating the relationship between time and adjusted parallax when a plurality of adjusted areas are grouped.
  • the target parallax is drawn with a thick broken line L50, and the relationship between the elapsed time and the adjusted parallax is drawn with a solid curve L51 as the relationship relating to the first adjusted area OB1 N.
  • the relationship related to the adjusted region OB2 N is drawn with a bold curve L52, and the relationship related to the third adjusted region OB3 N is drawn with a dashed-dotted curve L53.
  • the parallax adjustment indicated by the curves L51 to L53 considerable variations occur in the parallax after the adjustment even if the elapsed time is the same. In this case, the user is visually uncomfortable.
  • the parallax adjustment related to the first to third adjusted areas OB1 N to OB3 N is performed in a unified manner. Is preferred. Thereby, since the consistency of parallax between objects improves, the visual discomfort given to a user is suppressed appropriately.
  • a mode in which a plurality of adjusted regions are grouped with respect to an object whose distance from the camera 2 is within a predetermined value for example, ⁇ several tens of centimeters
  • a mode in which a plurality of adjusted areas in which the target parallax is included in a predetermined range for example, ⁇ several pixels
  • the amount of parallax adjustment is determined according to at least one or more information among the adjusted parallax, the changed target parallax, the time already used for parallax adjustment, the time required for parallax adjustment, and the like.
  • the parallax may be adjusted according to an adjustment pattern in which the parallax changes in a curve with time.
  • the adjustment pattern may be expanded or contracted in at least one of the time direction and the magnitude direction of the parallax, depending on the time required for parallax adjustment, the size of the target parallax, and the like.
  • the adjusted area used as a reference for parallax adjustment is an adjusted area closest to the center of the frame.
  • the evaluation value indicating the highest probability of searching for a corresponding point in the stereo image is the highest among the plurality of adjusted regions.
  • the adjustment area may be used as a reference for parallax adjustment.
  • the evaluation value when the corresponding point search process is performed by POC (phase-only correlation method), a value (for example, an average value, an integrated value, or the like) according to the peak value of the POC value for the adjusted region. ) May be adopted as the evaluation value.
  • the parallax adjustment performed by grouping a plurality of adjusted areas according to such distance information includes parallax adjustment when the target parallax is small, parallax adjustment when the target parallax is large, and target parallax.
  • One or more parallax adjustments among the parallax adjustment according to the fluctuation and the parallax adjustment according to the position in the frame may be appropriately combined.
  • the parallax is adjusted according to the magnitude of the target parallax, the fluctuation of the target parallax, the position on the screen, and the distance information.
  • the present invention is not limited to this, and other parallax adjustment methods may be employed.
  • the parallax may be adjusted by increasing / decreasing the time required for parallax adjustment according to the time during which a region to be adjusted related to an object exists in the moving image.
  • the parallax may be adjusted by increasing or decreasing the time required for the parallax adjustment according to the number of frames in the moving image having n frames that include the adjustment area related to an object.
  • the parallax adjustment unit 466 may adjust the parallax of each pixel in the adjusted region from the initial parallax to the target parallax for a number of frames corresponding to a predetermined time.
  • the predetermined time mentioned here is set to a time within 50% of the time during which a region to be adjusted relating to a certain object exists in the moving image.
  • parallax adjustment for example, according to a predetermined time, an adjustment pattern in which the parallax changes in a curve as time passes is expanded and contracted in at least one of the time direction and the magnitude direction of the parallax,
  • the parallax may be adjusted according to the adjustment pattern after expansion / contraction.
  • FIG. 20 schematically illustrates the relationship between the time and the adjusted parallax when the parallax of each pixel in the adjusted region is adjusted from the initial parallax to the target parallax in a number of frames corresponding to a predetermined time. It is a graph.
  • the target parallax dd ob is drawn by a thick broken line L60.
  • the predetermined time may be set so that the time required for adjusting the parallax related to each pixel in the adjusted area is always constant, or the parallax related to each pixel in the adjusted area depending on the size of the target parallax The time required for the adjustment may be adjusted uniquely.
  • the parallax is adjusted so that the parallax changes in a curve with the passage of time.
  • the parallax may be adjusted so that the parallax changes in proportion to the passage of time in each of the first period A1, the second period A2, and the third period A3. That is, the parallax change rate (also referred to as first change rate) obtained by dividing the parallax change amount in the first period A1 by the length of the first period A1 is the parallax change rate in the second period A2. It may be smaller than the change rate of parallax obtained by dividing by the length of the two periods A2 (also referred to as second change rate).
  • the parallax may be adjusted so that the parallax changes in proportion to the passage of time in each of the first period B1, the second period B2, and the third period B3. That is, the parallax change rate obtained by dividing the parallax change amount in the first period B1 by the length of the first period B1 divides the parallax change amount in the second period B2 by the length of the second period B2.
  • the parallax may be adjusted so as to be smaller than the change rate of the parallax obtained by this.
  • the target parallax is set using the principle of triangulation, but the present invention is not limited to this.
  • the target parallax may be set by other methods such as an MTD (Modified Time Difference) method and a CID (Computed Image Depth) method. That is, the target parallax may correspond to the separation distance between the Nth viewpoint and the Nth virtual viewpoint related to the Nth frame and the distance from the camera 2 to the subject when the Nth frame is captured.
  • the stereoscopic image is generated based on the moving image taken by the camera 2, but the present invention is not limited to this.
  • a moving image shot by another camera is stored in the storage medium 9 or the like, and is read from the storage medium 9 into the control unit 46 via the input / output unit 45, and a stereoscopic image is generated based on the moving image. May be.
  • the N-th frame G N is used directly as the N left-eye frame G N L, but the N-th frame G N from the N right-eye frames G N R is generated, It is not limited to this.
  • the N-th frame G N is used directly as the N right-eye frames G N R, N-th frame G N N-th left-eye frame G N L from may be generated.
  • the Nth frame G N is not used as either the Nth left eye frame G N L or the Nth right eye frame G N R, and the Nth frame for the left eye G based on the Nth frame G N.
  • N L and the N-th right eye frame G N R may be generated.
  • the Nth distance information is acquired by the distance measuring device 3, but the present invention is not limited to this.
  • the Nth distance information may be acquired by using a stereo image taken by a stereo camera or the like and using the principle of triangulation.
  • the stereo image can be used as it is as the Nth left eye frame G N L and the Nth right eye frame G N R. Become.
  • the parallax adjustment processing is switched between a case where the target parallax is large and a case where the target parallax is small.
  • the present invention is not limited to this.
  • the size of the target parallax may be divided into three or more levels, and parallax adjustment may be performed according to the level corresponding to the target parallax size.
  • the relationship between the Nth viewpoint and the Nth virtual viewpoint is described as being a standard positional relationship between the right eye and the left eye of a human, but is not limited thereto.
  • the relationship between the Nth viewpoint and the Nth virtual viewpoint may be another relationship such as the relationship between the right eye and the left eye of an insect.
  • various functions and various information processing are realized by executing the program PG in the control unit 46.
  • the present invention is not limited to this.
  • various functions and various information processing may be realized by a dedicated electronic circuit
  • various functions and various information processing may be realized by executing a dedicated electronic circuit and a program. .

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Processing Or Creating Images (AREA)

Abstract

La présente invention consiste à fournir une technologie qui réduit l'inconfort visuel subi par un utilisateur lors de la génération d'une image pseudo-mobile, vue depuis un point de vue virtuel. Pour parvenir à cet objectif, un dispositif de traitement d'image est muni : d'une unité de capture d'image qui capture une image mobile qui comprend une pluralité de trames ; d'une unité de paramétrage de parallaxe cible qui définit une parallaxe cible pour chaque trame, sur la base de règles prédéfinies pour chaque pixel ; d'une unité de paramétrage de parallaxe initiale qui définit la parallaxe initiale pour chaque pixel de la première trame parmi une pluralité de trames ; et d'une unité d'ajustement de parallaxe qui ajuste la parallaxe pour chaque pixel allant de la parallaxe initiale vers la parallaxe cible, pour une pluralité de trames liées à une première et à une seconde période qui se suivent séquentiellement, à partir de la première trame de la pluralité de trames. Dans le dispositif de traitement d'image, un premier taux de changement, obtenu en divisant le changement dans la parallaxe dans une première période par la longueur de la première période, est inférieur à un second taux de changement obtenu en divisant le changement dans la parallaxe dans une seconde période par la longueur de la seconde période.
PCT/JP2011/071079 2010-09-22 2011-09-15 Dispositif et procédé de traitement d'image et programme WO2012039340A1 (fr)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2012535012A JP5601375B2 (ja) 2010-09-22 2011-09-15 画像処理装置、画像処理方法、およびプログラム

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2010-211670 2010-09-22
JP2010211670 2010-09-22

Publications (1)

Publication Number Publication Date
WO2012039340A1 true WO2012039340A1 (fr) 2012-03-29

Family

ID=45873827

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2011/071079 WO2012039340A1 (fr) 2010-09-22 2011-09-15 Dispositif et procédé de traitement d'image et programme

Country Status (2)

Country Link
JP (1) JP5601375B2 (fr)
WO (1) WO2012039340A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2013235304A (ja) * 2012-05-02 2013-11-21 Sony Corp 画像処理装置および方法、並びにプログラム

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11164328A (ja) * 1997-11-27 1999-06-18 Toshiba Corp 立体映像表示装置
JP2000253422A (ja) * 1999-03-03 2000-09-14 Toshiba Corp ２次元映像からの立体映像生成方法
JP2010141446A (ja) * 2008-12-10 2010-06-24 Brother Ind Ltd ヘッドマウントディスプレイ及びヘッドマウントディスプレイにおける画像提示方法

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH09271043A (ja) * 1996-03-29 1997-10-14 Olympus Optical Co Ltd 立体映像ディスプレイ装置
JP5562122B2 (ja) * 2010-05-31 2014-07-30 キヤノン株式会社 画像処理装置及びその制御方法

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH11164328A (ja) * 1997-11-27 1999-06-18 Toshiba Corp 立体映像表示装置
JP2000253422A (ja) * 1999-03-03 2000-09-14 Toshiba Corp ２次元映像からの立体映像生成方法
JP2010141446A (ja) * 2008-12-10 2010-06-24 Brother Ind Ltd ヘッドマウントディスプレイ及びヘッドマウントディスプレイにおける画像提示方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2013235304A (ja) * 2012-05-02 2013-11-21 Sony Corp 画像処理装置および方法、並びにプログラム

Also Published As

Publication number Publication date
JP5601375B2 (ja) 2014-10-08
JPWO2012039340A1 (ja) 2014-02-03

Similar Documents

Publication Publication Date Title
JP5963422B2 (ja) 撮像装置、表示装置、コンピュータプログラムおよび立体像表示システム
US8760502B2 (en) Method for improving 3 dimensional effect and reducing visual fatigue and apparatus enabling the same
US9204140B2 (en) Display device and display method
KR101518531B1 (ko) 스테레오스코픽 모션 픽쳐들의 잠재적인 아이스트레인을 측정하기 위한 시스템 및 방법
WO2012086120A1 (fr) Appareil de traitement d'image, appareil de captation d'image, procédé de traitement d'image et programme
WO2013099169A1 (fr) Dispositif de photographie stéréoscopique
US9420261B2 (en) Image capturing apparatus, method of controlling the same and program
JPH08317429A (ja) 立体電子ズーム装置及び立体画質制御装置
JP2007052304A (ja) 映像表示システム
CN106303498B (zh) 视频显示控制方法和装置、显示设备
KR100726933B1 (ko) 고정된 양안 카메라의 자동 주시각 제어를 위한 영상신호처리 방법
US20120242791A1 (en) 3-d video processing device and 3-d video processing method
US9258546B2 (en) Three-dimensional imaging system and image reproducing method thereof
JP5601375B2 (ja) 画像処理装置、画像処理方法、およびプログラム
JP5741353B2 (ja) 画像処理システム、画像処理方法および画像処理プログラム
KR101747929B1 (ko) 입체영상의 화질을 개선하기 위한 장치 및 방법
US20140125652A1 (en) Look-ahead convergence for optimizing display rendering of sterioscopic videos and images
JP5838775B2 (ja) 画像処理方法、画像処理システムおよび画像処理プログラム
US20160065941A1 (en) Three-dimensional image capturing apparatus and storage medium storing three-dimensional image capturing program
CN102478756B (zh) 用来拍摄立体影片的方法与装置
CN106303315B (zh) 视频显示控制方法和装置、显示设备
KR101939243B1 (ko) 입체 깊이 조정 및 초점 조정
KR101907127B1 (ko) 입체 비디오 주밍 및 비디오 내의 전경 및 배경 검출
JPWO2012001958A1 (ja) 画像処理装置および方法並びにプログラム
JP2015029215A (ja) 立体画像処理装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 11826785

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2012535012

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 11826785

Country of ref document: EP

Kind code of ref document: A1