JP2013034084A - Depth creation support device, depth creation support method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a depth creation support device or the like which allows accurate and efficient creation of a depth image to be used by a parallax image generating device.SOLUTION: When buttons 21 are depressed, inputted images for creating a depth image are read in. Displayed respectively in a start frame display area 23 and a finish frame display area 24 are mask images of the start frame and the finish frame of a mask selected from a mask-selection pulldown menu 25. In a two-dimensional image display area 28, a preview image of a two-dimensional image is displayed. In a depth setting screen 26, the depth value curves of read masks are displayed. When a frame slider bar 34 and a depth value slider bar 37 are slid, a depth setting frame and a depth value can be changed.

Description

  The present invention relates to a depth production support apparatus, a depth production support method, and a program, and more particularly, to a depth production support apparatus that produces a depth image used by a parallax image generation apparatus that generates a parallax image for realizing stereoscopic vision. .

  Conventional three-dimensional video display systems are roughly classified into a two-parallax system and a multi-parallax system having three or more parallaxes. In either method, it is necessary to prepare a video in which the scene is viewed from the required number of viewpoints.

  The most direct video production method is to prepare cameras for the required number of parallaxes and shoot the same scene with a plurality of cameras arranged at predetermined intervals. Recently, cameras that can directly shoot two parallaxes are commercially available, but cameras that can directly shoot multiple parallaxes are not common.

  A method of creating a three-dimensional CG (Computer Graphics) image and setting a plurality of camera positions for rendering is also conceivable. The three-dimensional CG image includes depth information, and a parallax image corresponding to the multi-parallax method can be created from the three-dimensional image. However, when the number of camera positions is large, there is a problem that the number of renderings increases, which takes time and cost.

  Furthermore, most videos are not CG images but are real photographs, and it is required to create a three-dimensional video from the real photographs.

  There are several methods (so-called two-dimensional / three-dimensional conversion methods) for converting a multi-parallax image based on a plane (2D) image to produce a multi-parallax image based on a real image.

  However, there is no depth information in the method of converting to a multi-parallax image. Therefore, it is necessary to create depth information one by one in every scene (frame), and the work load is very high.

  Thus, as a method for setting the depth value, for example, Patent Document 1 proposes a technique for changing the mesh shape in accordance with a user instruction and setting the depth value. For example, Non-Patent Document 1 proposes a technique in which a user moves and rotates a selected portion of a video in a three-dimensional space to set a depth value.

Special table 2010-516155 gazette

Depth Director: A System for Adding Depth to Movies

  However, the technique described in Patent Document 1 has a problem that there are many items to be operated, and it takes a long time to set the depth information for one frame, and it is difficult to confirm the set depth value.

  On the other hand, the technique described in Non-Patent Document 1 has a problem that although the setting method is simplified, the depth values in different frames cannot be compared, so that the positional relationship tends to be inconsistent.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide depth production support that assists in accurately and efficiently producing a depth image used by a parallax image generation device. It is to provide a device or the like.

In order to achieve the above-described object, the first invention is a depth production support apparatus for producing a depth image used for generating a parallax image for realizing stereoscopic vision, and for each frame of an original video. On the other hand, reading means for reading a plurality of binarized mask images for identifying a specific subject, and a depth value curve displaying means for displaying a depth value curve indicating a depth value in each frame for each mask image And setting means for setting a depth value of a single frame in the depth value curve of the selected single mask image in the depth value curve display means; and for the frame for which the depth value has been set by the setting means. Interpolating means for interpolating depth values before and after, and depth image creating means for creating the depth image based on the interpolation result by the interpolating means. A depth production support apparatus according to symptoms.
According to the first invention, a depth image used by the parallax image generation device can be accurately and efficiently manufactured.

The setting means has a depth value designation component which is a screen component for designating a depth value in the depth value curve display means, and a pixel value indicating the depth value in conjunction with movement of the depth value designation component The image processing apparatus further includes depth image display means for displaying a preview of the depth image while changing.
Thereby, the user can easily confirm whether there is no sense of incongruity between subjects in the same frame.

The setting means includes a time designation component that is a screen component for designating time in the depth value curve display means, and the depth image display means is configured to display a depth image in conjunction with the movement of the time designation component. Change the frame.
Thereby, the user can easily confirm whether there is no sense of incongruity between frames.

Video display means for displaying the original video while changing the video time in conjunction with the movement of the time designation component is further provided.
Thereby, the user can easily confirm the finish of the depth image.

The image processing apparatus further includes label image creation means for creating a label image that is an image in which a label value for identifying a subject constituting the original video is replaced with a pixel value based on an interpolation result by the interpolation means.
Thereby, a label image can be created together with the depth image.

The image processing apparatus further includes a mask range detection unit that detects a range of a frame where the subject exists in any one mask image and determines a start frame and an end frame of the depth value setting range.
As a result, only the frames for which the depth value needs to be set can be displayed on the display means, and convenience is improved.

A second invention is a depth production support method of a depth production support apparatus for producing a depth image used for generating a parallax image for realizing stereoscopic vision, and is specified for each frame of the original video A reading step of reading a plurality of binarized mask images to identify a subject, a depth value curve displaying step of displaying a depth value curve indicating a depth value in each frame for each mask image, and the depth A setting step for setting a depth value of a single frame in the depth value curve of the selected single mask image in the value curve display step; and a depth value before and after the frame for which the depth value has been set in the setting step An interpolation step for interpolating the depth image, and a depth image creation step for creating the depth image based on the interpolation result of the interpolation step. Characterized in that it comprises a and.
According to the second invention, it is possible to accurately and efficiently produce a depth image used by the parallax image generating device.

  According to a third aspect of the present invention, a reading step of reading a plurality of binarized mask images for identifying a specific subject for each frame of the original video, and a depth value in each frame for each mask image A depth value curve display step for displaying a depth value curve indicating, and a setting step for setting a depth value of a single frame in a depth value curve of a single mask image selected in the depth value curve display step; An interpolation step for interpolating depth values before and after the frame for which the depth value has been set in the setting step, and a depth used for generating a parallax image for realizing stereoscopic vision based on an interpolation result by the interpolation step A depth image creating step for creating an image.

  The depth production support apparatus according to the first invention can be obtained by installing the program according to the third invention in the computer.

  According to the present invention, it is possible to provide a depth production support device and the like that can accurately and efficiently produce a depth image used by a parallax image generation device.

It is a hardware block diagram of the depth production assistance apparatus which concerns on embodiment of this invention. It is a figure which shows the input-output data of a depth production assistance apparatus and a parallax image generation apparatus. It is a figure which shows an example of a two-dimensional image. It is a figure which shows an example of the background image produced | generated based on the two-dimensional image shown in FIG. It is a figure which shows an example of a mask image. It is a figure which shows an example of another mask image. It is a figure which shows an example of another mask image. It is a figure which shows an example of a depth image. It is a figure for demonstrating matching of an input-output image. It is a flowchart explaining a depth production assistance process. It is an example of a display of a depth production assistance screen. It is the example of a display which expanded the depth setting screen. It is another example of a display of a depth setting screen. It is another example of a display of a depth setting screen. It is another example of a display of a depth setting screen. It is another example of a display of a depth setting screen. It is a figure explaining preparation of a label image.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[Embodiments of the present invention]
FIG. 1 is a hardware configuration diagram of a depth production support apparatus 1 according to an embodiment of the present invention. Note that the hardware configuration in FIG. 1 is an example, and various configurations can be adopted depending on applications and purposes.

  In the depth production support device 1, a control unit 11, a storage unit 12, a media input / output unit 13, a communication control unit 14, an input unit 15, a display unit 16, a peripheral device I / F unit 17, etc. are connected via a bus 18. Is done.

  The control unit 11 includes a CPU (Central Processing Unit), a ROM (Read Only Y Memory Y), a RAM (Random Access Memory Y), and the like.

  The CPU calls and executes a program stored in the storage unit 12, ROM, recording medium or the like to a work memory area on the RAM, drives and controls each device connected via the bus 18, and the depth production support device 1 The process to be described later is realized. The ROM is a non-volatile memory, and permanently holds a program such as a boot program and BIOS for the depth production support apparatus 1, data, and the like. The RAM is a volatile memory, and temporarily stores programs, data, and the like loaded from the storage unit 12, ROM, recording medium, and the like, and includes a work area used by the control unit 11 for performing various processes.

  The storage unit 12 is an HDD (hard disk drive) or the like, and stores a program executed by the control unit 11, data necessary for program execution, an OS (operating system), and the like. As for the program, a control program corresponding to an OS (operating system) and an application program for causing the depth production support apparatus 1 to execute processing described later are stored. Each of these program codes is read by the control unit 11 as necessary, transferred to the RAM, read by the CPU, and executed as various means.

  The media input / output unit 13 (drive device) inputs / outputs data, for example, media such as a CD drive (-ROM, -R, -RW, etc.), DVD drive (-ROM, -R, -RW, etc.) Has input / output devices.

  The communication control unit 14 includes a communication control device, a communication port, and the like, and is a communication interface that mediates communication between the depth production support device 1 and the network, and performs communication control between other devices via the network. . The network may be wired or wireless.

  The input unit 15 inputs data and includes, for example, a keyboard, a pointing device such as a mouse, and an input device such as a numeric keypad. An operation instruction, an operation instruction, data input, and the like can be performed on the depth production support apparatus 1 via the input unit 15.

  The display unit 16 includes a display device such as a CRT monitor and a liquid crystal panel, and a logic circuit (such as a video adapter) for realizing the video function of the depth production support device 1 in cooperation with the display device.

  The peripheral device I / F (interface) unit 17 is a port for connecting a peripheral device to the depth production support device 1, and the depth production support device 1 communicates data with the peripheral device via the peripheral device I / F unit 17. Send and receive. The peripheral device I / F unit 17 is configured by USB, IEEE 1394, RS-235C, or the like, and usually includes a plurality of peripheral devices I / F. The connection form with the peripheral device may be wired or wireless.

  The bus 18 is a path that mediates transmission / reception of control signals, data signals, and the like between the devices.

  FIG. 2 is a diagram illustrating input / output data of the depth production support device 1 and the parallax image generation device 2. As shown in FIG. 2, the depth production support apparatus 1 uses the two-dimensional image 3, the background image 4, the mask image 5, and the depth value 6 as input data, and the depth image 7 and the label image 8 as output data. The parallax image generation device 2 uses the two-dimensional image 3, the depth image 7, and the label image 8 as input data, and uses the parallax image 9 as output data.

  The two-dimensional image 3, the background image 4, and the mask image 5 may be generated by the control unit 11 of the depth production support apparatus 1, or acquired from the outside via the media input / output unit 13, the communication control unit 14, and the like. May be. In addition, the depth value 6 may be input via the input unit 15 of the depth production support apparatus 1 or a file in which the depth value 6 is defined in advance may be acquired from the outside.

  The two-dimensional image 3 is a frame image or a still image for one moving image (planar image). A moving image (planar image) includes, for example, a color still image of a predetermined number of frames per unit time (usually 30 frames per second), and each pixel has 256 gradations of RGB.

  The background image 4 is an image showing a sense of depth in the background. The background image 4 may be generated manually using photo retouching software or the like based on the two-dimensional image 3, or may be automatically generated using separate software.

  The mask image 5 is a binarized image for identifying a specific subject. The mask image 5 may be generated manually using photo retouching software or the like based on the two-dimensional image 3, or may be automatically generated using separate software.

  The depth value 6 is a value that defines the position of the subject in the depth direction. In the present invention, a range of pixel values having a depth value of 6 is expressed as 256 gradations from 0 to 255. 0 corresponds to black, 255 corresponds to white, and 1 to 254 correspond to gray (halftone).

  The depth production support device 1 supports the production of the depth image 7 based on the two-dimensional image 3, the background image 4, the mask image 5, and the depth value 6. In particular, the depth value 6 is set, This is characterized by a screen interface (FIGS. 11 to 16 described later) for confirming the result.

  The depth image 7 is an image in which the depth value from the camera to the subject related to the two-dimensional image 3 is replaced with a pixel value. Here, the camera means a virtual camera defined in the internal processing of the parallax image generation device 2.

  The depth image 7 is, for example, a gray scale with a pixel value range of 0 to 255 (8 bits), and 0 (black) is the farthest and 255 (white) is often the foreground. There may be. In the following, in order to make the explanation easy to understand, the depth image 7 has a pixel value range of 0 to 255, 0 (black) being the deepest and 255 (white) being the foremost.

  The label image 8 is divided into regions based on unique numerical values (label values) for identifying subjects constituting the scene recorded in the two-dimensional image 3 in association with the two-dimensional images 3, and the label values are converted into pixel values. This is an image replaced (labeled) with. For example, in a scene composed of a background, a building, and a person, L0, L1, and L2 are assigned as label values corresponding to the background, the building, and the person, respectively, and each pixel of the corresponding two-dimensional image 3 is assigned to which subject. Whether it belongs or not is expressed by replacing it with pixel values of L0, L1, and L2.

  The label image 8 may be generated manually using photo retouching software or the like based on the two-dimensional image 3, or may be automatically generated by the depth production support apparatus 1 as in the present invention.

  The parallax image 9 is an image for realizing a stereoscopic view. In the embodiment of the present invention, the parallax image 9 is an image for realizing stereoscopic vision by naked-eye observation, particularly using binocular parallax (about 60 mm to 70 mm).

  Although not described in detail in the embodiment of the present invention, a plurality of parallax images 9 are synthesized in accordance with the specifications of the display and displayed on the display, thereby realizing stereoscopic vision by naked eye observation. . A known technique may be used for the synthesis process of the plurality of parallax images 9.

  The parallax image generation device 2 generates a parallax image 9 based on the two-dimensional image 3, the depth image 7, and the label image 8. Details of the parallax image generating device 2 are described in Japanese Patent Application Nos. 2010-176864 and 2011-55878, which were previously filed by the present applicant.

  Next, display examples of input / output data of the depth production support apparatus 1 will be described with reference to FIGS.

  FIG. 3 shows an example of the two-dimensional image 3. The example of FIG. 3 shows one frame of a moving image in which a plurality of people are playing a ball game called a spall boule. FIG. 4 shows an example of the background image 4 generated based on the two-dimensional image 3 shown in FIG. 3 and 4 are the same frame. Note that the depth of the background image 4 need not be exact. The background image 4 may not be present.

  FIG. 5 shows an example of the mask image 5 having the same frame as the two-dimensional image 3 shown in FIG. In the example of FIG. 5, the image is a binarized image for identifying a person playing a ball game participant (a specific subject), and the pixel of the person being played is white and the other pixels are black. ing. Hereinafter, the mask image 5 of FIG. 5 is appropriately referred to as a mask image of the mask A.

  FIG. 6 shows an example of another mask image 5. In the example of FIG. 6, it is a binarized image for identifying two persons (a specific subject) who are watching a battle a little apart, and the pixels of the two persons are white, The other pixels are black. 6 is a different frame (different time) from FIGS. 3 to 5 and FIG. Hereinafter, the mask image 5 of FIG. 6 is appropriately referred to as a mask image of the mask B.

  FIG. 7 shows an example of another mask image 5. In the example of FIG. 7, it is a binarized image for identifying the opponent person (specific subject), and the opponent person's pixel is white and the other pixels are black. FIG. 7 is a different frame from FIGS. Hereinafter, the mask image 5 of FIG. 7 is appropriately referred to as a mask image of the mask C.

  In the two-dimensional image 3 shown in FIG. 3, one of the subjects of the mask B (two people being seen) is hidden behind the subject of the mask A (the person being played) and is not shown. . The subject of the mask C is located on the right side of the subject of the mask B and is not reflected in the two-dimensional image 3 shown in FIG.

  As described above, the depth production support apparatus 1 prepares the mask image of the mask A (FIG. 5), the mask image of the mask B (FIG. 6), and the mask image of the mask C (FIG. 7) in advance for each frame. A depth image 7 is produced for each frame based on the mask image 5 and the depth value 6.

  FIG. 8 shows an example of the depth image 7. The depth image 7 is an image in which a relative positional relationship between a plurality of subjects in the depth direction is defined in a simulated manner. FIG. 8 is a frame different from those shown in FIGS. 3 to 7 and is a frame immediately before the person who is playing throws the ball. In the example of FIG. 8, the subject of the mask A (the person who is playing) is in the foreground (white region), and the subject of the mask B (the two people who are watching) is in the back (gray region). The positional relationship in the depth direction is defined. Further, depth information of the background image 4 is also added.

  FIG. 9 is a diagram for explaining the association of input / output images handled by the depth production support apparatus 1.

  As shown in FIG. 9, the input image group includes a set of five images of a two-dimensional image, a mask image of mask A, a mask image of mask B, a mask image of mask C, and a background image for each frame. It has become. The output image group is a set of two images, a depth image and a label image, for each frame. For example, when the number of frames is 1000, 1000 two-dimensional images, 3000 mask images, 1000 background images, 1000 depth images, and 1000 label images are output.

  FIG. 10 is a flowchart for explaining the depth production support process executed by the depth production support apparatus 1. In the description of FIG. 10, specific operation contents will also be described with reference to FIGS. 11 to 17.

  First, when starting the depth production support process, the user displays a depth production support screen 20 as shown in FIG. In the case of the example of FIG. 11, the depth production support screen 20 includes a “mask read”, “movie read”, and “background read” button group 21, a mask range detection check button 22, a start frame display area 23, and an end frame display. An area 24, a mask selection pull-down 25, a depth setting screen 26, a depth image display area 27, a two-dimensional image display area 28, and an export button 29 are displayed.

  The button group 21 is operated (pressed) when performing mask reading, movie reading, and background reading. The mask range detection check button 22 is operated when detecting a frame range where the subject of the selected mask exists. In the start frame display area 23, a mask of the start frame in the detection range is displayed, and in the end frame display area 24, a mask of the end frame in the detection range is displayed. The mask selection pull-down 25 is operated when selecting a mask for depth setting. The depth setting screen 26 is a setting screen for changing the depth value of the mask selected by the mask selection pull-down 25.

  In the depth image display area 27, a preview of a depth image in which the positional relationship in the depth direction is defined in a simulated manner based on a plurality of read masks is displayed. In the two-dimensional image display area 28, a preview of the read two-dimensional image is displayed. The write button 29 is selected when saving a depth image.

  FIG. 12 is a display example in which the depth setting screen 26 is enlarged.

  As shown in FIG. 12, the vertical axis of the depth setting screen 26 represents the depth value (depth value), and the horizontal axis represents Time (frame). In the depth setting screen 26, depth value curves 31a, 31b, and 31c are curves indicating the depth values of the mask A, the mask B, and the mask C, respectively, and three are displayed in an overlapping manner on the initial screen. The start position 32 indicates the position of the start frame, and the end position 33 indicates the position of the end frame.

  The frame slider bar 34 is slid (moved) in the horizontal direction when a frame for setting a depth value is designated. A number 35 indicates a frame number designated by the frame slider bar 34. The frame position 36 indicates the position in the frame direction specified by the frame slider bar 34. On the initial screen, the start position 32 and the frame position 36 are displayed so as to overlap each other. The depth value slider bar 37 is slid in the vertical direction when a depth value is designated. The key frame deletion button 38 is selected when deleting a key frame of a currently selected mask (a frame in which a change is defined).

  On the depth production support screen 20 as described above, the user presses the button group 21 and instructs to read the input image group for producing the depth image.

  In step S1, the control unit 11 of the depth production support apparatus 1 reads an input image group based on an instruction from the user. For example, as shown in FIG. 9, for each frame, a set of five images, that is, a two-dimensional image, a mask image of mask A, a mask image of mask B, a mask image of mask C, and a background image are combined. The input image group is read. Further, when the mask range detection check button 22 is pressed by the user, a frame range that requires setting of a depth value is detected.

  Thereby, on the depth production support screen 20, the mask image of the start frame of the mask selected by the mask selection pull-down 25 is displayed in the start frame display area 23, and the mask selection pull-down 25 is displayed in the end frame display area 24. The mask image of the end frame of the mask selected in (1) is displayed, and a preview of the two-dimensional image is displayed in the two-dimensional image display area 28. The depth setting screen 26 displays depth value curves of a plurality of read masks.

  In step S2, the control unit 11 of the depth production support apparatus 1 determines whether or not the user has instructed input of a mask selection pull-down, that is, whether or not a new mask has been selected from the mask selection pull-down 25 of FIG. If it is determined that there is an instruction to input a mask selection pull-down, the process proceeds to step S3. In step S <b> 3, the control unit 11 of the depth production support apparatus 1 changes the depth setting mask based on an input instruction from the user.

  The control unit 11 of the depth production support apparatus 1 proceeds to step S4 when it is determined in step S2 that there is no instruction to input a mask selection pull-down, or after changing the depth setting mask in step S3.

  In step S4, the control unit 11 of the depth production support apparatus 1 determines whether or not the user has instructed to change the frame slider bar, that is, whether or not the frame slider bar 34 of FIG. If it is determined that there has been a change instruction, the process proceeds to step S5.

  For example, the user can change the depth setting frame by sliding the frame slider bar 34 to the right in the initial screen of the depth setting screen 26 shown in FIG.

  FIG. 13 shows a state where a single mask (for example, mask A) is selected by the mask selection pull-down 25 from the state of the depth setting screen 26 shown in FIG. 12, and the frame slider bar 34 is slid. When the mask A is selected by the mask selection pull-down 25, the start frame and the end frame are automatically set as the key frames 41a and 42a. The frame position 36 is also slid in conjunction with the slide of the frame slider bar 34. At this time, in conjunction with the change operation of the depth setting frame, a preview of depth images of a plurality of frames is continuously displayed in the depth image display area 27, and a two-dimensional image of the plurality of frames is displayed in the two-dimensional image display area 28. The preview is displayed continuously.

  As described above, the user can change the depth setting frame with a simple operation. Further, since the depth images of a plurality of frames can be continuously displayed, the user can easily confirm whether or not there is a sense of incongruity between the frames. Furthermore, since a plurality of frames can be continuously displayed by linking the depth image and the original image, the finish of the depth image can be easily confirmed.

  Returning to the description of FIG. In step S5, the control unit 11 of the depth production support apparatus 1 changes the depth setting frame based on a change instruction from the user.

  If it is determined in step S4 that there is no frame slider bar change instruction, or the depth setting frame is changed in step S5, the control unit 11 of the depth production support apparatus 1 proceeds to step S6.

  In step S6, the control unit 11 of the depth production support apparatus 1 determines whether or not the user has instructed to change the depth value slider bar, that is, whether or not the depth value slider bar 37 in FIG. If it is determined that there is an instruction to change the value slider bar, the process proceeds to step S7.

  For example, the user can change the depth value by sliding the depth value slider bar 37 upward on the depth setting screen 26 shown in FIG.

  FIG. 14 shows a state in which the depth value slider bar 37 has been slid from the state of the depth setting screen 26 shown in FIG. The depth value curve 31 a is deformed in conjunction with the slide of the depth value slider bar 37. Specifically, the intersection (moving point) of the frame position 34 designated by the frame slider bar 34 and the position in the depth value direction designated by the depth value slider bar 37 is slid up and down, and the position of the intersection is changed. When confirmed (for example, when the left mouse button is released, which is an operation for canceling the selection of the depth value slider bar 37), the key frame 43a is set at the intersection. At this time, the preview of the depth image displayed in the depth image display area 27 is continuously changed while changing the pixel value indicating the depth value in conjunction with the change operation of the depth value.

  As described above, the user can change the depth value with a simple operation. In addition, since it is possible to continuously display how the depth image changes when the depth value is changed, the user can easily confirm whether there is no sense of incongruity between subjects in the same frame. be able to.

  Returning to the description of FIG. In step S7, the control unit 11 of the depth production support apparatus 1 sets the depth value based on a change instruction from the user. In step S8, the control unit 11 of the depth production support apparatus 1 performs depth value interpolation based on the depth value set in step S7. Specifically, the interpolation is performed between the key frame 43a shown in FIG. 14 and the key frame 41a and the key frame 42a adjacent to the key frame 43a. In the example of FIG. 14, linear interpolation is applied, but the present invention is not limited to this. For example, polynomial interpolation, spline interpolation, Bezier curve interpolation, or the like may be used.

  In this way, if depth values are set for some frames, other frames are automatically interpolated within a range that does not cause a sense of incongruity. Will be significantly better.

  In step S9, the control unit 11 of the depth production support apparatus 1 creates a depth image in which the positional relationship in the depth direction is defined in a simulated manner according to the depth value interpolated in the process of step S8, and the depth image display area 27 is displayed.

  If it is determined in step S6 that there is no instruction to change the depth value slider bar, or the depth image is created and displayed in step S9, the control unit 11 of the depth production support apparatus 1 proceeds to step S10. move on.

  In step S10, the control unit 11 of the depth production support apparatus 1 determines whether or not there has been an instruction to press the write button, that is, whether or not the write button 29 in FIG. 11 has been pressed. If it is determined, the process returns to step S2, and the above-described processing is repeatedly executed.

  For example, the user can change the depth setting frame again by further sliding the frame slider bar 34 to the right on the depth setting screen 26 shown in FIG.

  FIG. 15 shows a state in which a key frame is newly created by sliding the frame slider bar 34 and sliding the depth value slider bar 37 from the state of the depth setting screen 26 shown in FIG. In conjunction with the slide of the frame slider bar 34, the frame position 36 is also slid. Also, a new key frame is created or the key frame is moved in conjunction with the slide of the depth value slider bar 37.

  Specifically, when the depth value slider bar 37 is moved, if there is no key frame in the frame, a new key frame is created, and if there is a key frame, the key frame moves. At this time, in conjunction with the change operation of the depth setting frame, a preview of depth images of a plurality of frames is continuously displayed in the depth image display area 27 and a two-dimensional image of the plurality of frames is displayed in the two-dimensional image display area 28. The preview is displayed continuously.

  As described above, since the user can change the depth setting frame and the depth value many times with a simple operation, a plurality of key frames can be set for one mask. is there. In addition, since the depth image of a plurality of frames can be displayed continuously, or the plurality of frames can be displayed continuously by linking the depth image and the original video, the user does not feel uncomfortable between the frames. Whether or not the depth image is finished can be easily confirmed.

  Further, for example, in the state of the depth setting screen 26 shown in FIG. 15, the user selects another mask (for example, mask B) from the mask selection pull-down 25 and changes the depth setting frame and the depth value as described above. You can also.

  FIG. 16 shows a state in which the mask B is selected from the mask selection pull-down 25 and the depth value slider bar 37 is slid in the state of the depth setting screen 26 shown in FIG. When the mask B is selected by the mask selection pull-down 25, the start frame and the end frame are automatically set as the key frames 41b and 42. Then, in conjunction with the slide of the depth value slider bar 37, the key frame 41b is slid up and down, and the position of the key frame 41b is changed by determining the position. At this time, the preview of the depth image displayed in the depth image display area 27 is continuously changed while changing the pixel value indicating the depth value in conjunction with the change operation of the depth value.

  As described above, the user can change the depth setting frame and the depth value for another mask with a simple operation. Also, for other masks, how the depth image changes can be displayed continuously, multiple frames of depth images can be displayed continuously, and the depth image and the original video can be linked together. Since the frames can be displayed continuously, the user can easily check whether there is no sense of incongruity between subjects in the same frame, whether there is no sense of incongruity between frames, and the depth image finish. .

  Returning to the description of FIG. In step S10, when the control unit 11 of the depth production support apparatus 1 determines that there is an instruction to press the write button, the process proceeds to step S11, and the depth setting frame and the depth value are changed (interpolated) as described above. The depth image 7 is stored in the storage unit 12. In step S <b> 12, the control unit 11 of the depth production support apparatus 1 creates a label image 8 in association with the two-dimensional image 3 and stores it in the storage unit 12.

  FIG. 17 is a diagram for explaining the creation of the label image 8.

  For example, as shown in FIG. 17A, it is assumed that the mask image 5d is a mask D of a subject of a square object, and the mask image 5e is a mask E of a subject of a circle object. By setting the depth value as described above, as shown in FIG. 17B, the depth is set so that the subject 7d of the mask D is closest (white) and the subject 7e of the mask E is deepest (black). It is assumed that a value is set and a depth image 7 is generated.

  As shown in FIG. 17C, for the label D corresponding to the subject 7d in front, the same area as the mask image 5D is labeled as the labeling area 8d of the label D.

  As shown in FIG. 17C, for the label E corresponding to the subject 7e in the back, the label E of the label E is an area (a hatched area) obtained by removing the overlapping portion with the label D from the area of the mask image 5e. Labeled as region 8e.

  As described above, the label image 8 is labeled with the same area as the mask image 5 for the subject that does not overlap with the other mask image 5 or is in front of the other mask image 5 (see FIG. 17). Label D). Further, the label image 8 overlaps with the other mask image 5, and for the subject behind the other mask image 5, an area excluding the overlapped portion from the area of the mask image 5 is labeled (FIG. 17 Label E).

[Effects of the embodiment of the invention]
1. Since the depth value can be set while the depth values for a plurality of frames of the same subject are displayed in comparison with each other, it is possible to set a depth value that does not feel uncomfortable between frames. Thereby, the user can easily confirm whether there is no sense of incongruity between frames.

  2. Since the depth value can be set while the depth values for a plurality of subjects in the same frame are displayed in comparison with each other, it is possible to set a depth value that does not feel uncomfortable between subjects in the same frame. Thereby, the user can easily confirm whether there is no sense of incongruity between subjects in the same frame.

  3. If the depth value is set for some frames, the device automatically interpolates within the range where there is no sense of incongruity for other frames. Compared to the conventional method in which the depth values for all frames are set manually. , Work efficiency will be greatly improved.

  4). Since the depth image and the original image can be linked and a plurality of frames can be continuously displayed, the finish of the depth image can be easily confirmed.

  5). Only frames that require setting of the depth value can be displayed, increasing convenience.

  The preferred embodiments of the depth production support apparatus and the like according to the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to such examples. It will be apparent to those skilled in the art that various changes or modifications can be conceived within the scope of the technical idea disclosed in the present application, and these naturally belong to the technical scope of the present invention. Understood.

DESCRIPTION OF SYMBOLS 1 ......... Depth production support device 3 ......... 2D image 4 ......... Background image 5 ......... Mask image 6 ...... Depth value 7 ......... Depth image 8 ... …… Label image

Claims (8)

A depth production support apparatus for producing a depth image used for generating a parallax image for realizing stereoscopic viewing,
Reading means for reading a plurality of binarized mask images for identifying a specific subject for each frame of the original video;
Depth value curve display means for displaying a depth value curve indicating a depth value in each frame for each mask image;
In the depth value curve display means, setting means for setting the depth value of a single frame in the depth value curve of the selected single mask image;
Interpolating means for interpolating depth values before and after the frame for which depth values are set in the setting means;
A depth image creating means for creating the depth image based on an interpolation result by the interpolation means;
A depth production support device characterized by comprising: The setting means has a depth value designation component which is a screen component for designating a depth value in the depth value curve display means,
The depth according to claim 1, further comprising a depth image display unit that displays a preview of the depth image while changing a pixel value indicating the depth value in conjunction with the movement of the depth value designation component. Production support device. The setting means has a time designation component which is a screen component for designating time in the depth value curve display means,
The depth production support apparatus according to claim 2, wherein the depth image display means changes the frame of the depth image in conjunction with the movement of the time designation component. The depth production support device according to claim 3, further comprising video display means for displaying the original video while changing the time of the video in conjunction with the movement of the time designation component. The apparatus further comprises label image creation means for creating a label image that is an image in which a label value for identifying a subject constituting the original video is replaced with a pixel value based on an interpolation result by the interpolation means. Item 10. The depth production support device according to Item 1. 2. The depth according to claim 1, further comprising a mask range detecting unit that detects a range of a frame in which a subject is present in any one mask image and determines a start frame and an end frame of a depth value setting range. Production support device. A depth production support method for a depth production support apparatus for producing a depth image used for generating a parallax image for realizing stereoscopic vision,
A reading step of reading a plurality of mask images binarized to identify a specific subject for each frame of the original video;
A depth value curve display step for displaying a depth value curve indicating a depth value in each frame for each mask image;
In the depth value curve display step, a setting step of setting the depth value of a single frame in the depth value curve of the selected single mask image;
An interpolation step of interpolating the depth values before and after the frame in which the depth value is set in the setting step;
A depth image creating step for creating the depth image based on the interpolation result of the interpolation step;
A depth production support method characterized by comprising: On the computer,
A reading step of reading a plurality of mask images binarized to identify a specific subject for each frame of the original video;
A depth value curve display step for displaying a depth value curve indicating a depth value in each frame for each mask image;
In the depth value curve display step, a setting step of setting the depth value of a single frame in the depth value curve of the selected single mask image;
An interpolation step of interpolating the depth values before and after the frame in which the depth value is set in the setting step;
A depth image creating step for creating a depth image to be used for a process of generating a parallax image for realizing stereoscopic viewing based on the interpolation result of the interpolation step;
A program for running