JP5994320B2 - Depth production support apparatus, depth production support method, and program - Google Patents

Description

  The present invention relates to a depth production support apparatus that produces a depth image used by a parallax image generation apparatus that generates a parallax image for realizing stereoscopic vision.

  Conventional three-dimensional video display systems are roughly classified into a two-parallax system and a multi-parallax system having three or more parallaxes. In either method, it is necessary to prepare a video in which the scene is viewed from the required number of viewpoints.

  As the most direct video production method, there can be considered a method in which cameras for the required number of parallaxes are prepared and the same scene is shot with a plurality of cameras arranged at predetermined intervals. Recently, cameras that can directly shoot two parallaxes are commercially available, but cameras that can directly shoot multiple parallaxes are not common.

  A method of creating a three-dimensional CG (Computer Graphics) image and setting a plurality of camera positions for rendering is also conceivable. The three-dimensional CG image includes depth information, and a parallax image corresponding to the multi-parallax method can be created from the three-dimensional image. However, when the number of camera positions is large, there is a problem that the number of renderings increases, which takes time and cost. Furthermore, most videos are not CG images but are real photographs, and it is required to create a three-dimensional video from the real photographs.

  There are several methods (so-called two-dimensional / three-dimensional conversion methods) for converting a multi-parallax image based on a plane (2D) image to produce a multi-parallax image based on a real image. However, there is no depth information in the method of converting to a multi-parallax image. Therefore, it is necessary to create depth information one by one in every scene (frame), and the work load is very high.

  Thus, as a method for setting the depth value, for example, Patent Document 1 proposes a technique for changing the mesh shape in accordance with a user instruction and setting the depth value. Patent Document 2 proposes a technique for drawing a contour line in accordance with a user instruction and setting a depth value for the contour line. Non-Patent Document 1 proposes a technique in which a user moves and rotates a selected portion of a video in a three-dimensional space to set a depth value.

  However, the technique described in Patent Document 1 has a problem that there are many items to be operated, and it takes a long time to set the depth information for one frame, and it is difficult to confirm the set depth value. Further, the technique described in Patent Document 2 not only takes a long time to draw contour lines and set depth information, but also makes it difficult to draw contour lines when the subject has a complicated shape. was there. Further, although the technique described in Non-Patent Document 1 has a simplified setting method, there is a problem in that the positional relationship is likely to be inconsistent because depth values in different frames cannot be compared.

  In view of this, the present inventors disclosed in Patent Document 3 (1) a technique for setting a depth value using a graph and a slide bar, (2) a technique for checking a depth value using a graph and a preview screen, ( 3) A technique for interpolating depth values between key frames, and (4) a technique for generating a depth image and a label image based on the above-described technique are proposed.

Special table 2010-516155 gazette Special table 2011-523323 gazette Japanese Patent Application No. 2011-168920

Depth Director: A System for Adding Depth to Movies

  However, in the technique disclosed in Patent Document 3, the inside of the mask area of the depth image is painted with one color (single pixel value), and there remains a problem that the local protrusion of the mask area cannot be expressed. Yes. Here, a mask area | region is an area | region set for every solid objects, such as a person and an object, for example.

In the technique disclosed in Patent Document 1, there are many items to be operated, and it takes a long time to set the depth information for one frame, and there is a conventional problem that it is difficult to confirm the set depth value. . Moreover, the confirmation method of a setting result synthesize | combines and displays a mesh-shaped wire frame on an original image, and there exists a subject that it is difficult to confirm the shape of a solid object.
Similarly, in the technique disclosed in Patent Document 2, there are many items to be operated, and it takes a long time to set the depth information for one frame, and the contour lines are correctly set for a three-dimensional object having a complicated shape. There is a problem that the operation is difficult.
The technique disclosed in Non-Patent Document 1 has a conventional problem that the positional relationship is likely to be inconsistent because the depth values in different frames cannot be compared. In addition, there is a problem that the degree of freedom in expressing the roundness of the three-dimensional object is low. Furthermore, the confirmation method of the setting result is to confirm the three-dimensional image in the three-dimensional space while changing the camera angle, and there is a problem that it is difficult to confirm the details of the shape of the three-dimensional object.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a depth production support device that supports production of a depth image expressing local unevenness of a three-dimensional object with a simple operation. Etc. is to provide.

In order to achieve the above-described object, the first invention is a depth production support apparatus for producing a depth image used for generating a parallax image for realizing stereoscopic vision, and includes a depth production support device in the original two-dimensional image. A mask image input unit that reads a plurality of binarized mask images to identify a specific subject being drawn, a contour line setting unit that sets a contour line that is a closed curve with respect to the mask image, and A depth value input means for setting a plurality of points along the contour line, and inputting a desired depth value for the set point, and the inside of the contour line based on the depth value input by the depth value input means Depth value calculating means for calculating the depth value of each pixel of the mask image so that the area becomes a smooth plane, and the depth value calculated by the depth value calculating means are set for each pixel of the mask image. And a depth production support apparatus characterized by comprising: a depth image generating means for generating the depth image using the mask image depth value is set.
According to the first invention, it is possible to produce a depth image expressing local unevenness of a three-dimensional object by an extremely simple operation of setting depth values at least two points along a contour line of a specific subject. It becomes. In addition, the boundary with the background can be clearly distinguished, and the computation can be prevented from becoming complicated.

In the first invention, the branch line input means for additionally inputting a branch line branched from the contour line set by the contour line setting means, and the branch line is connected to the contour line already set. Contour depth resetting means for setting a new contour line, wherein the depth value input means inputs a desired depth value for a plurality of points along the new contour line, and the depth It is desirable that the value calculating means calculates the depth value of each pixel of the mask image so that the inner area of the new contour line becomes a smooth plane.
As a result, the boundary of the overlapping portion of the subject can be drawn with the contour line, and the depth of the overlapping portion of the subject can be expressed. Increased freedom of depth image production.

In the first aspect of the invention, the depth image created by the depth image creation means has a degree of proximity between each pixel of the depth image and the contour, a predetermined rounding range, and a predetermined rounding thickness. It is desirable to further include rounding means for performing rounding processing based on the rounding process.
As a result, a natural object can be further rounded inside the contour line with a simple operation, and the depth expression of the image can be varied.

In the first invention, it is preferable that the depth value input means inputs different depth values for at least two points along the contour line.
Thereby, the contour line itself can be inclined in the depth direction, and a free depth expression can be realized by a simple operation.

Further, in the first invention, an auxiliary line input means for additionally inputting an auxiliary line not in contact with the contour line in the inner region of the contour line is further provided, and the depth value input means is further provided along the auxiliary line. Desired depth values are input to a plurality of points, and the depth value calculation means is an internal region of the contour line, and pixels other than the boundary are smooth with the contour line and the auxiliary line as a boundary. It is desirable to calculate the depth value of each pixel of the mask image so as to be a plane.
Thereby, the depth expression inside the outline can be diversified.

A second invention is a depth production support method for producing a depth image to be used for a process of generating a parallax image for realizing stereoscopic vision, in which a computer has a specific image drawn in the original two-dimensional image. A mask image input step for reading a plurality of binarized mask images for identifying a subject, a contour line setting step for setting a contour line that is a closed curve with respect to the mask image, and a plurality along the contour line A depth value input step for inputting a desired depth value for the set point, and a plane in which the inner area of the contour line is smooth based on the depth value input by the depth value input step. A depth value calculating step of calculating a depth value of each pixel of the mask image, and the depth value calculated by the depth value calculating step of the mask image Set pixel, a depth production support method characterized by performing a depth image generation step of generating the depth image using the mask image depth value is set, the.
According to the second invention, it is possible to produce a depth image expressing local unevenness of a three-dimensional object by an extremely simple operation of setting depth values at least two points along a contour line of a specific subject. It becomes. In addition, the boundary with the background can be clearly distinguished, and the computation can be prevented from becoming complicated.

According to a third aspect of the present invention, a mask image input step of reading a plurality of binarized mask images to identify a specific subject drawn in the original two-dimensional image, A contour line setting step for setting a contour line that is a closed curve, a depth value input step for setting a plurality of points along the contour line, and inputting a desired depth value for the set point, and the depth value Based on the depth value input in the input step, the depth value calculation step for calculating the depth value of each pixel of the mask image so that the inner area of the contour line becomes a smooth plane, and the depth value calculation step A depth image creation step for creating the depth image using the mask image with the depth value set for each pixel of the mask image. Tsu is a program to be executed and up, the.
The depth production support apparatus according to the first invention can be obtained by installing the program according to the third invention in the computer. In addition, the boundary with the background can be clearly distinguished, and the computation can be prevented from becoming complicated.

  According to the present invention, it is possible to provide a depth production support device that supports producing a depth image expressing local unevenness of a three-dimensional object with a simple operation.

It is a hardware block diagram of the depth production assistance apparatus 1 which concerns on embodiment of this invention. It is a figure which shows the input-output data of the depth production assistance apparatus 1 and the parallax image generation apparatus 2. FIG. 3 is a diagram illustrating an example of a two-dimensional image 3. FIG. It is a figure which shows an example of the background image 4 produced | generated based on the two-dimensional image 3 shown in FIG. It is a figure which shows an example of the mask image. It is a figure which shows an example of the other mask image. It is a figure which shows an example of the other mask image. It is a figure which shows an example of the depth image. It is a figure for demonstrating matching of an input-output image. It is a flowchart explaining a depth production assistance process. It is a display example of the depth production support screen 100. It is an example of the two-dimensional image 31 made into the object of depth production assistance processing. It is an example of the mask image 51 created from the two-dimensional image 31. It is an example of the outline 61 set to the mask image 51. FIG. It is an example of setting anchor points 71a to 71d and depth values 72a to 72d set along the contour line 61. It is a depth value calculation processing example of the inner region of the contour line 61. It is an image figure of the depth of an image. It is another image figure of the depth expressed by the depth production support process. It is a figure explaining a rounding process. It is an example of the two-dimensional image 32 made into the object of the depth production assistance process of 2nd Embodiment. 3 is an example of a mask image 52 created from a two-dimensional image 32. FIG. It is an example of the outline 62 set to the mask image 52. FIG. It is an example of branch lines 63a and 63b added to the contour line 62. It is a figure for demonstrating the process which produces | generates the new mask image 54 from the mask image 52 of the original mask image 52 and branch line 63a, 63b. This is an example of setting the contour line 64, anchor points 74a to 74h, and depth values 75a to 75h newly set for the new mask image 54. FIG. These are various examples of the contour line 6.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

[First Embodiment]
FIG. 1 is a hardware configuration diagram of a depth production support apparatus 1 according to an embodiment of the present invention. Note that the hardware configuration in FIG. 1 is an example, and various configurations can be adopted depending on applications and purposes.

  In the depth production support apparatus 1, a control unit 11, a storage unit 12, a media input / output unit 13, a communication control unit 14, an input unit 15, a display unit 16, a peripheral device I / F unit 17, etc. are connected via a bus 18. Is done.

  The control unit 11 includes a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), and the like.

  The CPU calls and executes a program stored in the storage unit 12, ROM, recording medium or the like to a work memory area on the RAM, drives and controls each device connected via the bus 18, and the depth production support device 1 The process to be described later is realized. The ROM is a non-volatile memory, and permanently stores programs such as a boot program and BIOS for the depth production support apparatus 1, data, and the like. The RAM is a volatile memory, and temporarily stores programs, data, and the like loaded from the storage unit 12, ROM, recording medium, and the like, and includes a work area used by the control unit 11 for performing various processes.

  The storage unit 12 is an HDD (hard disk drive) or the like, and stores a program executed by the control unit 11, data necessary for program execution, an OS (operating system), and the like. As for the program, a control program corresponding to an OS (operating system) and an application program for causing the depth production support apparatus 1 to execute processing described later are stored. Each of these program codes is read by the control unit 11 as necessary, transferred to the RAM, read by the CPU, and executed as various means.

  The media input / output unit 13 (drive device) inputs / outputs data, for example, media such as a CD drive (-ROM, -R, -RW, etc.), DVD drive (-ROM, -R, -RW, etc.) Has input / output devices.

  The communication control unit 14 includes a communication control device, a communication port, and the like, is a communication interface that mediates communication between the depth production support device 1 and the network, and performs communication control between other devices via the network. . The network may be wired or wireless.

  The input unit 15 inputs data and includes, for example, a keyboard, a pointing device such as a mouse, and an input device such as a numeric keypad. An operation instruction, an operation instruction, data input, and the like can be performed on the depth production support apparatus 1 via the input unit 15.

  The display unit 16 includes a display device such as a CRT monitor and a liquid crystal panel, and a logic circuit (such as a video adapter) for realizing the video function of the depth production support device 1 in cooperation with the display device.

  The peripheral device I / F (interface) unit 17 is a port for connecting a peripheral device to the depth production support device 1, and the depth production support device 1 communicates data with the peripheral device via the peripheral device I / F unit 17. Send and receive. The peripheral device I / F unit 17 is configured by USB, IEEE 1394, RS-235C, or the like, and usually includes a plurality of peripheral devices I / F. The connection form with the peripheral device may be wired or wireless.

  The bus 18 is a path that mediates transmission / reception of control signals, data signals, and the like between the devices.

  FIG. 2 is a diagram illustrating input / output data of the depth production support apparatus 1 and the parallax image generation apparatus 2. As shown in FIG. 2, the depth production support apparatus 1 receives a two-dimensional image 3, a background image 4, a mask image 5, a contour line 6, and a depth value 7 as input data, and outputs a depth image 8 and a label image 9. Data. The parallax image generation device 2 uses the two-dimensional image 3, the depth image 8, and the label image 9 as input data, and uses the parallax image 10 as output data.

  The two-dimensional image 3, the background image 4, and the mask image 5 may be generated by the control unit 11 (hereinafter abbreviated as “control unit 11”) of the depth production support apparatus 1, or the media input / output unit 13. Alternatively, it may be acquired from the outside via the communication control unit 14 or the like. Further, the contour line 6 and the depth value 7 may be input via the input unit 15 of the depth production support apparatus 1 or a file in which the contour line 6 and the depth value 7 are defined in advance may be acquired from the outside. Good.

  The two-dimensional image 3 is a frame image or a still image for one moving image (planar image). A moving image (planar image) includes, for example, a color still image of a predetermined number of frames per unit time (usually 30 frames per second), and each pixel has 256 gradations of RGB.

  The background image 4 is an image showing a sense of depth in the background. The background image 4 may be generated manually using photo retouching software or the like based on the two-dimensional image 3, or may be automatically generated using separate software.

  The mask image 5 is a binarized image for identifying a specific subject. The mask image 5 may be generated manually using photo retouching software or the like based on the two-dimensional image 3, or may be automatically generated using separate software.

  The contour line 6 is a line indicating a binary boundary of the mask image 5. The contour line 6 may be generated by the control unit 11 performing a contour line detection process, or may be input by the user using a mouse or the like.

The depth value 7 is a value that defines the position of the subject in the depth direction. In the present invention, a range of pixel values having a depth value of 7 is expressed as 256 gradations from 0 to 255. 0 corresponds to black, 255 corresponds to white, and 1 to 254 correspond to gray (halftone).
When several points (hereinafter referred to as anchor points) are designated along the contour line 6 by the user's input operation and a depth value 7 is input for each anchor point, the contour line is controlled by the control unit 11. A depth value 7 of each pixel in the inner area 6 is calculated and set.

  The depth production support device 1 supports production of the depth image 8 based on the two-dimensional image 3, the background image 4, the mask image 5, the contour line 6, and the depth value 7. In particular, the depth production support apparatus 1 according to the embodiment of the present invention sets a depth value for the mask image 5 (FIG. 10) in order to create a depth image that represents a local protrusion (or depression) of a three-dimensional object. (S3-S6, FIG. 15, FIG. 17-18, etc.).

  The depth image 8 is an image in which the depth value from the camera to the subject related to the two-dimensional image 3 is replaced with a pixel value. Here, the camera means a virtual camera defined in the internal processing of the parallax image generation device 2.

  The depth image 8 is, for example, a gray scale with a pixel value range of 0 to 255 (8 bits), and 0 (black) is the farthest and 255 (white) is often the foreground. There may be. In the following, in order to make the explanation easy to understand, the depth image 8 has a pixel value range of 0 to 255, 0 (black) being the deepest and 255 (white) being the foremost.

  The label image 9 is associated with each two-dimensional image 3, and is divided into regions by unique numerical values (label values) for identifying subjects constituting the scene recorded in the two-dimensional image 3, and the label values are represented by pixel values. This is an image replaced (labeled) with. For example, in a scene composed of a background, a building, and a person, L0, L1, and L2 are assigned as label values corresponding to the background, the building, and the person, respectively, and each pixel of the corresponding two-dimensional image 3 is assigned to which subject. Whether it belongs or not is expressed by replacing it with pixel values of L0, L1, and L2.

  The label image 9 may be generated manually using photo retouching software or the like based on the two-dimensional image 3, or may be automatically generated by the depth production support apparatus 1 as in the present invention.

  The parallax image 10 is an image for realizing stereoscopic viewing. In the embodiment of the present invention, the parallax image 10 is an image for realizing stereoscopic vision by naked-eye observation, particularly using binocular parallax (about 60 mm to 70 mm).

  Although not described in detail in the embodiment of the present invention, a plurality of parallax images 10 are synthesized in accordance with the specifications of the display, and displayed on the display, it is possible to realize stereoscopic vision by naked eye observation. . A known technique may be used for the synthesis process of the plurality of parallax images 10.

  The parallax image generation device 2 generates a parallax image 10 based on the two-dimensional image 3, the depth image 8, and the label image 9. Details of the parallax image generation device 2 are described in Japanese Patent Application Nos. 2010-176864 and 2011-55878, which were previously filed by the present applicant.

  Next, display examples of input / output data of the depth production support apparatus 1 will be described with reference to FIGS.

  FIG. 3 shows an example of the two-dimensional image 3. The example of FIG. 3 shows one frame of a moving image in which a plurality of people are playing a ball game called a spall boule. FIG. 4 shows an example of the background image 4 generated based on the two-dimensional image 3 shown in FIG. 3 and 4 are the same frame. Note that the depth of the background image 4 need not be exact. The background image 4 may not be present.

  FIG. 5 shows an example of the mask image 5 having the same frame as the two-dimensional image 3 shown in FIG. In the example of FIG. 5, the image is a binarized image for identifying a person playing a ball game participant (a specific subject), and the pixel of the person being played is white and the other pixels are black. ing. Hereinafter, the mask image 5 of FIG. 5 is appropriately referred to as a mask image of the mask A.

  FIG. 6 shows an example of another mask image 5. In the example of FIG. 6, it is a binarized image for identifying two persons (a specific subject) who are watching a battle a little apart, and the pixels of the two persons are white, The other pixels are black. 6 is a different frame (different time) from FIGS. 3 to 5 and FIG. Hereinafter, the mask image 5 of FIG. 6 is appropriately referred to as a mask image of the mask B.

  FIG. 7 shows an example of another mask image 5. In the example of FIG. 7, it is a binarized image for identifying the opponent person (specific subject), and the opponent person's pixel is white and the other pixels are black. FIG. 7 is a different frame from FIGS. Hereinafter, the mask image 5 of FIG. 7 is appropriately referred to as a mask image of the mask C.

  In the two-dimensional image 3 shown in FIG. 3, one of the subjects of the mask B (two people being seen) is hidden behind the subject of the mask A (the person being played) and is not shown. . The subject of the mask C is located on the right side of the subject of the mask B and is not reflected in the two-dimensional image 3 shown in FIG.

  As described above, the depth production support apparatus 1 prepares the mask image of the mask A (FIG. 5), the mask image of the mask B (FIG. 6), and the mask image of the mask C (FIG. 7) for each frame. A depth image 8 is produced for each frame based on the mask image 5 and the depth value 6.

  FIG. 8 shows an example of the depth image 8. However, the depth image 8 shown in FIG. 8 is an example in which the depth setting inside the object is not performed. The depth image 8 is an image in which the relative positional relationship between a plurality of subjects in the depth direction is defined in a simulated manner. FIG. 8 is a frame different from those shown in FIGS. 3 to 7 and is a frame immediately before the person playing is throwing the ball. In the example of FIG. 8, the subject of the mask A (the person who is playing) is in the foreground (white region), and the subject of the mask B (the two people who are watching) is in the back (gray region). The positional relationship in the depth direction is defined. Further, depth information of the background image 4 is also added.

  In the depth image 8 shown in FIG. 8, the inside of the subject of the mask A and the subject of the mask B (= the inside of the mask area) is painted with one color (single pixel value), and a stereoscopic effect (locally) Image with no irregularities or roundness). Therefore, in the embodiment of the present invention, the depth production support apparatus 1 performs depth setting inside the mask area (S3 to S8 in FIG. 10), and produces a depth image 8 having a stereoscopic effect inside the three-dimensional object.

  FIG. 9 is a diagram for explaining the association of input / output images handled by the depth production support apparatus 1.

  As shown in FIG. 9, the input image group includes a set of five images of a two-dimensional image, a mask image of mask A, a mask image of mask B, a mask image of mask C, and a background image for each frame. It has become. The output image group is a set of two images, a depth image and a label image, for each frame. For example, when the number of frames is 1000, 1000 two-dimensional images, 3000 mask images, 1000 background images, 1000 depth images, and 1000 label images are output.

  FIG. 10 is a flowchart for explaining the depth production support process executed by the depth production support apparatus 1. FIG. 11 is an example of the depth production support screen 100 provided by the depth production support device 1 in the depth production processing. FIGS. 12 to 19 illustrate the processing of each step of setting the image to be processed and the depth. FIG.

  First, when starting the depth production support process, the user causes the display unit 16 to display a depth production support screen 100 as shown in FIG. In the case of the example in FIG. 11, the depth production support screen 100 includes an image display area 101, a group of buttons such as a “load movie” button 102, a “read mask” button 103, and a “read background depth image” button 104, mask switching. A pull-down 105, an image switching pull-down 106, an outline setting unit 107, a depth (depth value) setting unit 111, a rounding setting unit 116, and a save button 120 are displayed. In the following description, the word “depth” may be replaced with “depth”.

  A “movie read” button 102, a “mask read” button 103, and a “read background depth image” button 104 read a movie (two-dimensional image 3), read a mask image 5, and read a background image 4, respectively. Is operated (pressed). The mask switching pull-down 105 is operated when the mask image 5 to be set for depth is selected. The image switching pull-down 106 is operated when switching the frame for setting the depth. In the image display area 101, a mask image 5 that is a target of the depth setting operation, a depth image in which the depth is set, and a GUI (Graphical User Interface) such as a mouse pointer necessary for the operation are displayed.

  The contour setting unit 107 includes a contour addition button 108, a contour deletion button 109, and a contour extraction button 110, and is operated when the contour line 6 is input to or deleted from the mask image 5. The contour addition button 108 is operated when the contour line 6 is input, the contour deletion button 109 is operated when the contour line 6 already drawn is deleted, and the contour extraction button 110 is the mask image 5 to be processed. It is operated when extracting the outline of the.

  The depth setting unit 111 includes a depth value input field 112, a setting button 112a, an anchor point deletion button 113, a grouping button 114, and a depth calculation button 115, and inputs a depth value for an anchor point that is a point for setting a depth. To be operated. The depth value input column 112 is an input column for inputting a depth value to an anchor point selected with a mouse or the like. The setting button 112 a is operated when setting the depth value input to the depth value input field 112 in the mask image 5. The anchor point deletion button 113 is operated when deleting an already set anchor point. The grouping button 114 is operated when grouping a plurality of anchor points (with the same depth value). The depth calculation button 115 is operated when starting the depth value calculation process.

Here, the setting operation of the anchor point and the depth value will be described.
The anchor points are set on the contour line 6 drawn on the mask image 5 and on the pixels adjacent to the contour line 6 and outside the contour line. In the following description, pixels on the contour line 6 are referred to as “inner pixels” or “inner side” of the contour line 6, and pixels adjacent to the outer side of the contour line 6 are referred to as “outer pixel” or “ Called “outside”. The “inner pixel” and the “outer pixel” are collectively referred to as “position along the contour line”.

  When the mouse pointer 53 is moved to a position along the contour line 6 on the mask image 5 by a mouse operation by the user, the control unit 11 accepts installation of an anchor point. In this state, for example, when a mouse button is clicked, an anchor point is set at that position. For example, a mark indicating an anchor point is drawn on the mask image 5. Furthermore, when an operation such as a click is performed in a state where the mouse pointer 53 is moved onto a specific anchor point by a mouse operation by the user, the anchor point is selected. When a depth value is entered in the depth value input field 112 with the anchor point selected, and the setting button 112a is pressed, the depth value is set at the position of the selected anchor point. The control unit 11 sets the set depth value for the corresponding pixel of the mask image 5. Further, the control unit 11 displays the depth value on the mark indicating the anchor point on the mask image 5. Alternatively, when the mouse is moved over the anchor point, information on the anchor point may be displayed in a pop-up form in the form of (x coordinate, y coordinate, depth value), for example.

The rounding setting unit 116 includes a skeleton use check box 117 and sliders 118 and 119. The skeleton use check box 117 is selected when rounding processing is performed using a skeleton, and the slider 118 is slid (moved) in the horizontal direction when the rounding range is designated, and the slider 119 has a rounding thickness. When specifying, it is slid (moved) horizontally.
The save button 120 is operated to save the setting contents of the anchor point and depth value for the mask image 5 and the created depth image.

Returning to the description of FIG.
In the depth production support screen 100 as described above, the user presses the movie reading button 102, the mask reading button 103, the background depth image reading button 104, and the like to instruct to read the input image group for producing the depth image. To do.

  In step S1, the control unit 11 reads an input image group based on an instruction from the user. When a movie (image of a plurality of frames) is read, the user selects a frame for setting and editing the depth value by pressing the image switching pull-down 106. When a plurality of mask images 5 are read for the selected frame, the mask image pull-down 105 is pressed by the user to select the mask image 5 for setting and editing the depth value. (Step S2). The control unit 11 displays the selected mask image 5 in the image display area 101. For example, it is assumed that a two-dimensional image 31 shown in FIG. 12, a mask image 51 shown in FIG. 13, a background image (not shown), and the like are read as an input image group.

When the user presses the contour extraction button 110 of the contour setting unit 107 (step S3), the control unit 11 detects the contour of the mask image 51 and draws the contour line 61 on the mask image 51 (step S4). Note that the outline 61 may be detected by using a general method of recognizing an edge (boundary) based on a change in pixel value, for example. The contour line 61 may be input by the user with a mouse or the like, or the shape of the locus may be changed.
FIG. 14 is an example of the contour line 61 set for the mask image 51. As shown in FIG. 14, the outline 61 of the mask image 51 is a closed curve drawn with one stroke. By adopting the closed curve, the calculation of the depth value (step S7) in the deformation simulation to be described later is not complicated, and the calculation processing load can be reduced.

Next, an anchor point is set at a position along the contour line 61 by the depth setting operation described above (step S5). When a depth value is input (step S6), the control unit 11 masks a mark indicating the anchor point. It is displayed on the image 51. Further, the depth value may be displayed on a mark indicating each anchor point.
In the example of FIG. 15, three anchor points 71a, 71b, 71c are installed inside the contour line 61, and one anchor point 71d is set outside. The depth value of the anchor point 71a is set to “150”, the depth values of the anchor points 71b and 71c are set to “100”, and the depth value of the anchor point 71d is set to “0”.
Thus, if different depth values are set for the inside and the outside of the contour line 61, the object can be given a depth with the contour line 61 as a boundary. In addition, you may make it give a uniform depth value about the depth value outside a contour line.

  Next, the control unit 11 performs a deformation simulation process (step S7). In the deformation simulation process, the control unit 11 first obtains the depth value of each point (an anchor point non-set point) along the contour line 61 by interpolation based on the set position and depth value of the anchor point. In the interpolation process, the control unit 11 obtains a depth value so that each point on the contour line 61 draws a smooth slope in the depth direction.

When the depth value of each point on the contour line 61 is calculated, the control unit 11 further calculates the depth value of each pixel inside the contour line 61 (each pixel excluding the contour line 61). The depth value of each pixel is obtained so that the inner region of the outline 61 draws a smooth plane.
There are various solutions for obtaining all the three-dimensional coordinates of the inner region based on the three-dimensional coordinates of the boundary (contour line 61). For example, a thin plate as shown in FIG. The solution of the bending problem can be used. In the thin plate bending problem, a smooth plane can be obtained by calculating the three-dimensional position of each point so that the strain energy is minimized. FIG. 16B is a diagram for explaining the solution by the iterative method. In the iterative method, pixels (points) excluding the boundary (outline 61) are sequentially scanned, and the depth value of each pixel is calculated using the following equation (1). This is repeated until the whole converges.

When the depth value of each point is obtained, the control unit 11 creates a depth image 8 having the calculated depth value as a pixel value and displays it in the image display area 101.
FIG. 17 and FIG. 18 are diagrams showing an image of deformation by an anchor point (depth value) set along the contour line 61. FIG. 17 is an image diagram showing a state in which each point on the contour line 61 has the same depth value. FIG. 18 further shows a part of the contour line 61 (the arm part of the subject) than other parts. It is the image figure which gave the deformation | transformation which the arm protrudes ahead by setting a big depth value.
FIGS. 17 and 18 are diagrams showing an image of deformation in the depth direction, and are not the depth image 8 created in the present embodiment or the parallax image itself created using the depth image 8. The generated depth image 8 does not include a shadow, an outline, an image inside the outline, or the like as shown in FIGS. 17 and 18, and is nothing but a grayscale image in which a depth value is set for each pixel.

  After the depth value is calculated by setting the anchor point (depth value) and the deformation simulation process, and the depth image 8 is created, the control unit 11 next performs a rounding process (step S8). That is, when the user operates the rounding setting unit 116 of the depth production support screen 100 and sets the rounding range and thickness by the sliders 118 and 119, the rounding range and rounding thickness are set based on the set rounding range and rounding thickness. The depth value of the depth image 8 is further recalculated. FIG. 19 is a diagram showing an outline of the rounding process. As shown in FIG. 19A, when the rounding range (R) is set to “3 pixels” and the rounding thickness (T) is set to “10” as the rounding parameter 76 by the user, the control unit 11 is a cross-sectional view 77 of FIG. 19B and a pixel value map 78 of FIG. 19C, so that the depth value smoothly changes by “10” at the three inner pixels of the contour line 61. The depth value of the corresponding pixel is obtained.

  When the skeleton use check box 117 of the rounding setting unit 116 on the depth production support screen 100 is ON, the control unit 11 performs rounding with reference to the skeleton. The rounding process with reference to the skeleton is a rounding process with reference to the degree of approach to the skeleton in addition to the range and thickness of the rounding. Details of the rounding process are described in Japanese Patent Application No. 2011-223906 filed earlier by the present applicant.

When the rounding process ends, the control unit 11 performs a depth setting process for another mask image.
That is, when there is a mask image (unset mask) for which the depth value has not been set (step S9; Yes), the process returns to step S2. In step S2, when the mask switching pull-down 105 is operated to select another mask, depth setting processing (step S3 to step S8) is performed for the selected mask.

  When the depth setting is completed for all the masks (step S9; No), the generated mask image 5 is combined (step S10), and the depth image and the label image are stored (step S11), and depth production support is performed. The process ends.

As described above, the depth production support apparatus 1 extracts and draws the outline 6 of the input mask image 5 and sets anchor points and depth values 7 to several points along the outline 6. Accept. The depth production support apparatus 1 interpolates the depth value 7 of each point on the contour line 6 according to the input anchor point position and depth value 7, and further, the inner region of the contour line 6 is a smooth plane. As described above, the depth value of each pixel in the internal region is calculated, and the depth image 8 is created.
Thereby, the user can obtain the depth image 8 in which the internal region changes smoothly only by a simple operation of setting depth values at least at two points along the contour line 61 of the mask image 5.
Further, the rounding process can express a further rounded depth inside the contour line, and a natural three-dimensional effect can be realized with a simple operation.

[Second Embodiment]
Next, a second embodiment of the present invention will be described.
In the second embodiment, the depth production support apparatus 1 adds a branch line to the already set contour line, and performs the resetting of the contour line based on the branch line and the depth value setting process.
Hereinafter, the depth production support method according to the second embodiment will be described with reference to FIGS. 10, 11, and 20 to 26.

  In the second embodiment, an object is to express the overlap of objects within the contour of the mask image 5. For this purpose, the depth production support apparatus 1 accepts an additional input of the branch line 63 with respect to the already set contour line 6 as exemplified in the first embodiment, and the branch line 63 is converted into the original contour line 6. And a new one-stroke outline 64 is generated, and anchor points can be set along the new outline 64.

  For example, in step S1 and step S2 in FIG. 10, the two-dimensional image 32 shown in FIG. 20 is read, the mask image 52 shown in FIG. 21 is read, and the mask image 52 is selected as an editing target.

  Next, when the user presses the contour extraction button 110 of the contour setting unit 107 in step S3 in FIG. 10, the control unit 11 detects the contour of the mask image 52 in step S4, and the contour line is displayed on the mask image 52. 62 is drawn. FIG. 22 is an example of the contour line 62 set for the mask image 52.

In the state where the contour line 62 is drawn as shown in FIG. 22, branch lines 63a and 63b as shown in FIG. 23 are added by the drawing operation by the user, and the contour extraction button 110 on the depth production support screen 100 is pressed again. Then, the control part 11 produces | generates the mask image 52 of branch line 63a, 63b (center mask image of FIG. 24). Then, the mask image 53 of the branch lines 63a and 63b is subtracted from the original mask image 52, and a new mask image 54 is generated. In the new mask image 54, the overlapping of the arms of the subject is expressed.
When a new mask image 54 is generated, the control unit 11 extracts the contour of the new mask image 54, generates a new contour line 64, and draws it. FIG. 25 is an example of a new contour line 64 set for the mask image 52. The new contour line 64 is also a closed curve drawn with one stroke.

  Thereafter, in step S5 and step S6, when an anchor point is set at a position along the new contour line 64 and a depth value is input, the control unit 11 displays a mark indicating the anchor point on the mask image 54. . In the example of FIG. 25, eight anchor points 74 a to 74 h are installed inside the contour line 64. Depth values 75a to 75h are set for the anchor points 74a to 74h, respectively.

Next, in step S7, the control unit 11 performs a deformation simulation process. The deformation simulation process is the same as that in the first embodiment.
That is, the control unit 11 first obtains the depth value of each point (an anchor point unset point) along the contour line 64 by interpolation, and the control unit 11 further determines each pixel (the contour line 64 in the contour line 64). The depth value of each pixel) is calculated. The depth value of each pixel is calculated so that the inner area of the outline 64 draws a smooth plane.
Further, in step S8, a rounding process may be performed.

  Thus, when the setting of the depth value for the mask image 52 (new mask image 54) is completed, the depth setting for another mask (step S9; Yes → step S3 to step S8) is repeated. When the depth setting is completed for all the masks (step S9; No), the generated mask image 5 is synthesized (step S10), the depth image 8 and the label image 9 are stored, and the depth production support process is performed. The process ends (step S11).

  As described above, if a branch line is added to the contour line and an anchor point is set along the branch line (appears in the new contour line 64), more detailed depth setting can be made in the inner area of the original contour line. Can be performed.

FIG. 26 shows setting examples of the contour line 6 for various images.
26A shows a car, FIG. 26B shows an animal, FIG. 26C shows a person, and FIG. 26D shows a headphone. Each outline 6 in FIGS. 26A to 26D is drawn with one stroke. As described above, since the new contour line 64 is drawn only by the user drawing the branch line 63 on the contour line 6 extracted from the mask image 5, there is an overlap inside the contour line of the subject. Even in such an image, the depth value can be set with the overlap as a boundary (contour line).

  As described above, the depth production support apparatus 1 according to the second embodiment of the present invention can further add the branch line 63 to the contour line 62 of the subject, and when the branch line 63 is additionally input, A new one-stroke outline 64 is generated by connecting the two. For this reason, the boundary of the overlapping part of the subject can be drawn by the contour line 64. Further, if the anchor point and the depth value are set along the new contour line 64 in which the boundary of the overlapping portion of the subject is expressed, the inner region of the contour line 64 is smooth as in the first embodiment. Since the depth value of each pixel is calculated so as to be a plane, the front and back of the overlapping portion of the subject can be expressed with a simple operation.

[Third Embodiment]
Next, a third embodiment of the present invention will be described.
In the second embodiment, a description has been given of the depth image generation processing in the case where a line in contact with the contour line 62 can be additionally input as the branch line 63 and the branch line is added. An independent line (hereinafter referred to as an auxiliary line) that does not touch the line may be additionally inputable.
That is, for example, when the two-dimensional image 32 in FIG. 20 and the mask image 52 in FIG. 21 are targets for depth production, the control unit 11 of the depth production support apparatus 1 is the same as in the first and second embodiments. First, the outline 62 of the mask image 52 is extracted according to the user's operation. Thereafter, the control unit 11 accepts addition of an auxiliary line by the user. Here, the user draws independent auxiliary lines inside the contour line 62 in order to express the irregularities inside the contour line 62 such as nose, muscle, or clavicle. The contour line 62 is a closed curve, but the auxiliary line is not necessarily a closed curve. Further, the control unit 11 accepts input of anchor points and depth values along the contour line, and also accepts input of anchor points and depth values along the auxiliary line.
Based on the input depth value, the control unit 11 calculates the depth value of each pixel so that the pixels other than the boundary (the contour line 61 and the auxiliary line) become a smooth plane with the contour line and the auxiliary line as a boundary. To do.

  As described above, if an auxiliary line independent of the contour line is added to the inside of the contour line (closed curve) and an anchor point is set along the contour line and the auxiliary line, the inner area of the original contour line is further increased. Detailed depth setting can be performed. The addition and deletion of the auxiliary line can be further continued after the depth value of each pixel is calculated.

The preferred embodiments of the depth production support apparatus and the like according to the present invention have been described above with reference to the accompanying drawings, but the present invention is not limited to such examples. For example, by combining the second embodiment and the third embodiment, it is possible to appropriately add contour lines, branch lines, and auxiliary lines, or to set anchor points along these lines. May be. In addition, a contour line can be newly set inside the contour line. The contour line, branch line, auxiliary line, and the like of the present invention are different from the contour line in that an arbitrary depth is given to each point on the line, and an intuitive and free operation can be realized.
In addition, it is obvious that those skilled in the art can arrive at various changes or modifications within the scope of the technical idea disclosed in the present application, and these naturally belong to the technical scope of the present invention. It is understood.

DESCRIPTION OF SYMBOLS 1 ......... Depth production support apparatus 3 ......... 2D image 4 ......... Background image 5 ......... Mask image 6 ...... Outline 7 ......... Depth value 8 ......... Depth image 9 ......... Label image

Claims (7)

A depth production support device for producing a depth image used for generating a parallax image for realizing stereoscopic viewing,
Mask image input means for reading a plurality of binarized mask images for identifying a specific subject drawn in the original two-dimensional image;
Contour setting means for setting a contour that is a closed curve for the mask image;
A depth value input means for setting a plurality of points along the contour line and inputting a desired depth value for the set points;
A depth value calculating means for calculating the depth value of each pixel of the mask image based on the depth value input by the depth value input means so that the inner area of the contour line is a smooth plane;
Depth image calculation means for setting the depth value calculated by the depth value calculation means to each pixel of the mask image, and generating the depth image using the mask image in which the depth value is set;
A depth production support device characterized by comprising: A branch line input means for additionally inputting a branch line branched from the contour line set by the contour line setting means;
A contour line resetting means for connecting the branch line with the contour line already set and setting a new contour line; and
The depth value input means inputs a desired depth value for a plurality of points along the new contour line,
2. The depth production support apparatus according to claim 1, wherein the depth value calculation unit calculates a depth value of each pixel of the mask image so that an inner area of the new contour line becomes a smooth plane. . Rounding that performs rounding processing on the depth image created by the depth image creating means based on the degree of proximity between each pixel of the depth image and the contour line, a predetermined rounding range, and a predetermined rounding thickness Attachment means,
The depth production support apparatus according to claim 1, further comprising: The depth value input means, to at least two points along the contour line, depth production support device according to claim 1, characterized in that the input different depth values respectively to one of claims 3. Auxiliary line input means for additionally inputting an auxiliary line not in contact with the contour line in the inner region of the contour line,
The depth value input means further inputs a desired depth value for a plurality of points along the auxiliary line,
The depth value calculation means is an inner area of the contour line, and the depth value of each pixel of the mask image is such that pixels other than the boundary are smooth planes with the contour line and the auxiliary line as a boundary. The depth production support apparatus according to claim 1, wherein: A depth production support method for producing a depth image used for generating a parallax image for realizing stereoscopic viewing,
Computer
A mask image input step for reading a plurality of mask images binarized to identify a specific subject drawn in the original two-dimensional image;
An outline setting step for setting an outline that is a closed curve with respect to the mask image;
A depth value input step for setting a plurality of points along the contour line and inputting a desired depth value for the set points;
A depth value calculating step of calculating a depth value of each pixel of the mask image based on the depth value input by the depth value input step so that an inner region of the contour line is a smooth plane;
A depth image creation step of setting the depth value calculated in the depth value calculation step to each pixel of the mask image and creating the depth image using the mask image in which the depth value is set;
Depth production support method characterized by executing. On the computer,
A mask image input step for reading a plurality of mask images binarized to identify a specific subject drawn in the original two-dimensional image;
An outline setting step for setting an outline that is a closed curve with respect to the mask image;
A depth value input step for setting a plurality of points along the contour line and inputting a desired depth value for the set points;
A depth value calculating step of calculating a depth value of each pixel of the mask image based on the depth value input by the depth value input step so that an inner region of the contour line is a smooth plane;
A depth image creation step of setting the depth value calculated in the depth value calculation step to each pixel of the mask image and creating the depth image using the mask image in which the depth value is set;
A program for running