JP2015103960A - Method, program and apparatus for specifying image depth - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image depth specification technique for image processing, enabling depth of an image to be more easily specified.SOLUTION: A division degree specification part 101 causes a user to specify a division degree of an input image 120. An image division part 102 divides the input image into a plurality of divided regions by the division degree specified by the user. An image display part 103 displays the divided input image 120. A depth specification part 104 causes the user to specify depth while causing the user to select a divided region on the displayed input image 120. An inclination specification part 105 causes the user to specify a horizontal inclination and a vertical inclination of the displayed input image 120. An image processing part 106 adds the horizontal and vertical inclinations specified by the user and a depth effect corresponding to the depth of the divided region unit specified by the user to the input image 120 and causes the image display part 103 to display the processed input image again.

Description

  The present invention relates to an image depth designation method, program, and apparatus for image processing that handles a three-dimensional image.

There is a technique for generating a three-dimensional image (hereinafter referred to as “3D image”) from a two-dimensional picture or photograph (hereinafter referred to as “2D image”).
There are two main methods for producing a 3D image. A method for simultaneously capturing a right-eye image and a left-eye image using two cameras, and a later editing of a 2D image captured by one camera. There is a method for generating a parallax image. The present invention relates to the latter method, and to 2D3D conversion technology.

  The problem of building 3D models from 2D images has received a great deal of attention in the computer vision literature. For an overview, see “A theory of shape by space carving” Intl. Journal of Computer Vision Vol. 38: 3, pp. 199-218, 2000.

  Most prior art computer vision methods have focused on automatically restoring the scene with little or no user intervention. As a result, computer vision methods make prior assumptions about scene geometry and reflectivity. These assumptions are often incorrect.

  Therefore, since it is difficult to automatically generate a 3D image desired by the user from the 2D image, it is general that the user sequentially gives instructions to the computer while watching the conversion process.

  As a technique necessary for generating a 3D image, there are segmentation and depth (also referred to as layer or depth).

  For segmentation when generating a 3D image, there is a technique of segmenting superpixels. A superpixel is a polygon pixel raster that is generally larger than a single pixel in a digital image and can be rendered with the same color and brightness. The main advantage of using superpixels for 3D construction is increased computational efficiency, and the superpixel representation significantly reduces the number of image primitives required compared to the pixel representation.

  As an example in which segmentation of superpixels is applied to 3D image generation, a technique for segmenting an image into superpixels using a clustering objective function is known (for example, a technique described in Patent Document 1). This objective function includes two components: an entropy rate of random walk and a balancing function. The entropy rate forms a compact, homogenous cluster, while the balance function produces clusters with a similar size. For clustering, a graph is constructed in which data points correspond to vertices and pair-wise similarities correspond to edge weights. Divide the graph by maximizing the objective function, subject to matroid constraints.

  Further, regarding the depth, the following technique is known (for example, the technique described in Patent Document 2). This technology uses a conventional 3D video (3DV) data format that includes both 2D video and depth maps to facilitate new video applications such as 3D television (3DTV) and free-viewpoint video (FVV). Can be used to allow additional views to be rendered at the user end. Examples of such a 3DV format include 2D plus depth (2D + Z) including 2D (2D) video and a corresponding depth map, and 2D + Z and occlusion video with concealment depth added. Layered Depth Video (LDV) and the like. According to a general aspect, a syntax element indicating an inter-layer dependency structure between 3D video (3DV) layers is generated. Based on the inter-layer dependency structure, an inter-layer reference for a picture is identified from the layers of the 3DV layers. The picture is encoded based at least in part on the inter-layer reference.

  Further, regarding the depth (depth), the following technique is known (for example, the technique described in Patent Document 3). This technique reduces the time and effort of the producer when generating a 3D image from a 2D image. An object of the present invention is to provide a technology capable of generating a quality 3D image, a step of generating a depth map of the input image based on the input image and a depth model, and a step of processing the generated depth map And generating an image of another viewpoint based on the input image and the processed depth map. In the step of processing the depth map, the gain of the depth map is individually subjected to gradation processing for each of a plurality of areas in the screen designated by a plurality of mask patterns set externally.

JP 2012-234528 A Special table 2012-525769 gazette JP 2013-156957 A

  However, the above-described conventional technology only explains the theory of the degree of division and the theory of depth (layer structure), and does not show a specific user interface.

  Further, conventionally, there has been no interface for designating the degree of division and depth by manual designation.

  An object of the present invention is to make it possible to easily specify the depth of an image.

  In one example, a method for designating a depth on an input image to generate a three-dimensional output image from the two-dimensional input image, the image dividing step of dividing the input image into a plurality of divided regions And an image display step for displaying the divided input image and a depth designation step for allowing the user to designate a depth while selecting a divided area on the displayed input image.

  According to the present invention, it is possible to easily specify the depth of an image.

It is a functional block diagram of one embodiment of the present invention. It is a figure which shows the hardware structural example of the computer which can implement | achieve one Embodiment of this invention. It is a figure which shows the example of a user interface of one Embodiment of this invention. It is operation | movement explanatory drawing (the 1) of one Embodiment of this invention. It is operation | movement explanatory drawing (the 2) of one Embodiment of this invention. It is operation | movement explanatory drawing (the 3) of one Embodiment of this invention.

  Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. FIG. 1 is a functional block diagram of an embodiment of the present invention. The present embodiment is a depth designation device 100 for designating a depth on an input image 120 in order to generate a three-dimensional output image 130 from the two-dimensional input image 120.

  The division degree designation unit 101 allows the user to designate the division degree of the input image 120.

  The image dividing unit 102 divides the input image into a plurality of divided regions with a degree of division designated by the user. The plurality of divided regions are calculated by, for example, superpixel segmentation processing.

  The image display unit 103 displays the divided input image 120.

  The depth designation unit 104 causes the user to designate a depth on the displayed input image 120 while allowing the user to select a divided area.

  The tilt designation unit 105 causes the input image 120 displayed to the user to designate, for example, a horizontal tilt and a vertical tilt using, for example, a slide bar.

  The image processing unit 106 adds, to the input image 120, a tilt effect specified by the user, for example, a horizontal tilt and a vertical tilt, and a depth effect corresponding to the depth in units of divided areas specified by the user. 103. A 3D image 130 is created by the image processing unit 106. Since this embodiment shows a user interface method for designating the depth of an image when creating a 3D image from 2D, the description of the 3D image creation method itself is omitted.

  FIG. 2 is a diagram illustrating a hardware configuration example of a computer capable of realizing one embodiment of the present invention.

  2 includes a CPU 201, a ROM (Read Only Memory) 202, a RAM (Random Access Memory) 203, an input device 204, an output device 205, an external storage device 206, a communication interface 207, and a portable device. A portable recording medium driving device 208 into which the recording medium 209 is inserted is included, and these are connected to each other by a bus 210. The configuration shown in the figure is an example of a computer that can implement the above system, and such a computer is not limited to this configuration.

  The CPU 201 controls the entire computer. The ROM 202 is a storage device that stores a depth designation processing program. A RAM 203 is a readable / writable memory that temporarily stores the program or data when executing the program or updating image data. The CUP 201 performs overall control of the depth designation device 100 of the present embodiment by reading the program from the ROM 202 or the external storage device 407 to the RAM 203 and executing it.

  The input device 204 detects an input operation by a user using a keyboard, a mouse, or the like, and notifies the CPU 201 of the detection result. The user uses the input device 404 to perform depth designation, which will be described later.

  The output device 205 outputs the input image 120 and the output image 130 in the external storage device 206 to a display display device and a printing device under the control of the CPU 201.

  The external storage device 206 is, for example, a hard disk storage device, and is used for storing the input image 120, the output image 130, and other various data.

  The portable recording medium driving device 208 accommodates a portable recording medium 209 such as an optical disk, SDRAM, or compact flash, and has an auxiliary role for the external storage device 206.

  The communication interface 207 is a device for connecting a communication line of, for example, a LAN (local area network) or a WAN (wide area network).

  The system according to the present embodiment is realized by the CPU 201 executing a program in which the functions of the functional units in FIG. 1 are installed. The program may be recorded and distributed in, for example, the external storage device 206 or the portable recording medium 209, or may be acquired from the network by the network connection device 207.

  Details of the depth designation processing in the present embodiment will be sequentially described below in accordance with the configuration diagram of FIGS. 1 and 2, the diagram showing the user interface of FIG. 3, and the operation explanatory diagrams of FIGS. 4 to 6. It should be noted that each part of FIG. 1 or FIG.

  FIG. 3 is a diagram showing an example of a user interface according to the present embodiment that constitutes the input device 204 and the output device 205 of FIG. The user interface of the present embodiment includes an image display unit 1, a slide bar 2 that determines a horizontal angle when displaying a 3D image, a slide bar 3 that determines a vertical angle when displaying a 3D image, and the degree of image division (number of divided areas). Is composed of a slide bar 4 and a surface selection 5.

  FIG. 3 shows an initial state immediately after the user selects the input image 120 to be converted to 3D. At this time, the input image 120 which is the selected 2D image is displayed on the image display unit 1.

  When the user moves the division degree instruction bar 4 to the right, the CPU 201 executes functional processing of the image dividing unit 102. As a result, as shown in FIG. 4A, a divided region divided into a plurality of white dividing lines 6 is displayed on the input image 120 displayed on the image display unit 1. In the division degree instruction bar 4, the left end designates the number of divisions = 1, and the right end designates the number of divisions = about 500 (which may ultimately be in units of one pixel). As a function of the region division processing by the image dividing unit 102, for example, a superpixel segmentation method described in Patent Document 1 may be used. A super pixel is a division method based on RGB pixel values of individual pixels, and has global features compared to individual pixels. Alternatively, as the function of the image dividing unit 102, region division may be performed using the Graph Cuts method described in the following document.

  Y. Boykov and G. Funka-Lea: “Interactive Graph Cuts for Optimal Boundary & Region Segmentation of Objects in ND Images”, Proceedings of “Interference Convence CV”. I, p. 105-112, July 2001.

  The slide bars 2 and 3 realize the function of the inclination specifying unit 105 in FIG. When the user operates the slide bar 2 for determining the horizontal angle, the CPU 201 executes functional processing of the image processing unit 106 in FIG. As a result, the input image 120 is viewed obliquely from the left or right and displayed on the image display unit 1 as illustrated in FIG. 4B. When the slide bar 2 is moved to one side with an inclination angle of 0, the inclination angle of the display image changes according to the amount of movement. In FIG. 4B, a state tilted to the right to some extent is displayed. If the slide bar 2 is moved to the right, an image tilted to the right is displayed. Similarly to the slide bar 2 that determines the horizontal angle, the slide bar 3 that determines the vertical angle is a state in which the input image 120 is viewed obliquely from above or below by the CPU 201 executing the function processing of the image processing unit 106. The vertical tilt angle of the input image 120 displayed on the image display unit 1 is controlled. This inclination may be seen not only in the horizontal direction and the vertical direction but also by rotating around.

  When the user selects a face number by face selection 5, the CPU 201 displays the cursor 7 as illustrated in FIG. The surface selection number determines the depth of the image assigned to the surface. The larger the number, the closer the front surface, and the smaller the number, the deeper surface is designated. In general, a background image is assigned to the back side, and an image of a person or the like is assigned to the front side. The color and size of the cursor 7 change depending on the selected surface number and the state of the division degree instruction bar 4. For example, the smaller the number of divisions, the smaller the color, and different colors are assigned corresponding to the surface numbers. FIG. 5A shows a state in which five faces can be selected and face number 3 is selected. The cursor 7 in this state is displayed in a middle thickness corresponding to the selected division number and a gray color assigned to the surface number 3.

  In a state where the cursor 7 is displayed, the user can drag the cursor 7 on the divided area with a mouse or the like. In the divided area where the cursor 7 is dragged, a locus of the same color as the color of the cursor 7 is displayed as shown in FIG. By designating divided areas with the cursor 7, the depths of the designated divided areas are made equal.

  The user interface for the surface selection 5 and the user interface for the draggable cursor 7 described above realize the function of the depth designation unit 100 in FIG.

  As described above, when the divided area is designated by the drag movement of the cursor 7 at the depth corresponding to the surface number selected by the surface selection 5, the CPU 201 executes the function processing of the image processing unit 106 in FIG. As a result, a depth effect corresponding to the face number (depth) specified by the user is added to the divided area specified by the user to the input image 120 and displayed on the image display unit 1. As a result, the divided area designated by the user is displayed on the image display unit 1 so as to pop out. FIG. 5B shows a state in which the user drags the cursor 7 on the divided area near the human face according to the locus 8. In addition, when the pop-out process is performed, a three-dimensional cross section is displayed on the left side of the locus 8. This cross section is displayed in the same color as the color of the cursor 7 so that the user can easily grasp the currently selected surface number. Thereby, in order to make the designated area visually conspicuous, an effect that a 3D image from an oblique direction can be displayed is obtained.

  FIG. 6A shows a state where the surface number 5 is selected by the surface selection 5 and the number of divisions is specified more finely by the division degree instruction bar 4. Since the number of divisions is fine, the cursor indicated by the black arrow is thin. It is controlled so that it can be easily moved by dragging a fine divided area with a thin cursor, and roughly dragging a large divided area with a thick cursor. Since the surface number of the fifth surface is selected, the cursor color is controlled to the color assigned to the fifth surface. FIG. 6B shows a state after the cursor is dragged. The thin black locus on the shoulder portion of the person is the portion where the cursor has been dragged.

  By repeating the above operation, the user can generate a 3D image. In this embodiment, a 3D image is displayed in a 3D style on a 2D display screen. However, since the 3D image actually has depth data, it is physically raised using a 3D printer or the like. It is possible to print 3D images with

As described above, the user can be confident that the degree of division (fineness) can be specified so that global and local selection is easy, and the viewpoint direction can be changed (horizontal angle and vertical angle) so that the uneven state can be easily understood. In the above description, the 3D map can be generated using the above function according to the required level (fineness of the finish) and can be repeated until the required level is satisfied.
Thereby, in this embodiment, every time the depth plane setting specified by the user is performed, the state in which the 3D image is actually created is sequentially displayed, so that the user can easily imagine the completed state, and the depth plane setting can be performed. It can be done easily.

Regarding the above embodiments, the following additional notes are disclosed.
(Appendix 1)
A method for designating a depth on an input image to generate a three-dimensional output image from a two-dimensional input image,
An image dividing step of dividing the input image into a plurality of divided regions;
An image display step for displaying the divided input image;
A depth specifying step for allowing a user to specify a depth while selecting the divided area on the displayed input image;
Image depth designation method for causing a computer to execute.
(Appendix 2)
The image depth designation method according to claim 1, wherein the image division step includes a division degree designation step for allowing a user to designate a division degree of the input image.
(Appendix 3)
The image depth designation method according to appendix 1 or 2, wherein a superpixel segmentation process is executed in the image dividing step.
(Appendix 4)
The image depth designation method according to any one of appendices 1 to 3, wherein, in the depth designation step, a process of designating all the divided areas selected when the user designates the depth as the depth is executed.
(Appendix 5)
An inclination specifying step for allowing the user to specify an inclination for the displayed input image;
An image processing step of adding to the input image a tilt specified by the user and a depth effect corresponding to the depth of the divided region unit specified by the user, and causing the image to be displayed in the image display step;
The image depth designation method according to any one of appendices 1 to 3, further causing the computer to execute the following.
(Appendix 6)
The image depth designation method according to claim 5, wherein the tilt designation step causes the user to designate a tilt in the horizontal direction or a tilt in the vertical direction.
(Appendix 7)
A program for designating a depth on the input image in order to generate a three-dimensional output image from the two-dimensional input image,
An image dividing step of dividing the input image into a plurality of divided regions;
A split image display step for displaying the split input image;
A depth specifying step for allowing the user to specify a depth while selecting the divided region on the displayed input image;
Depth designation program for causing computer to execute.
(Appendix 8)
An apparatus for designating a depth on the input image to generate a three-dimensional output image from the two-dimensional input image,
An image dividing unit for dividing the input image into a plurality of divided regions;
A divided image display unit for displaying the divided input image;
A depth designation unit that allows the user to designate a depth while selecting the divided area on the displayed input image;
An image depth specifying device comprising:
(Appendix 9)
The image depth designation device according to appendix 8, wherein the image division unit includes a division degree designation unit that allows a user to designate a division degree of the input image.

DESCRIPTION OF SYMBOLS 1 Image display part 2 Slide bar which determines horizontal angle 3 Slide bar which determines vertical angle 4 Divide degree instruction bar 5 Face selection 6 Boundary line of divided area 7 Cursor 8 Trajectory of drag movement 100 Image depth designation device 101 Divide degree designation Unit 102 Image division unit 103 Image display unit 104 Depth designation unit 105 Tilt designation unit 106 Image processing unit 120 Input image 130 Output image 201 CPU
202 ROM (Read Only Memory)
203 RAM (Random Access Memory)
204 Input Device 205 Output Device 206 External Storage Device 207 Communication Interface 208 Portable Recording Medium Drive Device 209 Portable Recording Medium 210 Bus

Claims (9)

A method for designating a depth on an input image to generate a three-dimensional output image from a two-dimensional input image,
An image dividing step of dividing the input image into a plurality of divided regions;
An image display step for displaying the divided input image;
A depth specifying step for allowing a user to specify a depth while selecting the divided area on the displayed input image;
Image depth designation method for causing a computer to execute.   The image depth designation method according to claim 1, wherein the image division step includes a division degree designation step for allowing a user to designate a division degree of the input image.   3. The image depth designation method according to claim 1, wherein a superpixel segmentation process is executed in the image dividing step.   The image depth designation method according to any one of claims 1 to 3, wherein in the depth designation step, a process of designating all the divided areas selected when the user designates the depth as the depth is executed. . An inclination specifying step for allowing the user to specify an inclination for the displayed input image;
An image processing step of adding to the input image a tilt specified by the user and a depth effect corresponding to the depth of the divided region unit specified by the user, and causing the image to be displayed in the image display step;
The image depth designation method according to claim 1, further causing the computer to execute.   6. The image depth designation method according to claim 5, wherein the tilt designation step causes the user to designate a tilt in the horizontal direction or a tilt in the vertical direction. A program for designating a depth on the input image in order to generate a three-dimensional output image from the two-dimensional input image,
An image dividing step of dividing the input image into a plurality of divided regions;
A split image display step for displaying the split input image;
A depth specifying step for allowing the user to specify a depth while selecting the divided region on the displayed input image;
Depth designation program for causing computer to execute. An apparatus for designating a depth on the input image to generate a three-dimensional output image from the two-dimensional input image,
An image dividing unit for dividing the input image into a plurality of divided regions;
A divided image display unit for displaying the divided input image;
A depth designation unit that allows the user to designate a depth while selecting the divided area on the displayed input image;
An image depth specifying device comprising:   The image depth designation device according to claim 8, wherein the image division unit includes a division degree designation unit that allows a user to designate a division degree of the input image.