JP7170052B2 - 3D image generation device, 3D image generation method, and program - Google Patents

Description

The present invention relates to a three-dimensional image generation device, a three-dimensional image generation method, and a program, and more particularly to a three-dimensional image generation device, a three-dimensional image generation method, and a program for generating parallax images from two-dimensional images.

2. Description of the Related Art Conventionally, techniques have been proposed for generating three-dimensional image data by adding depth information to a subject (object) and background of an input two-dimensional image.

For example, in Patent Document 1, a subject area is extracted from an input subject image (two-dimensional image), a parallax map representing the depth distribution of the subject is extracted from the subject area, and a multi-view image ( A technique for generating a parallax image) is described.

Further, in Japanese Patent Application Laid-Open No. 2002-200010, a contour equidistant from the position of a lens photographed from an input two-dimensional image is extracted by image processing, a plurality of partial images corresponding to the contour are generated, and the partial images are generated. Techniques are described for generating right-eye and left-eye images arranged in correspondingly positioned layers.

Japanese Patent Application Laid-Open No. 2003-47027 JP 2008-153805 A

Here, in the technique described in Patent Literature 1, the user himself/herself designates a subject area that he/she wants to project as a stereoscopic image. On the other hand, in the technology described in Patent Document 2, contour detection (object detection) is automatically performed. Subject detection that is performed automatically, such as the technique described in Patent Document 2, does not necessarily perform subject detection that is suitable for a three-dimensional image. Specifically, when the detected subject is used as the foreground and other subjects than the detected subject are used as the background, the number of foreground subjects may be too large, or the depth relationship between the foreground subjects may be complicated or fine. If it is too large, the three-dimensional effect may not be well expressed. Therefore, when subject detection is automatically performed and a three-dimensional image is generated using the detected subject as the foreground, it is necessary to select a subject that satisfactorily expresses the three-dimensional effect.

The above-mentioned Patent Documents 1 and 2 do not mention selection of a subject from the viewpoint of well expressing a stereoscopic effect.

The present invention has been made in view of such circumstances, and its object is to provide a three-dimensional image that appropriately expresses a natural three-dimensional effect even when a subject is extracted from a two-dimensional image by image processing. An object of the present invention is to provide a three-dimensional image generating device, a three-dimensional image generating method, and a program capable of generating an image.

A three-dimensional image generating apparatus, which is one aspect of the present invention for achieving the above object, includes an image input unit for inputting a two-dimensional image, and an object of the two-dimensional image by image processing the two-dimensional image. a subject extraction unit for extracting a subject; a subject determination unit for determining a main subject and a non-main subject among the subjects based on subject-related information; and a parallax image generation unit that generates two or more parallax images by relatively shifting the foreground and the background, using the rest as the background.

According to this aspect, regarding subjects extracted by performing image processing on a two-dimensional image, a main subject and a non-main subject are determined based on the subject-related information, and a parallax image is obtained with the main subject as the foreground and the non-main subject as the background. is generated. As a result, according to this aspect, a three-dimensional image that appropriately expresses a natural three-dimensional effect is generated.

Preferably, a background generator is provided for complementing an area other than the main subject in the two-dimensional image by image processing to generate the background, and the parallax image generator generates the parallax image based on the background generated by the background generator. to generate

Preferably, the subject-related information includes at least one of position information from the center of the two-dimensional image, depth information, subject size information, and reliability of subject extraction by the subject extraction unit.

Preferably, the subject determining section adds another area to the main subject based on the subject-related information.

Preferably, the three-dimensional image generation device includes a depth estimation unit for estimating depth information of a subject from a two-dimensional image, and the subject determination unit determines the main subject based on the depth information estimated by the depth estimation unit. .

Preferably, the parallax image generator generates three or more parallax images.

Preferably, the three-dimensional image generation device includes a display section that displays parallax images.

Preferably, the three-dimensional image generation device includes a shift amount adjustment reception unit that receives adjustment of the shift amount of the parallax images, and the parallax image generation unit adjusts the parallax images based on the shift amount received by the shift amount adjustment reception unit. Adjust shift.

Preferably, the three-dimensional image generating device includes a background changing unit that replaces the background with another image, and the parallax image generating unit generates two or more parallax images with the main subject as the foreground and the other image as the background. .

Preferably, the three-dimensional image generation device includes a foreground changing unit that replaces the foreground with another image, and the parallax image generation unit uses the other image as the foreground and generates a parallax image from the foreground and the background.

Preferably, the three-dimensional image generation device includes a lenticular print data generation unit that generates lenticular print data based on the parallax image.

According to another aspect of the present invention, there is provided a three-dimensional image generating method comprising steps of inputting a two-dimensional image, performing image processing on the two-dimensional image to extract a subject of the two-dimensional image, and determining a main subject and a non-main subject based on the subject relationship information; removing the non-main subject in the two-dimensional image, setting the main subject as the foreground, setting the non-main subject as the background, and comparing the foreground and the background relative to each other; and C. shifting to generate two or more parallax images.

According to another aspect of the present invention, a program comprises steps of inputting a two-dimensional image, extracting a subject of the two-dimensional image by performing image processing on the two-dimensional image, determining a main subject based on the subject-related information; removing a non-main subject in the two-dimensional image; setting the main subject as the foreground; setting the non-main subject as the background; A computer is caused to execute a three-dimensional image generation process including the step of generating the above parallax images.

According to the present invention, with respect to subjects obtained by image processing a two-dimensional image, a main subject and a non-main subject are determined based on subject-related information, and a parallax image is obtained with the main subject as the foreground and the non-main subject as the background. is generated, a three-dimensional image in which a natural three-dimensional effect is appropriately expressed is generated.

FIG. 1 is a diagram showing the appearance of a computer. FIG. 2 is a block diagram showing a functional configuration example of the 3D image generating device. FIG. 3 is a diagram showing an example of a two-dimensional image. FIG. 4 is a diagram showing a subject extracted from a two-dimensional image. FIG. 5 is a diagram illustrating an example of first depth information acquired by a first depth information acquisition unit; FIG. 6 is a diagram showing a layer configuration of a three-dimensional image. FIG. 7 is a diagram showing the determined main subject. FIG. 8 is a diagram showing the background after the subject has been extracted. FIG. 9 shows a conceptual diagram in which the foreground and background are arranged based on depth information. FIG. 10 is a diagram illustrating an example of a parallax image generated by a parallax image generation unit; FIG. 11 is a flow chart showing a method for generating a three-dimensional image. FIG. 12 conceptually shows a case where another object is determined as the main subject. FIG. 13 is a block diagram showing a functional configuration example of a 3D image generating device.

Preferred embodiments of a three-dimensional image generating apparatus, a three-dimensional image generating method, and a program according to the present invention will be described below with reference to the accompanying drawings.

FIG. 1 is a diagram showing the appearance of a computer equipped with the 3D image generating device of the present invention.

The computer 3 is equipped with a three-dimensional image generation device 11 (FIG. 2), which is one aspect of the present invention. A two-dimensional image 100 is input to the computer 3, and a display section including a monitor (display section) 9 and an input section including a keyboard 5 and a mouse 7 are connected. The illustrated form of the computer 3 is an example, and a device having functions similar to those of the computer 3 can include the three-dimensional image generating device 11 of the present invention. For example, it is possible to mount the three-dimensional image generation device 11 on a tablet terminal.

The computer 3 displays the parallax images generated by the 3D image generating device 11 (FIG. 2). Then, the user confirms the parallax image displayed on the monitor 9 and inputs a correction command using the keyboard 5 and the mouse 7 . For example, the user confirms the parallax image displayed on the monitor 9 and inputs the shift amount adjustment via the keyboard 5 and the mouse 7 .

FIG. 2 is a block diagram showing a functional configuration example of the 3D image generating device 11. As shown in FIG. The hardware structure for executing various controls of the three-dimensional image generating apparatus 11 shown in FIG. 2 is various processors as shown below. For various processors, the circuit configuration can be changed after manufacturing such as CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), which is a general-purpose processor that executes software (program) and functions as various control units. Programmable Logic Device (PLD), which is a processor, ASIC (Application Specific Integrated Circuit), etc. be

One processing unit may be composed of one of these various processors, or composed of two or more processors of the same type or different types (for example, a plurality of FPGAs, or a combination of a CPU and an FPGA). may Also, a plurality of control units may be configured by one processor. As an example of configuring a plurality of control units with a single processor, first, as represented by a computer such as a client or a server, a single processor is configured by combining one or more CPUs and software. There is a form in which a processor functions as multiple controllers. Secondly, as typified by System On Chip (SoC), etc., there is a form of using a processor that realizes the functions of the entire system including multiple control units with a single IC (Integrated Circuit) chip. be. In this way, various control units are configured using one or more of the above various processors as a hardware structure.

The three-dimensional image generation device 11 includes an image input unit 13, a subject extraction unit 15, a subject determination unit 17, a background generation unit 19, a depth estimation unit 31, a parallax image generation unit 21, a shift amount adjustment reception unit 23, and data for lenticular printing. A generation unit 29 , a display control unit 25 and a storage unit 27 are provided. The storage unit 27 stores programs, information related to various controls of the three-dimensional image generation device 11, and the like. Also, the display control unit 25 controls display by the monitor 9 . A monitor 9 displays parallax images.

A two-dimensional image is input to the image input unit 13 . A two-dimensional image has an object, which is a subject, and a background. When a 3D image is generated from a 2D image, the object and the background are placed on different layers. Here, a three-dimensional image is an image generated so as to give a viewer a stereoscopic effect. A three-dimensional image includes, for example, a parallax image composed of a left-eye image and a right-eye image, and an image composed of lenticular printing data.

FIG. 3 is a diagram showing an example of a two-dimensional image input to the image input unit 13. As shown in FIG. A two-dimensional image 100 has an object 101 , an object 103 and an object 105 and a background 107 .

The subject extraction unit 15 ( FIG. 2 ) extracts the subject of the two-dimensional image 100 by image processing the two-dimensional image 100 . A well-known technique is used as the image processing technique for extracting the subject from the two-dimensional image 100 performed by the subject extraction unit 15 . In addition, the subject extracting unit 15 can extract the subject from the two-dimensional image 100 using a machine-learned detector as an example of image processing technology. Note that the subject extracting unit 15 may calculate the reliability of being a subject and assign the reliability to each extracted subject.

FIG. 4 is a diagram showing a subject extracted from the two-dimensional image 100 by the subject extraction unit 15. As shown in FIG.

The subject extraction unit 15 performs image processing on the two-dimensional image 100 to extract objects 101, 103, and 105 as subjects. The subject extraction unit 15 detects objects 101, 103, and 105 included in the two-dimensional image 100, detects the contours of each object, and extracts each object as a subject.

The depth estimation unit 31 ( FIG. 2 ) estimates the depth information of the object from the two-dimensional image 100 . Then, the parallax image generator 21 generates parallax images based on the estimated depth information. The depth estimation unit 31 includes a first depth information acquisition unit 33 and a second depth information acquisition unit 35 . The first depth information acquisition unit 33 acquires first depth information composed of depth information for each predetermined region of the two-dimensional image 100 . The first depth information acquisition unit 33 acquires first depth information by image processing the two-dimensional image 100 . The first depth information is detailed depth information, and a representative value (second depth information) is calculated based on this detailed depth information. The degree of detail of the first depth information varies depending on the size of the predetermined area. For example, the first depth information for each minute area (for each 1×1 pixel) is extremely detailed information. Also, even the first depth information for every 5×5 pixels is sufficiently detailed information, and the representative value can be calculated.

For example, the first depth information acquisition unit 33 acquires first depth information using a known depth estimation algorithm from the two-dimensional image 100 . Also, the first depth information acquisition unit 33 may calculate the first depth information by machine learning.

Also, the first depth information acquisition unit 33 may acquire first depth information that is input separately from the two-dimensional image 100 . For example, the first depth information acquisition unit 33 may acquire the first depth information acquired by a measuring device provided in the imaging device when capturing the two-dimensional image 100 . That is, the first depth information acquisition unit 33 acquires depth information of the objects (101, 103, and 105) of the two-dimensional image 100 and the background 107, which are input separately from the two-dimensional image 100 and measured by the measuring instrument. get. Here, the measuring device includes a stereo camera, a depth sensor, or a ToF (Time Of Flight) camera.

FIG. 5 is a diagram showing an example of the first depth information acquired by the first depth information acquisition unit 33, showing a distance image 400. As shown in FIG.

The distance image 400 has distance information for each pixel. Specifically, it has distance information for each pixel in the object 101 , the object 103 , the object 105 , and the background 107 . Since each pixel has distance information, for example, if the object 101 has unevenness, the distance image 400 also reflects the difference in the distance information resulting from the unevenness.

Based on the first depth information corresponding to the regions of the object 101, the object 103, the object 105, and the background 107, the second depth information acquisition unit 35 obtains a representative value (second depth information) regarding the depth of each region. Calculate Specifically, the second depth information acquisition unit 35 acquires depth information based on the average value, median value, or mode value of the first depth information corresponding to the object 101, the object 103, the object 105, and the background 107. to obtain a representative value of By obtaining the representative value by the second depth information obtaining unit 35, for example, even if there are a plurality of values of the first depth information in the area of the object 101, one layer in which the object 101 is placed can be determined. can decide. The depth estimation unit 31 then determines the depth positions of the first layer and the second layer based on the second depth information.

FIG. 6 is a diagram showing the layer configuration of a three-dimensional image generated from the two-dimensional image 100. As shown in FIG. In the following description, the first layer will be referred to as an object layer, and the second layer will be referred to as a background layer. Also, there may be one or more object layers.

The depth estimation unit 31 generates data for arranging the object and the background 107 on each layer based on the representative values acquired by the second depth information acquisition unit 35 . The object 101 is placed on the first object layer 121 . Objects 103 and 105 are placed on a second object layer 123 . Also, the background 107 is placed on the background layer 125 .

Object 101 has representative value a1, object 103 has representative value a2, and object 105 has representative value a2. Also, the background 107 has a representative value a3. The depth estimation unit 31 arranges the object and the background 107 on each layer according to this representative value. Thus, by arranging the object and the background 107 on each layer according to the representative value calculated based on the first depth information, a more natural depth relationship can be constructed.

The subject determination unit 17 (FIG. 2) determines a main subject and a non-main subject among the extracted subjects based on the subject-related information. Here, subject-related information is information for determining main subjects and non-main subjects among a plurality of subjects, and is information for determining main subjects so that a natural three-dimensional effect can be appropriately expressed in a three-dimensional image. is. For example, the subject-related information includes at least one of position information from the center of the two-dimensional image 100, depth information, size information of the subject, and reliability of subject extraction by the subject extraction unit 15. FIG.

When positional information from the center of the two-dimensional image 100 is used as the subject-related information, the subject determination unit 17 determines the subject closest to the center of the two-dimensional image 100 as the main subject. Further, when positional information from the center of the two-dimensional image 100 is adopted as the subject-related information, the subject determination unit 17 may determine a subject within a predetermined range from the center of the two-dimensional image 100 as the main subject. good.

Further, when the depth information is adopted as the subject-related information, the subject determination unit 17 determines the foremost subject as the main subject. Further, when depth information is adopted as the subject-related information, the subject determination unit 17 may determine a subject within a predetermined range of distance in the depth direction as the main subject.

Further, when subject size information is adopted as the subject-related information, the subject determination unit 17 determines the subject with the largest size as the main subject. Further, when subject size information is adopted as the subject-related information, the subject determination unit 17 may determine a subject having a size equal to or larger than a predetermined threshold value as the main subject. Note that the subject determination unit 17 may determine the main subject based on the size information after normalizing the size of the subject based on the depth information estimated by the depth estimation unit 31 .

Further, when the subject determination unit 17 adopts the reliability of subject extraction by the subject extraction unit 15 as the subject relation information, the subject determination unit 17 determines the subject with the highest reliability as the main subject. Further, when the reliability of extraction of the subject by the subject extraction section 15 is adopted as the subject relation information, the subject determination section 17 determines a subject with a degree of reliability equal to or higher than a predetermined threshold as the main subject.

FIG. 7 is a diagram showing main subjects determined from the extracted subjects shown in FIG. The object 101 has been determined as the main subject by the subject determination unit 17 . On the other hand, the object 103 and the object 105 (FIG. 4) are determined as non-main subjects, and the object determination unit 17 deletes the objects 103 and 105 determined as non-main subjects.

In the case shown in FIG. 7, the subject determination unit 17 determines the main subject based on the estimated depth information. Specifically, the objects 101, 103, and 105 have representative values as depth information as described with reference to FIG. Specifically, the object 101 has a representative value a1, and the objects 103 and 105 have a representative value a2. Then, the subject determination unit 17 determines the foremost object 101 as the main subject, and determines the objects 103 and 105 as non-main subjects. As described above, the subject determination unit 17 determines the main subject based on the depth information, so that a natural three-dimensional effect can be appropriately expressed in the three-dimensional image.

The background generator 19 (FIG. 2) generates the background 107 used for the parallax images. Specifically, the background generating unit 19 generates the background 107 by interpolating regions other than the main subject in the two-dimensional image 100 by image processing. The background generation unit 19 complements areas other than the main subject in the two-dimensional image 100 using a known technique.

FIG. 8 is a diagram showing the background 107 after the subject has been extracted. When the subject extracting unit 15 extracts the object 101, the object 103, and the object 105 as subjects, blank portions 111, 113, and 115 corresponding to the object 101, the object 103, and the object 105 exist in the background 107. do. Since the object 101 has been determined as the main subject, the blank portions 113 and 115 corresponding to the non-main subjects, the objects 103 and 105, are complemented using the background 107. FIG.

The parallax image generation unit 21 (FIG. 2) removes the non-main subject in the two-dimensional image 100, sets the main subject as the foreground, sets the non-main subject as the background 107, and shifts the foreground and the background 107 relatively to form a two-dimensional image. The above parallax images are generated.

FIG. 9 shows a conceptual diagram in which the foreground (object 101) and background 107 are arranged based on depth information.

The arrangement position differs depending on the depth information of the background 107 and the object 101 respectively. Specifically, the position of the layer of the background 107 is indicated by B, the position of the layer of the object 101 is indicated by F, and the positions differ according to the depth information. The parallax image generator 21 generates parallax images using the difference in arrangement based on the depth information.

FIG. 10 is a diagram showing an example of a parallax image generated by the parallax image generator 21. As shown in FIG. The parallax image generator 21 generates a left-eye image 131 and a right-eye image 133 based on the depth information of the background 107 and foreground (object 101) shown in FIG. In FIG. 10, as parallax images, a left-eye image 131 and a right-eye image 133 are superimposed on an image 135 viewed from the front (indicated by dotted lines). In left-eye image 131 , the position of object 101 has been shifted in the direction of arrow L with respect to background 107 . Also, in the right-eye image 133 , the position of the object 101 is shifted in the direction of arrow R with respect to the background 107 . In the above description, the case where the parallax image generation unit 21 generates two parallax images has been described, but the present invention is not limited to this. For example, the parallax image generator 21 may generate three or more parallax images. For example, when the parallax image generator 21 generates three parallax images, a left-eye image, a right-eye image, and a front image are generated. A front image is an image in which the object 101 is not shifted with respect to the background 107 .

By alternately displaying the left-eye image 131 and the right-eye image 133 on the monitor 9 , the user can confirm the stereoscopic effect of the object 101 . That is, by alternately displaying the left-eye image 131 and the right-eye image 133 at a predetermined cycle, the user can confirm the stereoscopic effect of the object 101 . Here, the alternate display means, for example, alternately displaying the left-eye image 131 and the right-eye image 133 at a cycle of 0.1 to 0.5 seconds.

The shift amount adjustment reception unit 23 (FIG. 2) receives adjustment of the shift amount of the parallax image. For example, the user confirms the left-eye image 131 and the right-eye image 133 alternately displayed on the monitor 9, inputs the shift amount to the shift amount adjustment reception unit 23 via the keyboard 5 or the mouse 7, and adjusts the shift amount. I do. The parallax image generation unit 21 adjusts the shift amount of the parallax image based on the shift amount received by the shift amount adjustment reception unit 23 . Thereby, the user can confirm the alternate display of the left-eye image 131 and the right-eye image 133 and adjust the shift amount between the object 101 and the background 107 . Here, the shift amount refers to the amount by which the foreground is shifted with respect to the background, and the stereoscopic effect of the parallax image can be adjusted by adjusting the shift amount. In this way, the user can confirm the alternate display of the left-eye image 131 and the right-eye image 133 and adjust the amount of shift, so that the lenticular printing data obtained in the final step can also obtain the three-dimensional effect intended by the user. be able to.

The lenticular printing data generator 29 (FIG. 2) generates lenticular printing data based on the parallax image. For example, the lenticular printing data generation unit 29 generates lenticular printing data based on the left-eye image 131 and the right-eye image 133 .

Next, a three-dimensional image generation method will be described. FIG. 11 is a flow chart showing a three-dimensional image generating method (three-dimensional image generating process) using the three-dimensional image generating apparatus 11. As shown in FIG.

First, the two-dimensional image 100 is input to the image input unit 13 (step S10). After that, the subject extraction unit 15 extracts the object 101, the object 103, and the object 105 as subjects from the two-dimensional image 100 (step S11). Then, the subject determination unit 17 determines the object 101 as the main subject and the objects 103 and 105 as the non-main subjects based on the subject relation information (step S12). After that, the parallax image generator 21 generates a parallax image of the object 101 from the foreground and the background 107 (step S13).

Each configuration and function described above can be appropriately realized by arbitrary hardware, software, or a combination of both. For example, a program that causes a computer to execute the above-described processing steps (procedures), a computer-readable recording medium (non-temporary recording medium) recording such a program, or a computer capable of installing such a program However, it is possible to apply the present invention.

As described above, according to this aspect, regarding subjects obtained by image processing the two-dimensional image 100, the main subject and the non-main subject are determined based on the subject-related information, and the main subject is set as the foreground. Since the parallax image is generated with the background 107 other than the main subject, a three-dimensional image that appropriately expresses a natural three-dimensional effect is generated.

<Another example of determining the main subject>
Next, another example of determining the main subject will be described. In the above description, only the object 101 is determined as the main subject from among the objects 101, 103, and 105 using the subject relationship information of the depth information, but the present invention is not limited to this. Depending on the subject-related information, another object or multiple objects may be determined as the main subject.

FIG. 12 conceptually shows a case where another object is determined as the main subject.

The subject determination unit 17 determines the object 101 and the object 105 as main subjects. The subject determination unit 17 determines the main subject based on position information from the center O of the image as subject-related information. Specifically, since the object 101 and the object 105 are within the range of the distance b from the center O of the angle of view, the subject determination unit 17 determines the object 101 and the object 105 as main subjects. ing. As for the object 105, most of the object 105 (for example, 80% or more) is within the range of the distance b, so the subject determination unit 17 determines the object 105 as the main subject.

As described above, the subject determination unit 17 can determine a plurality of subjects as main subjects according to subject-related information.

The subject determination unit 17 can also add other areas to the main subject. Here, the other area means an area that has not been extracted by the subject extracting section 15 . The subject determination unit 17 adds an area located within a predetermined range (near position) to the main subject based on the determined main subject. Further, the subject determination unit 17 adds an area having a size within a predetermined range (similar size) to the main subject based on the determined main subject. In addition, the subject determination unit 17 performs size normalization using the depth information, and uses the determined main subject as a reference, and adds an area within a predetermined range of size (similar size) to the main subject. good. The subject determining unit 17 may add an area having a predetermined range of reliability (reliability of similarity) to the main subject based on the determined main subject.

In this way, the subject determination unit 17 can include even a region that has not been extracted by the subject extraction unit 15 as the main subject.

<Other examples>
In the above description, the foreground and background 107 are obtained from the two-dimensional image 100 input to the image input unit 13 to generate parallax images, but the present invention is not limited to this. For example, the parallax image generator 21 may replace the background 107 with another image, or replace the foreground with another image.

FIG. 13 is a block diagram showing a functional configuration example of the three-dimensional image generating device 11 of this example.

The three-dimensional image generation device 11 includes an image input unit 13, a subject extraction unit 15, a subject determination unit 17, a background generation unit 19, a depth estimation unit 31, a parallax image generation unit 21, a shift amount adjustment reception unit 23, and data for lenticular printing. A generation unit 29 , a display control unit 25 , a storage unit 27 , a background change unit 41 and a foreground change unit 43 are provided. In addition, the same code|symbol is attached|subjected to the location which already demonstrated in FIG. 2, and description is abbreviate|omitted.

The background changing unit 41 replaces the background 107 with another image. Specifically, the background changing unit 41 replaces the background 107 of the two-dimensional image 100 with another image for the main subject determined by the subject determining unit 17 in the two-dimensional image 100 . Here, another image is, for example, a background image of another two-dimensional image that has already been input to the image input unit 13 .

When the background changer 41 replaces the background 107 with another image, the parallax image generator 21 generates two or more parallax images with the main subject as the foreground and the other image as the background. Thereby, a parallax image having a different background from the two-dimensional image 100 from which the foreground is obtained can be generated.

The foreground changing unit 43 replaces the foreground with another image. Specifically, the foreground changing unit 43 replaces the obtained background 107 in the two-dimensional image 100 with another image for the foreground of the two-dimensional image 100 . Here, the other image is the main subject obtained from another two-dimensional image that has already been input to the image input unit 13, for example.

The parallax image generator 21 uses another image as the foreground and generates a parallax image from the foreground and the background 107 . Thereby, a parallax image having a different foreground than the two-dimensional image 100 from which the background is obtained can be generated.

Although examples of the present invention have been described above, it goes without saying that the present invention is not limited to the above-described embodiments, and that various modifications are possible without departing from the spirit of the present invention.

3: Computer 5: Keyboard 7: Mouse 9: Monitor 11: Three-dimensional image generation device 13: Image input unit 15: Subject extraction unit 17: Subject determination unit 19: Background generation unit 21: Parallax image generation unit 23: Shift amount adjustment Reception unit 25 : Display control unit 27 : Storage unit 29 : Lenticular printing data generation unit 31 : Depth estimation unit 33 : First depth information acquisition unit 35 : Second depth information acquisition unit 41 : Background change unit 43 : Foreground Change unit 100: two-dimensional image 101: object 103: object 105: object 107: background

Claims (14)

an image input unit to which a two-dimensional image having a background and a plurality of subjects is input;
a subject extraction unit that extracts the plurality of subjects other than the background of the two-dimensional image by performing image processing on the two-dimensional image;
a subject determination unit that determines a main subject and a non-main subject among the plurality of subjects based on subject-related information;
The non-main subject in the two-dimensional image is eliminated, the main subject is set as the foreground, the non-main subject is set as the background, and the foreground and the background are relatively shifted to generate two or more parallax images. a parallax image generation unit that
A three-dimensional image generation device comprising: A background generating unit that generates the background by complementing an area other than the main subject in the two-dimensional image by image processing,
The three-dimensional image generation device according to claim 1, wherein the parallax image generator generates the parallax image based on the background generated by the background generator. The subject-related information includes at least one of position information from the center of the two-dimensional image, depth information, size information of the plurality of subjects, and reliability of extraction of the plurality of subjects by the subject extraction unit. 3. The three-dimensional image generation device according to claim 1 or 2, comprising: 4. The three-dimensional image according to any one of claims 1 to 3, wherein the subject determining unit adds an area within a predetermined range, which is not extracted by the subject extracting unit, to the main subject with reference to the main subject. generator. A depth estimation unit that estimates depth information of the subject from the two-dimensional image,
5. The three-dimensional image generating apparatus according to any one of claims 1 to 4, wherein the subject determining section determines the main subject based on the depth information estimated by the depth estimating section. 6. The three-dimensional image generation device according to any one of claims 1 to 5, wherein the parallax image generator generates three or more parallax images. 7. The three-dimensional image generation device according to any one of claims 1 to 6, further comprising a display unit that displays the parallax image. A shift amount adjustment reception unit that receives adjustment of the shift amount of the parallax image,
The three-dimensional image generation device according to any one of claims 1 to 7, wherein the parallax image generation unit adjusts the shift of the parallax image based on the shift amount received by the shift amount adjustment reception unit. A background changing unit that replaces the background with another image,
9. The three-dimensional image generation device according to any one of claims 1 to 8, wherein the parallax image generator generates the two or more parallax images using the main subject as a foreground and the other image as a background. A foreground changing unit that replaces the foreground with another image,
The three-dimensional image generation device according to any one of claims 1 to 8, wherein the parallax image generator uses the other image as a foreground and generates the parallax image from the foreground and the background. The three-dimensional image generating apparatus according to any one of claims 1 to 10, further comprising a lenticular printing data generation unit that generates lenticular printing data based on the parallax image. receiving a two-dimensional image having a background and a plurality of objects ;
a step of extracting the plurality of subjects other than the background of the two-dimensional image by performing image processing on the two-dimensional image;
determining a main subject and a non-main subject among the plurality of subjects based on subject-related information;
The non-main subject in the two-dimensional image is eliminated, the main subject is set as the foreground, the non-main subject is set as the background, and the foreground and the background are relatively shifted to generate two or more parallax images. and
A three-dimensional image generation method comprising: receiving a two-dimensional image having a background and a plurality of objects ;
a step of extracting the plurality of subjects other than the background of the two-dimensional image by performing image processing on the two-dimensional image;
determining a main subject and a non-main subject among the plurality of subjects based on subject-related information;
The non-main subject in the two-dimensional image is eliminated, the main subject is set as the foreground, the non-main subject is set as the background, and the foreground and the background are relatively shifted to generate two or more parallax images. and
A program that causes a computer to execute a three-dimensional image generation process including. A non-transitory computer-readable recording medium, wherein when instructions stored on the recording medium are read by a computer,
receiving a two-dimensional image having a background and a plurality of objects ;
a step of extracting the plurality of subjects other than the background of the two-dimensional image by performing image processing on the two-dimensional image;
determining a main subject and a non-main subject among the plurality of subjects based on subject-related information;
The non-main subject in the two-dimensional image is eliminated, the main subject is set as the foreground, the non-main subject is set as the background, and the foreground and the background are relatively shifted to generate two or more parallax images. and
A recording medium that causes a computer to execute a three-dimensional image generation process comprising:

Images