JP2018042106A - Image output device, image output method, and imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image output device capable of, when representing multiple subjects that are photographed in a picked-up image, in a stereoscopic manner, efficiently representing the multiple subjects in the stereoscopic manner within a range of an upper limit value for thickness of a device that outputs a stereoscopic image.SOLUTION: The present invention relates to an image output device 100 comprising: acquisition means for acquiring image data including multiple subjects; generation means 111 for generating depth information corresponding to distances of the multiple subjects in the image data; detection means 110 for detecting feature information for each of regions of the multiple subjects in the image data; conversion means 112 for converting the generated depth information into depth information based on the feature information that is detected for each region; and output means 113 for outputting stereoscopic image data based on the image data and the depth information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image output apparatus, an image output method, and an imaging apparatus, and more particularly to an image output apparatus that outputs a stereoscopic image from an input image and distance information.

  2. Description of the Related Art In recent years, 3D printers, which are apparatuses that form a three-dimensional object by laminating recording materials such as resin and gypsum powder, have started to be widely used. Then, using this 3D printer, an apparatus and a system for outputting a three-dimensional relief of a photographed image, that is, a three-dimensional image, based on a photographed image and distance information indicating the depth have been provided. In addition to 3D printers, there are various 3D printer output methods, such as a method of forming a 3D by thermally deforming a thermally expandable sheet, embossing printing, embossing printing, and laser engraving. ing. In any of the output methods, there is a fear that there is an upper limit on the thickness of the stereoscopic image to be output, and that the cost increases because more recording materials are stacked as the thickness is increased. A 3D printer has a large maximum height difference compared to other methods, so that a 3D image can be output as it is, but a large amount of recording material is required.

  Therefore, Patent Document 1 and Patent Document 2 describe a stereoscopic image output method having an upper limit value of the thickness and a stereoscopic image output method when the thickness is compressed. Specifically, Patent Document 1 discloses a method of performing a three-dimensional output so that a difference between the boundaries of a plurality of adjacent subjects is more conspicuous than the actual one. In other words, a method is disclosed in which a three-dimensional output is performed so that the boundary of the person 1 is closer to the front than the actual boundary at the boundary between the person 1 and the rear person 2 in the depth direction. . By increasing the boundary thickness difference, it is possible to emphasize the positional relationship in the depth direction for each subject even when the thickness is compressed. Further, in Patent Document 2, in a stereoscopic image output method having an upper limit value of thickness, the height of each pixel is divided by the upper limit value of thickness, and the height of the pixel is replaced with the remainder to obtain a stereoscopic image. Is output so that the value falls within the upper limit of the thickness.

JP 2008-213320 A Japanese Patent No. 4998317

  However, the conventional stereoscopic image output device consumes a large amount of recording material when outputting a captured image including a plurality of objects of interest as a stereoscopic image, or is an optimal stereoscopic image within the upper limit of the thickness. Sometimes I couldn't get an expression. For example, in the apparatus of Patent Document 1, the boundary line can be emphasized, but it is difficult to sufficiently express the stereoscopic effect because the stereoscopic effect of the subject itself, that is, the unevenness in the subject is compressed. Further, in the apparatus of Patent Document 2, the area where the height in the z-axis direction changes gradually changes in a sawtooth shape every time it is divisible by the upper limit value of the thickness. If expressed in terms of expression, it will result in a stereoscopic image with a large discomfort.

  In view of the above problems, the present invention efficiently expresses a plurality of subjects within the range of the upper limit value of the thickness of an apparatus that outputs a stereoscopic image when the subjects are captured in a stereoscopic manner. It is an object of the present invention to provide an image output apparatus that can perform the above-described process.

  In order to solve the above-described problem, an image output apparatus according to the present invention includes an acquisition unit that acquires image data including a plurality of subjects, and generation that generates depth information corresponding to the distance between the plurality of subjects in the image data. Means for detecting feature information for each region of the plurality of subjects in the image data; and converting the generated depth information into depth information based on the detected feature information for each region. Conversion means for outputting, and output means for outputting stereoscopic image data based on the image data and the depth information.

  According to the present invention, when a plurality of subjects appearing in a captured image are three-dimensionally represented, an image that can efficiently represent a plurality of subjects three-dimensionally within the range of the upper limit value of the thickness of a device that outputs a three-dimensional image. An output device can be provided.

It is a block diagram of a digital camera with a stereoscopic image data output function of an embodiment. It is the schematic which illustrated the picked-up image, distance information, and three-dimensional output. It is the schematic which illustrated the picked-up image, distance information, and three-dimensional output. It is a flowchart which shows the flow which produces | generates a solid output. It is a block diagram of a distance information conversion part. It is the schematic which illustrated the produced | generated solid output data. It is the schematic which illustrated the solid output data reflecting thickness information. It is the schematic which illustrated the solid output data which reflected the relationship of the characteristic.

  The best mode for carrying out the present invention will be described below with reference to the drawings.

  A configuration of a digital camera with a stereoscopic image data output function according to the present embodiment will be described with reference to FIG. A digital camera (image output device) 100 shown in FIG. 1 is a digital camera body with a stereoscopic image data output function. The internal bus 102 is an internal bus that connects each block. The system control unit 101 controls each block connected by the internal bus 102. The FLASH memory controller 103 reads / writes nonvolatile data from / to the FLASH memory 104 based on a request from the system control unit 101. The system control unit 101 operates based on a program stored in the FLASH memory 104 and stores non-volatile data such as a captured image in the FLASH memory 104. The DRAM controller 105 reads and writes data to and from the DRAM 106 based on requests from each block. The system control unit 101 and each block are used by being stored in the DRAM 106 such as temporary data during operation.

  The operation unit 107 is configured by buttons or the like provided on the exterior of the digital camera, and the user can transmit instructions to the system control unit 101 by operating the buttons. The imaging unit 108 is an imaging unit that photographs a subject. The imaging unit 108 includes a lens that captures an optical image of a subject and an image sensor such as a CCD or a CMOS element that converts the optical image into an electrical signal. The image sensor measures the amount of focus shift, that is, the defocus amount for each area of the captured optical image.

  The image signal processing unit 109 stores the image data obtained by photographing the subject by controlling the imaging unit 108 in the DRAM 106 based on an instruction from the system control unit 101. Simultaneously with the shooting, the defocus amount measured by the imaging unit 108 is stored in the DRAM 106 together with the image data. Further, the image data is reduced for display and stored in a display frame memory provided in the DRAM 106. The feature detection unit (detection unit) 110 detects the face of a person as a target subject from the image data stored in the DRAM 106 and notifies the system control unit 101 of the position and size. The system control unit 101 assigns a score of the subject of interest based on the position and size, and treats the subject of interest with the highest score as the main subject.

  The thickness information acquisition unit 114 acquires thickness information that is an upper limit value of the thickness output by a device that materializes stereoscopic output such as an external 3D printer. The distance information generation unit (generation unit) 111 generates a distance map (depth information) for each area of the image based on the image data stored in the DRAM 106 and the defocus amount, and stores the distance map in the DRAM 106. The distance information conversion unit (conversion unit) 112 converts the distance map into three-dimensional output depth information according to the control of the system control unit 101 based on the result of the feature detection unit 110. The stereoscopic output generation unit (output unit) 113 combines the two pieces of image data stored in the DRAM 106 and the stereoscopic output depth information converted by the distance information conversion unit 112, and outputs them as stereoscopic image data to the outside. By connecting an external 3D printer or the like as an output destination, the output stereoscopic image data can be materialized. The LCD controller 115 reads the contents of the display frame memory stored in the DRAM 106 and displays the contents on an LCD (Liquid Crystal Display) 116.

  The configuration of the digital camera with the stereoscopic image data output function illustrated in FIG. 1 is an example, and the configuration of the digital camera with the stereoscopic image data output function according to the present invention can be performed as long as the operations described below can be performed. Is not limited to the configuration shown in FIG.

  Next, stereoscopic image data output processing of the digital camera 100 with a stereoscopic image data output function will be described. First, a captured image of the digital camera 100 and additional information acquired at the time of shooting will be described with reference to FIGS. Here, the additional information in the present embodiment is a distance map of the captured image and feature information detected from the captured image. FIG. 2A is a schematic view of the situation when an image is taken as seen from the side. The digital camera 100 is a digital camera body, and persons 201 and 202 as subjects are located at different distances d1 and d2 from the digital camera 100. The axis z is the optical axis direction of the camera, and the axis y is an axis perpendicular to the ground. An image 200 illustrated in FIG. 2B is a captured image obtained using the digital camera 100 according to the capturing state illustrated in FIG. In addition to the persons 201 and 202, a tree 203 is shown in the background. The axis x indicates the horizontal direction, and the axis y indicates the vertical axis.

  A distance map 220 illustrated in FIG. 2C is a distance map with respect to the captured image 200 obtained at the time of capturing illustrated in FIG. The distance map 220 is generated by the distance information generation unit 111 using the defocus amount for each area of the image data acquired from the imaging unit 108. In the distance map 220, it is shown that a dark color area is located on the near side and a thin area is located on the back side. The person 201 close to the digital camera 100 is the darkest, and then the person 202 is a little darker.

  FIG. 2D is a diagram in which frames 211 and 212 surround the positions and sizes of the persons 201 and 202 detected by the feature detection unit 110 as the subject of interest with respect to the photographed image 200. The digital camera 100 according to the present embodiment acquires the captured image of FIG. 2B, the distance map of FIG. 2C as additional information, and the feature information of FIG.

  Next, a description will be given of a stereoscopic output process performed based on the captured image and the additional information. FIG. 3 is a diagram illustrating a relationship between a schematic diagram in which a situation when an image is captured is viewed from the side and a three-dimensional output. The distance information conversion unit 112 converts the distance map 220 shown in FIG. 2C into depth information of stereoscopic output. The feature detection unit 110 converts the existence range H1 in the depth direction of the persons 201 and 202 detected as the subject of interest into a three-dimensional range and converts it into three-dimensional output depth information. As shown in FIG. 3, an infinite region far from the person 202 is the background 240, and the persons 201 and 202 have a depth. The three-dimensional output generation unit 113 performs three-dimensionalization based on the captured image 200 shown in FIG. 2B and the depth information of the three-dimensional output converted by the distance information conversion unit 112.

  Next, with reference to FIG. 4, processing for generating stereoscopic output data according to the present embodiment will be described. FIG. 4 is a flowchart illustrating a process in which the three-dimensional output generation unit 113 generates a three-dimensional output based on the control of the system control unit 101. First, in step S <b> 301, the captured image and the defocus amount are stored in the DRAM 106 by the signal captured by the imaging unit 108 via the image signal processing unit 109. Next, in step S302, the distance information generation unit 111 generates a distance map corresponding to the captured image using the defocus amount acquired in step S301. In step S303, the feature detection unit 100 detects the subject to be featured and their positional relationship from the captured image stored in the DRAM 106.

  Next, in step S304, based on the subject detected in step S303 and their positional relationship, the distance information conversion unit 112 converts the depth information into three-dimensional output so as to satisfy the maximum thickness constraint given in advance. . In step S305, the three-dimensional output generation unit 113 performs three-dimensionalization based on the three-dimensional output depth information converted by the distance information conversion unit 112. The three-dimensional output data generated by the three-dimensional output generation unit 113 is stored in the DRAM 106, and the three-dimensional output generation flow ends. The three-dimensional output data generated in this way can be materialized by, for example, an external 3D printer connected by sequentially outputting the output data in a slice form from the background.

  Next, the conversion process of the distance information conversion unit 112 according to the present embodiment will be described in detail with reference to FIG. FIG. 5 is a block diagram illustrating a configuration of the distance information conversion unit 112. As shown in FIG. 5, the distance information conversion unit 112 performs distance map conversion processing using a subject conversion unit 150, a thickness information conversion unit 151, and a distance comparison conversion unit 152.

  Next, the processing of the subject conversion unit 150 will be described with reference to FIG. FIG. 6A is a schematic view of the situation when an image is taken from the side. A depth H <b> 2 illustrated in FIG. 6A indicates a range in the depth direction of the person 201, and a depth H <b> 3 indicates a range in the depth direction of the person 202. In the subject conversion unit 150, the range of the depths H2 and H3 of the persons 201 and 202 detected by the feature detection unit 110 from the distance map 220 of FIG. The depth information of the two three-dimensional outputs is converted. FIG. 6B is a side view of the three-dimensional figures of the persons 201 and 202 using the three-dimensional output depth information converted by the subject conversion unit 150. Using the converted depth information of the three-dimensional output, the depths H2 and H3 are added in the z-axis direction from the background 240 to each of the persons 201 and 202, and are three-dimensionalized. As described above, the subject conversion unit 150 converts the depth map into depth information of a stereoscopic output for each subject detected by the feature detection unit 110.

  The thickness information conversion unit 151 and the distance comparison conversion unit 152 will be described with reference to FIGS. The thickness information conversion unit 151 and the distance comparison conversion unit 152 further perform conversion processing on the depth information of the stereoscopic output converted by the subject conversion unit 150. First, the processing of the thickness information conversion unit 151 will be described with reference to FIG. The thickness information conversion unit 151 performs conversion using the thickness information acquired by the thickness information acquisition unit 114. FIG. 7 is a diagram illustrating a method for converting the depth information of the stereoscopic output when the converted depth information of the stereoscopic output exceeds the upper limit value of the thickness. The thickness information H illustrated in FIG. 7 is the thickness information acquired by the thickness information acquisition unit 114. The distance comparison conversion unit 152 performs conversion so as to be within the constraints of the thickness information H according to the comparison result between the thickness information H and the depth information H3 or H2 of the three-dimensional output. That is, when the depth information H3 or H2 of the three-dimensional output becomes larger than the acquired thickness information H, the depth information is compressed for each person 201, 202 in accordance with the acquired thickness information H, and the thickness information H Conversion is performed so as to fit within the constraints. In the present embodiment, since the depth of the persons 201 and 202 exceeds the upper limit value of the thickness, the thickness information H2 and H3 are compressed so that the depth of the persons 201 and 202 is smaller than the thickness information H. .

  Next, the processing of the distance comparison conversion unit 152 will be described with reference to FIG. The distance comparison / conversion unit 152 performs conversion using the positional relationship in the distance in the depth direction of the feature. FIG. 8 is a diagram illustrating a method of converting the converted depth information of the three-dimensional output by reflecting the positional relationship between the distances in the depth direction of the persons 201 and 202. The distance in the depth direction is the distances d1 and d2 indicating the positions in the depth direction of the persons 201 and 202 shown in FIG. As can be seen from the distances d1 and d2 in FIG. 2A, the person 201 is located at a distance close to the z-axis direction with respect to the person 202. However, in the depth information of the three-dimensional output shown in FIG. 6B, the depth information H2 of the person 201 is located farther than the depth information H3 of the person 202. Therefore, the depth information of the three-dimensional output is converted by reflecting the distances d1 and d2 of the persons 201 and 202 in the depth direction. Three methods for converting the depth information of the stereoscopic output based on the position in the depth direction will be described below.

  In the first conversion method, as shown in FIG. 8A, an offset (offset addition amount) is added to the height difference of the person 201 so that the height difference of the person 201 is larger than the height difference of the person 202 from the background 240. 501 is added, and conversion is performed so as to be larger than the height difference of the person 202. In FIG. 8B, the conversion is performed so that the height difference of the person 202 becomes smaller than that of the person 201 by adding a negative offset to the height difference of the person 202 in contrast to FIG. 8A. In the third part (C) of FIG. 8, the height difference of the person 202 is converted to be smaller than the height difference of the person 201 by changing the compression ratio. By using these three conversions properly or in combination, the positional relationship of the distance for each feature is reflected in the depth information of the three-dimensional output. As for proper use, when the depth information has a margin with respect to the upper limit value of the thickness using the thickness information described above, the method of FIG. 8A is used. On the contrary, when it is close to the upper limit value of the thickness, control is performed so as not to exceed the upper limit value of the thickness by using FIG. 8B or 8C.

  In this embodiment, a 3D printer is cited as a method for outputting stereoscopic output data as an entity, but the same effect can be obtained even when a stereoscopic image is output by another method. In this embodiment, the distance map is generated from the defocus amount for each region obtained from the camera. However, a plurality of images taken at different focal lengths using parallax images obtained by a two-lens imaging system are used. You may produce | generate by another method, such as detecting the area | region for every focal distance from an image. In the present embodiment, the digital camera has a function of capturing an image. However, an externally captured image and its distance map are input and output from a 3D printer or the like that outputs stereoscopic output data. It may be a device or a program on a computer.

  Moreover, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

DESCRIPTION OF SYMBOLS 100 Digital camera 111 Distance information generation part 112 Distance information conversion part 113 Three-dimensional output generation part

Claims (9)

Acquisition means for acquiring image data including a plurality of subjects;
Generating means for generating depth information corresponding to the distances of the plurality of subjects of the image data;
Detecting means for detecting feature information for each region of the plurality of subjects in the image data;
Conversion means for converting the generated depth information into depth information based on the detected feature information for each region;
Output means for outputting stereoscopic image data based on the image data and the depth information;
An image output apparatus comprising: The image output apparatus according to claim 1, wherein the conversion unit converts the depth information so as to satisfy a predetermined maximum thickness constraint. The image output apparatus according to claim 1, wherein the generation unit generates the depth information by using a defocus amount for each region of the image data. Comparing means for comparing depth information for each region of the plurality of subjects with the predetermined maximum thickness,
The image output apparatus according to claim 2, wherein the conversion unit converts the depth information into the depth information according to a comparison result of the comparison unit. The image output apparatus according to claim 4, wherein the conversion unit adds an offset addition amount in the thickness direction for each region according to the result of the comparison unit. The image output apparatus according to claim 4, wherein the conversion unit changes a compression ratio of the distance for each region according to a result of the comparison unit. The image output apparatus according to claim 1, wherein the detection unit detects a human face as the feature information from the image. An acquisition step of acquiring image data including a plurality of subjects;
A generation step of generating depth information corresponding to the distances of the plurality of subjects of the image data;
A detection step of detecting feature information for each region of the plurality of subjects in the image data;
A conversion step of converting the generated depth information into depth information based on the detected feature information for each region;
An output step of outputting stereoscopic image data based on the image data and the depth information;
An image output method comprising:   The program for functioning a computer as each means of the image output device of any one of Claims 1-7.

Images