JP2013206035A - Image processing device, imaging device, and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique capable of easily acquiring the three-dimensional shape of a subject with a small operation amount.SOLUTION: An image processing device comprises: storage means that stores figure information on a three-dimensional shape of each of a plurality of basic figures; feature amount extraction means that extracts an image feature amount of a subject from at least one captured image obtained by capturing the subject; and shape acquisition means that obtains a three-dimensional shape of the subject on the basis of the image feature amount and the figure information.

Description

  The present invention relates to an image processing device, an imaging device, and an image processing program.

  2. Description of the Related Art Conventionally, a technique for generating and acquiring a stereoscopic image of a subject using images obtained by imaging the subject from a plurality of different directions has been developed.

  For example, by using a multi-resolution singularity filter and image matching processing, a plurality of images captured at a plurality of positions around the subject are combined to obtain and display an image of the subject viewed from an arbitrary direction. There is a technology (see, for example, Patent Document 1).

JP 2002-95011 A

  However, in the prior art, it is necessary to perform complicated processing such as a multi-resolution singularity filter and image matching processing on a plurality of images, which takes time.

  Further, for example, to generate data such as three-dimensional CAD and computer graphics (CG) based on the obtained image, the amount of data is enormous and the usability is poor.

  SUMMARY OF THE INVENTION In view of the above-described problems of the prior art, an object of the present invention is to provide a technique that can easily acquire a three-dimensional shape of a subject with a small amount of calculation.

  In order to solve the above-described problem, an aspect of the image processing apparatus illustrating the present invention includes a storage unit that stores graphic information of a three-dimensional shape of each of a plurality of basic figures, and at least one captured image obtained by imaging a subject. Feature amount extraction means for extracting the image feature amount of the subject, and shape acquisition means for obtaining the three-dimensional shape of the subject based on the image feature amount and graphic information.

  Further, the shape acquisition means deforms at least one basic figure based on the image feature amount and the figure information and matches it with the subject, and based on the degree of similarity between the deformed and matched basic figure and the subject. Thus, the three-dimensional shape of the subject may be obtained.

  In addition, the shape obtaining unit obtains the surface and the arrangement of the surface forming the subject using the image feature amount, and deforms and combines at least one basic figure based on the image feature amount and the obtained arrangement of the surface. But you can.

  In addition, an image extraction unit that extracts a partial image of an image region corresponding to a surface from the captured image as a surface pattern is provided, and the shape acquisition unit is based on the partial image extracted by the image extraction unit when there are a plurality of captured images. Thus, it may be determined whether the surface in one captured image is the same as the surface in the other captured image, and the arrangement of the surfaces forming the subject may be obtained.

  Further, the shape acquisition means may use the basic figure having the highest degree or a combination thereof as the three-dimensional shape of the subject.

  In addition, an operation member that receives an instruction from the user and a display unit are provided, and the shape acquisition unit selects a basic figure that is a candidate for the three-dimensional shape of the subject or a combination thereof according to the degree from among a plurality of basic figures. The basic figure displayed on the display means and selected by the operation of the operation member by the user or a combination thereof may be the three-dimensional shape of the subject.

  In addition, imaging information at the time of imaging of the subject may be added to the captured image, and the shape acquisition unit may calculate the size of the subject based on the imaging information.

  One aspect of an imaging apparatus illustrating the present invention includes an imaging unit that captures an image of a subject and generates a captured image, and the image processing apparatus of the present invention.

  Moreover, you may provide the information addition means to add the imaging information at the time of imaging of a to-be-photographed object to a captured image.

  One aspect of the image processing program exemplifying the present invention is a feature amount extraction procedure for extracting an image feature amount of a subject from at least one captured image obtained by imaging the subject, the image feature amount and a three-dimensional shape of each of a plurality of basic figures. Based on the graphic information, the computer is caused to execute a shape acquisition procedure for obtaining the three-dimensional shape of the subject.

  According to the present invention, the three-dimensional shape of a subject can be easily acquired with a small amount of calculation.

1 is a block diagram showing a configuration of a digital camera according to an embodiment. 7 is a flowchart showing processing operations of a digital camera according to an embodiment. The figure which shows an example of the still image which imaged the building as a photographic subject The figure which shows an example of fitting of a basic figure to the building shown in FIG. The figure which shows the example which converted into the image which looked at the partial image of each surface of the building shown in FIG. 3 from the front. The figure which shows the example which displays the reference image of the subject's three-dimensional shape candidate on a monitor in descending order of similarity The flowchart which shows the processing operation of the digital camera which concerns on other embodiment. The figure which shows an example of the still image which each imaged the building shown in FIG. 3 from two different directions, respectively The figure which shows an example of the still image which imaged another building as a subject

<< One Embodiment >>
FIG. 1 is a block diagram showing a configuration of a digital camera according to an embodiment of the present invention.

  The digital camera of the present embodiment includes an imaging optical system 11, an imaging device 12, an AFE 13, an image processing unit 14, a memory 15, a monitor 16, a recording I / F 17, a CPU 19, an operation member 23, a release button 24, a touch panel 25, and a bus. Composed. The image processing unit 14, the memory 15, the monitor 16, the recording I / F 17, and the CPU 19 are connected to each other via a bus so that information can be transmitted. The operation member 23, the release button 24, and the touch panel 25 are each connected to the CPU 19.

  The imaging optical system 11 includes a plurality of lens groups including a zoom lens and a focusing lens. The lens position of the imaging optical system 11 is adjusted in the optical axis direction by a lens driving unit (not shown). For simplicity, the imaging optical system 11 is illustrated as a single lens in FIG.

  The image pickup device 12 is a device that picks up an image of an object scene formed by a light beam that has passed through the image pickup optical system 11. The output of the image sensor 12 is input to the AFE 13. Note that the image sensor 12 of the present embodiment may be a progressive scan type solid-state image sensor (CCD or the like) or an XY address type solid-state image sensor (CMOS or the like).

  A plurality of light receiving elements are arranged in a matrix on the light receiving surface of the image sensor 12. In each light receiving element of the image sensor 12, red (R), green (G), and blue (B) color filters are arranged according to a known Bayer array. Therefore, each light receiving element of the imaging element 12 outputs an image signal corresponding to each color by color separation in the color filter. Thereby, the image sensor 12 can acquire a color image.

  Here, in the photographing mode of the digital camera, the image pickup device 12 picks up a recorded image (main image) in response to a full pressing operation of the release button 24. Further, the imaging element 12 in the shooting mode captures a composition confirmation image (through image) at a predetermined time interval even during shooting standby. The through image data is output from the image sensor 12 by thinning-out readout. The through image data is used for image display on the monitor 16 and various arithmetic processes by the CPU 19.

  The AFE 13 is an analog front end circuit that performs analog signal processing on the output of the image sensor 12. The AFE 13 performs correlated double sampling, image signal gain adjustment, and image signal A / D conversion. The output of the AFE 13 is connected to the image processing unit 14. In the present embodiment, the imaging device 12 and the AFE 13 constitute an imaging unit.

  The image processing unit 14 performs various types of image processing (for example, color interpolation processing, gradation conversion processing, contour enhancement processing, white balance adjustment) on the digital image signal for one frame.

  The memory 15 stores image data captured by the digital camera, a control program executed by the CPU 19, an image processing program, and the like. In addition, the memory 15 of the present embodiment stores in advance graphic information regarding the three-dimensional shape of each of a plurality of basic figures such as a triangular pyramid, a rectangular parallelepiped, a cylinder, and a sphere as a reference for obtaining the three-dimensional shape of the subject of the main image. Yes. Further, the memory 15 temporarily stores image data in the pre-process and post-process of image processing by the image processing unit 14. The memory 15 can be a non-volatile semiconductor memory or the like.

  The monitor 16 is a display means such as a liquid crystal monitor, and displays various images in accordance with instructions from the CPU 19.

  A connector for connecting the storage medium 18 is formed in the recording I / F 17. The recording I / F 17 executes data writing / reading with respect to the storage medium 18 connected to the connector. The storage medium 18 is composed of a hard disk, a memory card incorporating a semiconductor memory, or the like. In FIG. 1, a memory card is shown as an example of the storage medium 18.

  The CPU 19 is a processor that comprehensively controls each unit of the digital camera. By executing the control program, the CPU 19 performs known AF control by a phase difference detection method or a contrast detection method, automatic exposure (AE) calculation, auto white balance calculation, and the like based on the data of the through image. In addition, the CPU 19 performs various arithmetic processes in the imaging mode, and also performs a file generation process of the captured image, a display process on the setting screen, and the like. Furthermore, the CPU 19 of this embodiment operates as a feature amount extraction unit 20, a shape acquisition unit 21, and an image extraction unit 22 by executing the image processing program.

  The feature amount extraction unit 20 applies known vertex extraction processing, edge extraction processing, and the like to the main image to extract image feature amounts such as the vertex and edge amount of the subject.

  The shape acquisition unit 21 obtains the three-dimensional shape of the subject based on the image feature amount of the subject extracted by the feature amount extraction unit 20 and the graphic information of the basic figure. First, the shape acquisition unit 21 obtains the contour of the subject, the surface that forms the three-dimensional shape, and the arrangement thereof based on the vertex position and edge amount distribution extracted by the feature amount extraction unit 20. The shape acquisition unit 21 changes the ratio of the vertical, horizontal, and height of the basic figure based on the obtained vertex of the subject, the outline, and the arrangement of the surface, and scales, shifts, and rotates the basic figure to deform it. And adjust to the subject. In other words, the shape acquisition unit 21 deforms the basic figure to fit the subject so that the placement of the vertex, contour, and surface projected on the main image plane of the transformed basic figure is the same as the subject placement. .

  For example, the shape acquisition unit 21 according to the present embodiment has a similarity degree that is a degree of similarity according to the amount of deviation in the vertex, outline, and surface of the basic figure after deformation projection and the subject, as an index indicating that they are the same. Is used. The shape acquisition unit 21 calculates the similarity of the maximum output of the evaluation function for each basic figure, and obtains the three-dimensional shape of the subject based on the similarity value.

  The image extraction unit 22 extracts a partial image of the image area of the main image corresponding to the surface obtained by the shape acquisition unit 21 as a pattern of the surface. The image extraction unit 22 converts the partial image of each surface into image data (hereinafter referred to as reference image data) when viewed from the front, and temporarily stores it in an internal memory (not shown) of the CPU 19. Note that the conversion to the reference image data by the image extraction unit 22 is performed based on, for example, the arrangement of the vertices and contours of the surface. In addition, the pattern in the present embodiment deals with a sign placed on the surface of the subject, a logo mark, and the like, as well as a window arranged on the surface (wall) of the building.

  The operation member 23 includes, for example, a command dial, a cross-shaped cursor key, a determination button, and the like. The operation member 23 receives various inputs of the digital camera from the user. For example, the operation member 23 is used for an operation for switching the operation mode of a digital camera, an input operation on a setting screen, and the like.

  The release button 24 receives from the user an instruction input for starting an AF operation before imaging by a half-press operation and an instruction input for starting an imaging operation by a full-press operation.

  The touch panel 25 constitutes a part of the operation member 23, detects the position of a stylus (or a fingertip or the like) in contact with the panel surface, and outputs the detected position information to the CPU 19 to accept an instruction input from the user. The configuration of the touch panel 25 may be either a capacitance type or a pressure sensitive type. The touch panel 25 of the present embodiment is composed of a transparent panel having the same shape as the monitor 16 and is stacked on the entire surface of the monitor 16. The user inputs an instruction to the CPU 19 via the touch panel 25 while visually recognizing the GUI button displayed on the monitor 16.

  Next, the processing operation by the digital camera of this embodiment will be described with reference to the flowchart of FIG. Note that the digital camera of the present embodiment has a still image mode and a moving image mode having a single shooting mode and a continuous shooting mode as shooting modes. Further, the digital camera has an operation mode of processing for obtaining the three-dimensional shape of the subject from the main image as a pseudo three-dimensional mode.

  The digital camera is assumed to be set in advance to the still image mode of the single shooting mode as the shooting mode. For example, a single still image (main image) obtained by capturing an object of the building 30 as shown in FIG. The process used will be described. The same operation is performed for subjects other than the building 30.

  The CPU 19 receives a power-on instruction by turning on the power button on the operation member 23 by the user, and turns on the digital camera. The CPU 19 initializes the digital camera. The CPU 19 causes the image sensor 12 to capture a through image at a frame rate such as 30 fps, and causes the monitor 16 to display the through image. The CPU 19 starts processing from step S101.

  Step S101: The CPU 19 accepts an imaging instruction by a user fully pressing the release button 24, and causes the imaging device 12 to image the building 30 in the still image mode of the single shooting mode. The image processing unit 14 performs various image processes on the digital image signal output from the AFE 13. The CPU 19 generates an image file in which the image data subjected to the image processing is a still image 40. The CPU 19 records the image file of the still image 40 on the storage medium 18 or the like.

  Note that the CPU 19 of the present embodiment serves as information addition means for generating image files such as exposure conditions and the like, information on the AF area, and the lens movement amount of the imaging optical system 11 by a lens driving unit (not shown). Metadata that conforms to the exchangeable image file format for digital still cameras (Exif) standard in which information is written is added to the header area of the image file.

  Step S102: The CPU 19 accepts a setting change instruction from the shooting mode to the pseudo three-dimensional mode based on the operation of the operation member 23 and the touch panel 25 by the user. The CPU 19 reads the image file of the still image 40 captured in step S101 from the storage medium 18 or the like. Note that the CPU 19 corrects aberrations in advance for the read still image 40 or adjusts brightness, contrast, hue, and the like so that the image processing unit 14 can easily perform processing by the feature amount extraction unit 20 or the like. It is preferable to do.

  Step S103: The feature amount extraction unit 20 applies a known vertex extraction process, an edge extraction process, or the like to the still image 40 read in step S102, and extracts the vertex and edge amount of the building 30.

  Step S104: The shape acquisition unit 21 obtains a surface on which the building 30 is formed using the vertices 31a to 31f and the edge amount of the building 30 extracted in step S103. Specifically, the shape acquisition unit 21 obtains the contour 32a from, for example, the distribution of edge amounts connecting the vertices 31a and 31b extracted by the feature amount extraction unit 20. Similarly, the shape acquisition unit 21 obtains other contours 32b to 32g. The shape acquisition part 21 calculates | requires the surfaces 33 and 34 which form the building 30, and its arrangement | positioning based on arrangement | positioning of vertex 31a-31f and outline 32a-32g.

  Step S105: The image extraction unit 22 extracts partial images in the image regions of the surfaces 33 and 34 from the still image 40, and converts them into reference image data when viewed from the front (FIG. 5). The CPU 19 temporarily records the reference image data of the surfaces 33 and 34 in an internal memory (not shown).

  Step S106: The shape obtaining unit 21 obtains the three-dimensional shape of the building 30 based on the vertices 31a to 31f and the surfaces 33 and 34 obtained in Step S104 and the graphic information of the basic figure. That is, the shape acquisition unit 21 calculates the ratio of the vertical, horizontal, and height of the basic figure, for example, the triangular prism 50 (dotted line) shown in FIG. 4 based on the arrangement of the vertices 31a to 31f and the surfaces 33 and 34 of the building 30. It is deformed by changing, enlarging / reducing, shifting, or rotating, and fitting to the building 30 is performed until the maximum output similarity value by the evaluation function is obtained in the triangular prism 50. The shape acquisition unit 21 stores the ratio of the vertical, horizontal, and height, the enlargement ratio (or reduction ratio), the shift amount, the rotation angle, and the like of the triangular prism 50 at the time when the maximum output similarity value is obtained. Record in the figure).

  In the present embodiment, the shape behind the building 30 is unknown because only one still image 40 obtained by capturing the building 30 from one direction is used. Therefore, the shape acquisition unit 21 of the present embodiment adjusts each basic figure to the building 30 based on only the vertices 31a to 31f and the surfaces 33 and 34 while the shape behind the building 30 remains unknown.

  Step S107: As shown in FIG. 6, the shape acquisition unit 21 sets the plurality of basic graphics as candidates for the three-dimensional shape of the building 30 in descending order of the similarity value obtained in Step S106. Displayed on the monitor 16. FIG. 6 shows an example in which four basic figures A to D are displayed in descending order of the similarity value, but a number other than four basic figures may be displayed.

  Step S108: The CPU 19 receives a control signal for selecting one of the basic figures A to D from the user via the operation member 23 or the touch panel 25, and the shape acquisition unit 21 determines that the selected basic figure is the building 30. It is assumed that the three-dimensional shape. The CPU 19 generates a three-dimensional CAD or CG object file of the building 30 based on the graphic information of the selected basic graphic. The CPU 19 stores the generated object file in the storage medium 18 or the like, and ends a series of processes.

  The shape acquisition unit 21 may include the reference image data of the surfaces 33 and 34 extracted by the image extraction unit 22 when generating the file. Thereby, for example, the user can easily create a CG image of the building 30.

  Further, the CPU 19 determines the size of the building 30 based on the AF area information and the lens movement amount of the imaging optical system 11 by a lens driving unit (not shown) among the imaging information added to the still image 40. For example, the lengths of the contours 32a to 32g) may be calculated, and the shape acquisition unit 21 may include the size information of the building 30 in the object file or the like. In other words, for example, by capturing images of furniture and refrigerators seen in a store as a subject and generating object files of 3D CAD and CG, the user appropriately attaches furniture and refrigerators to his / her room or kitchen in the virtual space. It can be easily arranged at a small scale, and the atmosphere when actually purchased and arranged can be confirmed.

  As described above, in this embodiment, only a single still image is used by matching a basic figure to a subject based on the image feature quantity of the subject and making the most similar basic figure a solid shape of the subject. Even in this case, the three-dimensional shape of the subject can be easily acquired with a small amount of calculation.

In addition, an object file such as a three-dimensional CAD or CG of the three-dimensional shape of the subject can be easily generated based on the graphic information of the most similar basic figure.
<< Other embodiments >>
A digital camera according to another embodiment of the present invention is the same as the digital camera according to one embodiment shown in FIG. Therefore, the same reference numerals are given to the components constituting the digital camera of the present embodiment, and detailed description thereof is omitted.

  On the other hand, the difference between the digital camera according to the present embodiment and that according to the first embodiment is that the shape acquisition unit 21 (1) obtains the three-dimensional shape of the subject using a plurality of still images, and (2) each The object is to obtain the positional relationship between the surfaces forming the subject imaged in the image based on the reference image data.

  A processing operation performed by the digital camera of this embodiment will be described with reference to the flowchart of FIG. In the present embodiment, it is assumed that the plurality of still images are two still images, and the subject is the building 30 shown in FIG. 3 as in the first embodiment.

  The CPU 19 receives a power-on instruction by turning on the power button on the operation member 23 by the user, and turns on the digital camera. The CPU 19 initializes the digital camera. The CPU 19 causes the image sensor 12 to capture a through image at a frame rate such as 30 fps, and causes the monitor 16 to display the through image. The CPU 19 starts processing from step S201.

  Step S201: The CPU 19 accepts an imaging instruction by a user fully pressing the release button 24 twice, and causes the image sensor 12 to image the building 30 twice in the still image mode of the single shooting mode. The image processing unit 14 performs various types of image processing on the digital image signal output from the AFE 13 for each imaging. The CPU 19 writes the processed image data as a still image in the data area and, along with the exposure conditions and the like, information on the AF area and imaging information such as the lens movement amount of the imaging optical system 11 by a lens driving unit (not shown). An image file is generated in which the metadata conforming to the written Exif standard is added to the header area. The CPU 19 records the generated two image files on the storage medium 18 or the like.

  FIG. 8 shows an example of two still images 60 and 65. The two still images 60 and 65 are images of the building 30 taken from two different directions. That is, the still image 60 is an image of the building 30 taken obliquely from above in the same direction as in FIG. 3, and the still image 65 is the building 30 taken obliquely from above in a direction opposite to the still image 60 by 180 °. It is what was imaged.

  Step S202: The CPU 19 accepts a setting change instruction from the shooting mode to the pseudo three-dimensional mode based on the operation of the operation member 23 and the touch panel 25 by the user, similarly to step S102 of the embodiment. The CPU 19 reads the image files of the still images 60 and 65 captured in step S201 from the storage medium 18 or the like.

  Step S203: The feature amount extraction unit 20 applies the known vertex extraction processing and edge extraction processing to the still images 60 and 65 read in step S202, respectively, and extracts the vertex and edge amount of the building 30.

  Step S204: The shape acquisition unit 21 uses the vertices 70a to 70g and the edge amount of the building 30 extracted from the still image 60, as in step S104 of the embodiment, and the surfaces 61 to 63 that form the building 30. And its placement. Similarly, the shape acquisition unit 21 uses the vertices 75a to 75g and the edge amount of the building 30 extracted from the still image 65 to obtain the surfaces 66 to 68 and their arrangement.

  Step S205: The image extraction unit 22 extracts partial images in the image regions of the surfaces 61 to 63 and 66 to 68 from the still images 60 and 65, respectively, and converts them into reference image data when viewed from the front. The CPU 19 temporarily records the reference image data of the surfaces 61 to 63 and 66 to 68 in an internal memory (not shown).

  Step S206: The shape acquisition unit 21 obtains an arrangement relationship between the surfaces 61 to 63 of the building 30 in the still image 60 and the surfaces 66 to 68 in the still image 65 based on the reference image data of each surface. For example, the shape acquisition unit 21 performs correlation processing with reference image data of the surface 61 of the still image 60 and the surface 66 of the still image 65, and determines whether the surface 61 and the surface 66 are the same surface. As illustrated in FIG. 8, the shape acquisition unit 21 determines that the surface 63 and the surface 68 corresponding to the roof are the same surface, and determines that the surfaces 61, 62, 66, and 67 are different surfaces. Based on this determination result, the shape acquisition unit 21 determines that the surfaces 61, 62, 66, and 67 are arranged around the surface 63 (68) of the building 30, respectively.

  Step S207: The shape acquisition unit 21 determines the three-dimensional shape of the building 30 based on the arrangement of the vertices 70a to 70g and 75a to 75g and the surfaces 61 to 63 and 66 to 68 obtained in Step S206 and the graphic information of the basic figure. Ask for. That is, the shape acquisition unit 21 converts each basic figure into its basic figure based on the arrangement of the vertices 70a to 70g and 75a to 75g and the surfaces 61 to 63 and 66 to 68 of the building 30 as in step S106 of the embodiment. By changing the ratio of length, width, height, scaling, shifting, rotating, etc., it is deformed and adjusted to the building 30 until the maximum output similarity value by the evaluation function is obtained for each basic figure. Do.

  Step S208: The shape acquisition unit 21 converts the plurality of basic figures into the three-dimensional structure of the building 30 in descending order of the similarity value obtained in Step S207 from among the plurality of basic figures as in Step S107 of the embodiment. The shape candidates are displayed on the monitor 16 as shown in FIG.

  Step S209: The CPU 19 receives a control signal for selecting one of a plurality of candidate basic figures from the user via the operation member 23 or the touch panel 25, and the shape acquisition unit 21 determines that the selected basic figure is a building. It is assumed that there are 30 solid shapes. The CPU 19 generates a three-dimensional CAD or CG object file of the building 30 based on the graphic information of the selected basic graphic. The CPU 19 stores the generated object file or the like in the storage medium 18 or the like, and ends a series of processes.

  For example, when the shape acquisition unit 21 determines in step S206 that the three-dimensional shape of the building 30 is a rectangular parallelepiped based on the positional relationship between the surfaces 61, 62, 63 (68), 66, and 67, the rectangular parallelepiped. It is possible to fit the building 30 with only the basic figure, and omit the processing in step S208 and the selection by the user in step S209.

  As described above, in this embodiment, the basic figure is combined with the subject based on the image feature quantity of the subject, and the most similar basic figure is set as the three-dimensional shape of the subject. Can be easily obtained.

  In addition, even when obtaining the three-dimensional shape of a subject based on a plurality of images, whether or not the surfaces captured in different images are the same surface based on the partial images of the surfaces without performing alignment between the images. It is possible to obtain the three-dimensional shape of the subject simply by determining whether or not.

In addition, an object file such as a three-dimensional CAD or CG of the three-dimensional shape of the subject can be easily generated based on the graphic information of the most similar basic figure.
<< Additional items of embodiment >>
(1) In the above-described embodiment, an example in which the CPU 20 implements each process of the feature amount extraction unit 20, the shape acquisition unit 21, and the image extraction unit 22 by software has been described, but each of these processes is performed using an ASIC. May be realized in hardware.

  (2) The image processing apparatus of the present invention is not limited to the example of the digital camera of the above embodiment. For example, the computer may function as the image processing apparatus of the present invention by causing the computer to read an image and causing the computer to execute an image processing program.

  In addition, the computer is placed on a network, an image transmitted from an external user is received via the network, a three-dimensional shape of a subject imaged in the image is obtained, and the result is obtained by 3D CAD or CG. The present invention is also applicable to an online service that is sent back to the external user as an object file or the like. In this case, the image transmitted from an external user is added with imaging conditions such as the AF area information and the lens movement amount of the imaging optical system 11 by a lens driving unit (not shown) together with the exposure condition. Preferably it is.

  (3) In the above embodiment, the three-dimensional shape of the subject is obtained using a still image, but the present invention is not limited to this, and a through image or a moving image may be used.

  (4) In the above embodiment, at least two surfaces of the still image building 30 shown in FIG. 3 or FIG. 8 are captured, but the present invention is not limited to this, and only one surface may be captured. .

  (5) In the above embodiment, the basic figures that are candidates for the three-dimensional shape of the subject are displayed on the monitor 16 in descending order of the similarity value in step S107 or step S208, but the present invention is not limited to this. For example, the shape acquisition unit 21 may set the basic figure having the highest similarity as the solid shape of the subject and display only the basic figure on the monitor 16.

  (6) In the above embodiment, the subject is fitted with one basic figure, but the present invention is not limited to this, and a combination of two or more basic figures may be used to obtain the three-dimensional shape of the subject. .

  For example, the shape acquisition unit 21 may align the basic mark such as a circle or a rectangle as a component of the building 30 instead of a pattern such as a window of the building 30 or a logo of a signboard as a pattern, and paste it on the corresponding surface. .

  For example, as shown in FIG. 9, in the case of a still image 80 obtained by imaging a building 30 ′ in which a columnar tower is additionally arranged on the roof of the building 30, the shape acquisition unit 21 is a tower above the building 30 ′. It is also possible to divide into the parts of the building and the lower part of the building, and perform the fitting with the basic figure for each part.

  Alternatively, the shape acquisition unit 21 first performs alignment with a large frame of the building 30 ′ with one basic graphic as in the above embodiment. In step S107 or step S208, when one basic graphic is selected by the user, for example, the CPU 19 displays a message dialog such as “Do you want to fit in more detail?” On the monitor 16. When the control signal for agreeing to the message is received by the user via the operation member 23 or the touch panel 25, the CPU 19 proceeds to step S106 or step S207. The shape acquisition unit 21 determines that when the building 30 'is combined with the selected basic figure, the tower part having a low similarity is composed of another basic figure, and the upper tower and the lower building And divided into parts. The shape acquisition unit 21 may perform alignment with the basic figure for each part again. In addition, about the lower building, it is preferable to match | combine with the basic figure selected first.

  Moreover, it is preferable that the shape acquisition part 21 displays the order of the candidate of the solid shape of building 30 'in order of the combination with the largest sum of the value of the similarity in the basic figure matched about each part, for example. .

  (7) In the above embodiment, the CPU 19 generates a three-dimensional CAD or CG object file of the subject based on the graphic information of the selected basic graphic, but the present invention is not limited to this. For example, the CPU 19 may generate an image file in which a subject is projected from an arbitrary viewpoint direction based on the selected basic figure.

  (8) In the above embodiment, the shape behind the object that has not been imaged is left unclear, and the basic figure is combined based on the vertex, outline, and surface arrangement of the object, but the present invention is not limited to this. . For example, the shape acquisition unit 21 records information on which basic figure is selected or tends to be selected by the user according to the vertex, outline, and surface arrangement of the subject, so that Even when the shape behind is unknown, a more optimal basic figure may be displayed on the monitor 16 as a candidate for the three-dimensional shape of the subject based on the above information together with the similarity.

  (9) In the above embodiment, the subject size information is obtained based on the AF area information as the imaging information, the lens movement amount of the imaging optical system 11 by the lens driving unit (not shown), and the like. The invention is not limited to this, and a value input by the user via the operation member 23 or the touch panel 25 may be used as subject size information.

  From the above detailed description, features and advantages of the embodiments will become apparent. It is intended that the scope of the claims extend to the features and advantages of the embodiments as described above without departing from the spirit and scope of the right. Further, any person having ordinary knowledge in the technical field should be able to easily come up with any improvements and modifications, and there is no intention to limit the scope of the embodiments having the invention to those described above. It is also possible to use appropriate improvements and equivalents within the scope disclosed in.

DESCRIPTION OF SYMBOLS 11 ... Imaging optical system, 12 ... Image sensor, 13 ... AFE, 14 ... Image processing part, 15 ... Memory, 16 ... Monitor, 17 ... Recording I / F, 18 ... Storage medium, 19 ... CPU, 20 ... Extraction of feature-value , 21 ... shape acquisition part, 22 ... image extraction part, 23 ... operation member, 24 ... release button, 25 ... touch panel

Claims (10)

Storage means for storing three-dimensional figure information of each of the plurality of basic figures;
Feature quantity extraction means for extracting an image feature quantity of the subject from at least one captured image obtained by imaging the subject;
A shape acquisition means for obtaining a three-dimensional shape of the subject based on the image feature and the graphic information;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The shape acquisition means deforms at least one basic figure based on the image feature quantity and the graphic information to fit the subject, and deforms and fits the basic figure and the subject. An image processing apparatus that obtains a three-dimensional shape of the subject based on a degree of similarity. The image processing apparatus according to claim 2,
The shape acquisition means uses the image feature amount to determine a surface forming the subject and the arrangement of the surface, and based on the image feature amount and the determined arrangement of the surface, the at least one basic figure An image processing apparatus characterized by deforming and combining. The image processing apparatus according to claim 3.
Image extraction means for extracting a partial image of an image region corresponding to the surface from the captured image as a pattern of the surface;
In the case where there are a plurality of the captured images, the shape acquisition unit determines whether a surface in one captured image is the same as a surface in another captured image based on the partial image extracted by the image extraction unit. An image processing apparatus comprising: determining and determining an arrangement of a surface on which the subject is formed. The image processing apparatus according to any one of claims 2 to 4,
The image processing apparatus according to claim 1, wherein the shape acquisition means sets the basic figure having the highest degree or a combination thereof as the three-dimensional shape of the subject. The image processing apparatus according to any one of claims 2 to 4,
An operation member for receiving an instruction from the user;
Display means,
The shape acquisition means displays, on the display means, a basic figure that is a candidate for the three-dimensional shape of the subject, or a combination thereof, among the plurality of basic figures, and operates the operation member by the user. An image processing apparatus characterized in that the basic figure selected by (2) or a combination thereof is a three-dimensional shape of the subject. The image processing apparatus according to any one of claims 1 to 6,
Imaging information at the time of imaging the subject is added to the captured image,
The shape processing means calculates the size of the subject based on the imaging information. An imaging unit that images a subject and generates a captured image;
An image processing apparatus according to any one of claims 1 to 8,
An imaging apparatus comprising: The imaging device according to claim 8,
An image pickup apparatus comprising: information addition means for adding image pickup information at the time of image pickup of the subject to the picked-up image. A feature amount extraction procedure for extracting an image feature amount of the subject from at least one captured image obtained by imaging the subject;
A shape acquisition procedure for obtaining the three-dimensional shape of the subject based on the image feature amount and the graphic information of the three-dimensional shape of each of the plurality of basic figures;
An image processing program for causing a computer to execute.

Images