JP2018032947A - Image processing device and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate an image suitable for copy generation.SOLUTION: An image processing device comprises: normal line acquisition means which acquires normal line information in each region constituting a subject; image acquisition means which acquires plural images obtained by imaging the subject under imaging conditions different from each other; and generation means which, on the basis of the images and the normal line information, generates a correction image constituted so as to meet a predetermined imaging conditions with respect to each region constituting the subject.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an image generation technique used for creating a copy.

  In recent years, it has become widespread to create a copy of a cultural property in order to achieve both the activities of disclosing precious cultural properties and protecting and preserving them in the future. As an example of creating a duplicate, if the cultural property is a Japanese painting such as a barrier painting or folding screen, in order to acquire the image without damaging the cultural property, take a picture with a digital camera and output the captured image in full size The technique of doing is common.

  Patent Document 1 discloses a technique for restoring original color information of a subject from captured images at a plurality of angles. In addition, when a shielding object appears in front of a subject in a captured image at a plurality of angles, color information can be generated by averaging pixel values of image data at a photographing angle at which the shielding object is not reflected. Have been described.

Japanese Patent Laying-Open No. 2015-1792

  By the way, depending on the subject, color information obtained by photographing may differ depending on photographing conditions such as a photographing angle. For this reason, for example, when a work that has undergone deformation such as deflection or warpage due to deterioration over time, a photographed image having a color different from the original color is obtained depending on the deformation. However, the technique described in Patent Document 1 described above cannot cope with such a problem.

  The present invention has been made in view of such a problem, and an object thereof is to provide a technique capable of generating an image suitable for creating a duplicate.

  In order to solve the above-described problems, an image processing apparatus according to the present invention has the following configuration. That is, the image processing apparatus includes a normal acquisition unit that acquires normal information of each region constituting the subject, an image acquisition unit that acquires a plurality of captured images obtained by capturing the subject under a plurality of different shooting conditions, Generating means for generating a corrected image configured to satisfy a predetermined shooting condition for each region constituting the subject based on the plurality of shot images and the normal line information;

  ADVANTAGE OF THE INVENTION According to this invention, the technique which enables the production | generation of an image suitable for replica production can be provided.

It is a figure explaining the various data used by 1st Embodiment. 1 is a block diagram illustrating a configuration of an image processing system according to a first embodiment. 3 is a flowchart illustrating an operation of the image processing apparatus according to the first embodiment. It is a figure explaining normal line information. It is a detailed flowchart of the shape correction process (S3) in a correction | amendment part. It is a figure explaining a shape correction image. It is a figure explaining the corresponding coordinate between a picked-up image and a shape correction image. It is a detailed flowchart of the production | generation process (S4) in a production | generation part. It is a block diagram which shows the structure of the image processing system which concerns on 2nd Embodiment. It is a flowchart which shows operation | movement of the image processing apparatus in 2nd Embodiment.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the drawings. The following embodiments are merely examples, and are not intended to limit the scope of the present invention.

(First embodiment)
As a first embodiment of an image processing apparatus according to the present invention, an image processing apparatus that generates an image based on a photographed image obtained by a photographing unit will be described below as an example.

<System configuration>
FIG. 1 is a diagram illustrating various data used in the first embodiment. The image processing apparatus 101 according to the first embodiment acquires a plurality of photographed image data 3 obtained by photographing the subject 1 at a plurality of photographing angles from the photographing unit 102. Further, the image processing apparatus 101 acquires normal line information 4 indicating the surface shape of the subject 1 from the measurement unit 103. The image processing apparatus 101 generates the restored image data 6 based on the plurality of captured image data 3 and the normal information 4 and outputs the restored image data 6 to the output unit 109.

  Here, as shown in FIG. 1, the photographing unit 102 is in a state where the angle formed by the illumination direction of the subject 1 by the illumination device 2 that is an illumination unit and the photographing direction of the subject 1 by the photographing unit 102 is 45 degrees. Assume that shooting is performed. Of course, the angle is not limited to 45 degrees and may be configured to be another angle.

  The image processing apparatus 101 corrects the image shape using the normal line information 4 for each of the plurality of photographed image data 3 obtained at a plurality of photographing angles (angles A and B). Specifically, projective transformation for correcting the deformation due to the shooting angle and the shape correction of the image depending on the deformation (deflection etc.) of the subject 1 itself are performed. Here, a plurality of shape-corrected image data 5 having the same image size is generated. Then, the image processing apparatus 101 generates restored image data 6 using the normal line information 4 and the plurality of shape correction image data 5.

  FIG. 2 is a block diagram illustrating a configuration of the image processing system according to the first embodiment. The image processing system includes an image processing apparatus 101, a photographing unit 102, a measurement unit 103, and an output unit 109. The image processing apparatus 101 includes an image acquisition unit 104, an information acquisition unit 105, a correction unit 106, a holding unit 107, and a generation unit 108.

  As described above, the image processing apparatus 101 performs processing for acquiring captured image data of a subject and normal information of the subject and outputting restored image data. The photographing unit 102 is, for example, a digital camera that photographs a subject and outputs photographed image data. The measurement unit 103 is a three-dimensional shape measurement camera that acquires surface shape information of a subject and outputs normal information of the subject.

  The image acquisition unit 104 is a functional unit that acquires captured image data captured by the imaging unit 102. The information acquisition unit 105 is a functional unit (normal line acquisition unit) that acquires the normal information of the subject measured by the measurement unit 103. The correction unit 106 is a functional unit that performs shape conversion on a subject image included in captured image data captured at various angles to generate shape corrected image data.

  The holding unit 107 is a functional unit that stores the shape correction image data corrected by the correction unit 106 and the normal information acquired by the information acquisition unit 105. The generation unit 108 is a functional unit that generates two-dimensional restored image data that reproduces the original color of the subject from the normal information and the shape-corrected image data. The output unit 109 is a printing device that prints the restored image data on a recording medium such as paper.

  Note that the image processing apparatus 101 can be realized by a combination of a computer (PC) and a software program, for example. Of course, some or all of the functional units may be realized by hardware such as an ASIC (Application Specific Integrated Circuit).

<Operation of image processing system>
FIG. 3 is a flowchart showing the operation of the image processing apparatus in the first embodiment.

  In step S <b> 1, the image acquisition unit 104 acquires the captured image data 3 captured by the capturing unit 102. The captured image data 3 is, for example, image data in a bitmap format such as TIFF or JPG.

  In step S <b> 2, the information acquisition unit 105 acquires the normal information 4 of the subject 1 measured by the measurement unit 103. The normal line information 4 is data representing the normal direction for each region of the subject 1 created based on the three-dimensional shape of the surface of the subject.

  FIG. 4 is a diagram for explaining normal line information. FIG. 4A is a diagram exemplarily showing the three-dimensional shape data 401 based on the surface shape information of the subject acquired by the measurement unit 103. The three-dimensional shape data 401 is data representing a three-dimensional shape for generating normal line information, for example, wire frame data. The wire frame data represents the shape of the object as a collection of minute planes. That is, normal information can be generated by calculating the normal direction from each of the minute planes.

  FIG. 4B is an enlarged view of one mesh region 402 surrounded by a wire frame on the three-dimensional shape data 401. The normal vector N with respect to the mesh region 402 can be calculated as the following formula (1), where the coordinates of the four points in the mesh region 402 are ABCD.

  By calculating this normal vector in each mesh region, normal information of the entire region can be acquired. Note that the number of meshes is preferably changed in accordance with a shape change such as deflection or warping of the subject. A fine mesh is required at a location where the subject's shape change is large, but rough mesh data may be used as long as there is almost no change in the shape of the subject and it is a flat part.

  For example, the amount of paper curl can be calculated as a curvature value, and the number of normal line information data can be determined according to the curvature value. When the curvature value of the paper curl is 1 / R, the length of 4πR × 5/360 = 0.17R corresponding to the length of the arc of 5 degrees is set as the number of lattice points on the paper surface. In this case, for example, when the curvature value of the paper curl is 10 cm, the interval between the lattice points is 1.7 cm. The format of the solid shape data is not limited to the wire frame. Any data format such as polygon data or surface data may be used as long as the data can read the normal direction of each part.

  FIG. 4C is a conceptual diagram of normal information. The normal line information 403 includes information related to the normal line for each mesh region surrounded by the wire frame of the two-dimensional subject subjected to shape correction. Here, it is assumed that three-dimensional vector information is stored as a normal angle for each mesh region. In addition, here, the mesh shape is described by a square lattice, but the present invention is not limited to this shape, and an arbitrary mesh shape such as a triangle may be used.

  In step S <b> 3, the correction unit 106 corrects the subject shape in the captured image data 3 and stores the shape corrected image data 5 in the holding unit 107. Details of the process of S3 will be described later.

  In step S4, the generation unit 108 generates two-dimensional restored image data 6 based on the normal line information 4 and the plurality of shape-corrected image data 5 so that the colors are observed from the normal direction of each region of the subject. . Details of the process of S4 will be described later. Thereafter, in step S5, the generation unit 108 outputs the generated restored image data 6 to the output unit 109.

<Operation in Correction Unit 106>
FIG. 5 is a detailed flowchart of the shape correction process (S3) in the correction unit. FIG. 6 is a diagram for explaining shape-corrected image data.

  FIG. 6A shows photographed image data and a shape-corrected image for each photographing angle. As shown in the figure, two-dimensional shape-corrected image data having the same number of pixels is generated by correcting the photographing angle and the shape change of the subject with respect to the image data photographed at different photographing angles. That is, the shape-corrected image data is image data that has been transformed so that the normal direction of each region constituting the subject image in the captured image data is the same direction. Here, the mesh data whose normal line shape has been calculated is applied to the captured image data, and corrected image data is generated by performing image conversion for each mesh region as shown in FIG. 6B.

  In step S <b> 501, the correction unit 106 generates mesh data having the same number of regions as the normal information 403 using normal information according to the subject shape of the captured image data I. For example, the mesh data may be generated by restoring from the normal information 403 to the mesh data 401 in the three-dimensional space, and performing projective transformation so that the mesh data is captured at the corresponding imaging angle.

  Note that when the shape change of the subject is simple, the mesh data may be generated without using the normal line information. For example, the edges of the captured image are extracted to identify the contour of the subject, a point cloud corresponding to a mesh lattice is formed so as to be equidistant with respect to the edge line segment of the subject, and the grid points are moved to the subject edge. It is also possible to generate mesh data by drawing straight lines in parallel.

  In step S502, the correction unit 106 secures a data area of the shape correction image data I ′. For example, when the number of vertical and horizontal pixels of the shape-corrected image data I ′ is w pixels and h pixels, w × h pixels where the upper left coordinates are (1, 1) and the lower right coordinates are (w, h). A memory for storing pixel value data is secured.

  In step S503, the correction unit 106 sets the initial value of the coordinate position P ′ for performing the process of generating the pixel value after shape conversion to (1, 1).

In step S504, the correction unit 106 detects the position of the mesh to which the coordinate position P ′ to be processed belongs, and selects points that constitute the mesh. First, the number of meshes generated in S501 in the vertical and horizontal directions is set to Nm_w and Nm_h, respectively. In this case, the vertical and horizontal pixel numbers Pix _w and Pix _h of one mesh data on the shape-corrected image data I ′ are calculated by the following formulas (2) and (3).

  Next, the position of the mesh corresponding to the point P ′ (xp ′, yp ′) on the shape-corrected image data I ′ is defined as the horizontal direction Mw-th with respect to the upper-left direction and the vertical direction Mh-th. (4) and (5) are calculated. Note that int () is a function for deriving an integer value (truncating after the decimal point).

  Based on the results of the mathematical formulas (4) and (5), the points constituting the mesh in the horizontal direction Mw and the vertical direction Mh are selected.

  FIG. 7 is a diagram for explaining the corresponding coordinates between the captured image and the shape-corrected image. 7A shows one mesh area on the captured image data I, and FIG. 7B shows a mesh area in the shape-corrected image data I ′ corresponding to the mesh area in FIG. 7A. Represents. A ′ to D ′ are detected as points constituting the mesh including the processing coordinate P ′.

  In step S505, the correction unit 106 calculates a coordinate position P on the captured image data I corresponding to the coordinate position P ′ being processed. In FIG. 7B, a vector A′P ′ based on the point A ′, which is a re-neighbor point of the point P ′, is calculated as follows using the vector A′B ′ and the vector A′D ′.

The coordinates of the points A ′, B ′, D ′, and P ′ are respectively (x _{A ′} , y _{A ′} ), (x _{B ′} , y _{B ′} ), (x _{D ′} , y _{D ′} ), ( xP _′ , yP _′ ). In that case, the coefficients i and j relating to the vector A′B ′ and the vector A′D ′ are calculated as in the following formulas (7) and (8).

  Using the calculated values of i and j, the vector AP in FIG. 7A is calculated as in the following formula (9).

The coordinates (x _P , y _P ) of the point P can be calculated using the vector AP and the coordinates of the point A according to the equation (9).

  In step S506, the correction unit 106 sets the pixel value at the coordinate position P on the photographed image data I obtained at S505 as the pixel value at the coordinate position P 'on the shape correction image data I'.

  In step S507, the correction unit 106 determines whether the acquisition of color information for all coordinates has been completed. If the acquisition has not been completed, the next coordinates are set in S508, and the processes in S504 to S506 are performed.

  The above processing is performed also on the captured image data at other imaging angles, and the shape correction image data I ′ and the imaging angle data corresponding to each imaging angle are stored in the holding unit 107.

<Operation in Generation Unit 108>
FIG. 8 is a detailed flowchart of the generation process (S4) in the generation unit.

  In step S801, the generation unit 108 secures a memory area for the restored image data I ″. Here, the restored image data I ″ is data obtained by performing color correction on the shape-corrected image data I ′, and has a shape. This is image data having the same number of pixels as the corrected image data I ′.

  In step S <b> 802, the generation unit 108 acquires normal line information corresponding to the coordinate position that is currently focused on. For the acquisition, the same method as that shown in S504 can be used. That is, the position of the mesh area with respect to the coordinate position is acquired, and the normal information for the corresponding mesh area is acquired from the normal information 403 for the mesh area.

  In step S <b> 803, the generation unit 108 confirms whether the holding unit 107 has the shape-corrected image data I ′ under the corresponding predetermined shooting conditions (shooting angle and illumination angle) based on the normal line information. Here, the shape-corrected image data I ′ in which the normal direction of the subject coincides with the direction of the photographing angle and the illumination angle is irradiated in a direction of 45 degrees with respect to the normal direction is searched. If the shape-corrected image data I 'exists, the process proceeds to S804, and if it does not exist, the process proceeds to S805.

  In step S804, the generation unit 108 acquires the pixel value of the coordinate (color information of the corresponding region) in the shape-corrected image data I ′ in the corresponding normal direction, and the pixel of the corresponding coordinate in the restored image data I ″. Note that x and y are coordinate values within the range of the corresponding mesh region.

In step S805, the generation unit 108 derives the pixel value of the restored image data I ″ by using the two shape-corrected image data I ′ in the vicinity of the predetermined photographing condition. Here, the two angles in the vicinity are used. Assuming that the shape-corrected image data corresponding to is I ′ ₁ and I ′ ₂ , pixel value data is calculated as in the following formula (12), where k is a value between 0 and 1.

Note that with respect to Equation (12), the weight value k may be determined by an arbitrary method. The average of the two image data may be set to k = 0.5, and the angle difference between the two adjacent angles N ₁ and N ₂ with respect to the original normal angle N that is the predetermined condition described above. You may determine k from the ratio of these. In addition, two or more shape correction image data I ′ may be used, or interpolation processing is not performed, and only one shape correction image data having the closest normal angle may be used.

  In step S806, the generation unit 108 determines whether the acquisition of color information has been completed for all coordinates in the restored image data I ″ after color correction. If there is an unacquired area, the process returns to S802. If the acquisition of color information at all coordinates is completed, the process ends.

  Note that, in the restored image data I ″, if a subtle color or luminance difference between meshes occurs due to fluctuations in the photographing device or illumination device, and unevenness on the grid is noticeable, the restored image data I after S806. Further image processing may be performed on “. Further, the image processing may be performed on either the shape-corrected image data I ′ or the captured image data I in advance.

  As described above, according to the first embodiment, the restored image data is generated based on the normal information for each region of the subject and a plurality of photographed image data obtained by photographing the subject. Specifically, photographic image data that satisfies a predetermined photographic condition is selected for each region of the subject image. By generating restored image data with this configuration, even when shooting is performed in a state where the subject is deflected or warped, the restored image data having color information that is observed without deflection is generated. It becomes possible.

(Second Embodiment)
In the second embodiment, a mode in which restored image data is generated by determining and shooting for each region a shooting angle suitable for obtaining color information based on the normal information of the subject will be described.

<System configuration>
FIG. 9 is a block diagram illustrating a configuration of an image processing system according to the second embodiment. As in the first embodiment, the image processing system includes an image processing apparatus 101, a photographing unit 102, a measuring unit 103, and an output unit 109. The image processing apparatus 101 includes an image acquisition unit 104, an information acquisition unit 105, a correction unit 106, a holding unit 107, and a generation unit 108. However, the image processing system according to the second embodiment further includes an imaging angle control unit 901 and an illumination angle control unit 902 that perform imaging control in the imaging unit 102.

  The shooting angle control unit 901 is a functional unit that controls the position and orientation of the shooting unit 102 (shooting angle with respect to the subject) based on the normal information of the subject. The illumination angle control unit 902 is a functional unit that controls the position and orientation of the illumination device (the illumination angle with respect to the subject) based on the normal information of the subject.

<Operation of the entire image processing system>
FIG. 10 is a flowchart illustrating the operation of the image processing apparatus according to the second embodiment.

  In step S101, the information acquisition unit 105 acquires normal line information of the subject. The normal information is assumed to be normal information for each mesh region as in the first embodiment.

  In step S102, the information acquisition unit 105 outputs the normal information of the attention area of the subject to be imaged to the imaging angle control unit 901 and the illumination angle control unit 902.

  In step S103, the shooting angle control unit 901 controls the position and orientation of the camera so that the shooting direction of the camera with respect to the subject matches the normal angle direction N of the region of interest. Further, the illumination angle control unit 902 controls the illumination angle so that the illumination angle with respect to the subject is 45 degrees with respect to the normal angle direction N of the region of interest. Thereafter, photographing by the photographing unit 102 is performed.

In step S104, the image acquisition unit 104 acquires the captured by the photographing unit 102 the captured image data _{I N.} In step S105, the correction unit 106, similarly to the first embodiment, carried out shape correction of the captured image data I _N, and generates a shape correction image data I _'N. Further, the generation unit 108 sets and stores the pixel value of the attention area in the shape-corrected image data I ′ _{N as} the pixel value of the corresponding mesh area of the restored image data. Note that x and y are coordinate values within the range of the corresponding mesh region.

  In step S <b> 106, the generation unit 108 determines whether the acquisition of color information has been completed for all mesh regions of the restored image data. If the acquisition has not been completed, the next region of interest is set and the processing of S102 to S105 is performed. When the acquisition is completed, the process proceeds to S107.

  In step S107, the generation unit 108 outputs the generated restored image data 6 to the output unit 109 such as a printing apparatus.

  As described above, according to the second embodiment, normal information for each area of the subject is acquired prior to shooting, and the position and orientation of each area of the subject is determined based on the normal information. With this configuration, it is possible to efficiently acquire captured image data used for generating restored image data. That is, it is not necessary to perform unnecessary shooting or image processing, and it is possible to efficiently generate restored image data.

(Other examples)
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  DESCRIPTION OF SYMBOLS 101 Image processing apparatus; 102 Image | photographing part; 103 Measuring part; 104 Image acquisition part; 105 Information acquisition part; 106 Correction part; 107 Holding part; 108 Generation part;

Claims (10)

Normal acquisition means for acquiring normal information of each region constituting the subject;
Image acquisition means for acquiring a plurality of captured images obtained by capturing the subject under a plurality of different shooting conditions;
Generating means for generating a corrected image configured to satisfy a predetermined shooting condition for each region constituting the subject based on the plurality of shot images and the normal line information;
An image processing apparatus comprising: The image processing apparatus according to claim 1, wherein the plurality of shooting conditions include at least a condition regarding an illumination direction with respect to the subject and a shooting direction with respect to the subject. The image processing apparatus according to claim 1, wherein the predetermined imaging condition is a captured image from a direction that forms a predetermined angle with respect to a normal direction of each region. The generating unit selects a photographed image satisfying the predetermined photographing condition from the plurality of photographed images for each region constituting the subject, determines color information of a corresponding region in the selected photographed image, and performs the correction The image processing apparatus according to claim 1, wherein the image processing apparatus uses color information in the image. Based on the normal line information, the image processing apparatus further includes a shape correction unit that generates a shape-corrected image in which the distortion of the image shape based on the shooting direction with respect to the subject and the distortion of the image shape based on the distortion of the shape of the subject are corrected.
5. The generation unit according to claim 1, wherein the generation unit generates the correction image based on a plurality of shape correction images generated from the plurality of photographed images and the normal line information. The image processing apparatus described. The image according to claim 5, wherein the shape correction unit generates the shape correction image by deforming the image of the subject so that a normal direction in each region of the image is the same direction. Processing equipment. A shooting control unit that determines a shooting direction with respect to the subject based on the normal line information, controls a position and orientation of a shooting unit that shoots the subject, and controls the shooting unit to take the plurality of shot images; The image processing apparatus according to claim 1, further comprising: an image processing apparatus according to claim 1; The image processing according to claim 7, wherein the photographing control unit further determines an illumination direction with respect to the subject based on the normal line information, and controls a position and orientation of an illumination unit that illuminates the subject. apparatus. A normal acquisition step of acquiring normal information of each region constituting the subject;
An image acquisition step of acquiring a plurality of captured images obtained by capturing the subject under a plurality of different shooting conditions;
Based on the plurality of captured images and the normal line information, a generation step of generating a correction image configured to satisfy a predetermined capturing condition for each region constituting the subject;
An image processing method comprising:   A program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 8.

Images