JP2005049935A - Image separation method, and image processing method and device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To separate an object inside a photography image into parts each having a prescribed property, and to apply a prescribed image process to each the separated part. <P>SOLUTION: When photographing the object, two-dimensional distance information in each point on the surface of the object is acquired, a deflection angle spectroscopic information is acquired, the object inside the photography image is separated into the parts on the basis of the acquired distance information and deflection angle spectroscopic information about the object, and data processing independent in each the part is applied to image data of the object separated into the parts. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

[0001]
BACKGROUND OF THE INVENTION
The present invention relates to an image separation method, an image processing method, and an apparatus, and in particular, performs predetermined image processing that separates an image into parts based on predetermined image information related to texture, and emphasizes the texture for each separated part. The present invention relates to an added image separation / processing technique.
[0002]
[Prior art]
Today, with advances in optical equipment and image processing technology, it has become possible to capture a subject with a camera or read a subject with a scanner to obtain a high-quality image. It is difficult to reproduce the texture such as fine irregularities on the surface, and it is desired to reproduce the texture of the subject more realistically.
[0003]
An actual subject is usually a three-dimensional object, and a light source that illuminates the subject is necessary for photographing the subject. When the subject is illuminated, the brightness of the subject surface changes depending on the location. The brightness varies depending on the manner of reflection on the subject surface, and the manner of reflection varies depending on the material of the subject surface.
In general, the reflection of light on the surface of an object includes diffuse (internal) reflection that represents the color of an object and specular (surface) reflection that represents gloss.
[0004]
Diffuse reflected light is light that is incident on the surface and then exits the surface after multiple scattering within the material, and is reflected with the same intensity in all directions. Diffuse reflections are often seen on rough surfaces such as plaster and chalk.
The specular reflection light is generated by direct reflection on the surface of the subject. Specular reflection causes highlights on parts of the surface when metal or well-polished pottery is illuminated. The specular reflection light is reflected most strongly in the direction of regular reflection where the incident surface and the reflection surface are equal. Generally, when a subject is illuminated, both diffuse reflection light and specular reflection light are generated.
Therefore, in order to obtain an image that realistically reproduces the subject, it is necessary to faithfully reproduce the change in reflected light due to the difference in the material of the subject surface.
[0005]
When reproducing the texture of a subject, conventionally, image processing based on information on the surface shape of the subject, specular reflection light component, diffuse reflection light component, light source direction, and the like has been performed.
For example, assuming that a shooting target (subject) is one object, two-dimensional distance information (unevenness information) on the subject surface is measured using a laser range finder, and the light source direction is changed by a multiband camera. A method is known in which image processing is performed by capturing a single image and acquiring declination (angle) spectrum information and distance information of the subject (for example, see Non-Patent Document 1).
Further, in order to reproduce an object realistically, a method has been proposed in which spectral image measurement and three-dimensional image measurement are combined to simultaneously acquire object shape information and spectral information and reproduce the object (for example, Non-patent document 2).
[0006]
[Non-Patent Document 1]
Osaka Electro-Communication University, Shoji Tominaga, Toshinori Matsumoto, Toshihiro Tanaka “Picture recording and video generation using 3D shape information 10-4”
Color Forum JAPAN2001 p. 155-p. 158
[Non-Patent Document 2]
Nara Institute of Science and Technology, Yoshiaki Manabe “Spectroscopic Image and Shape Measurement”
Journal of the Japan Photography Society, 2002, 65, 4 p. 284-p. 290
[0007]
[Problems to be solved by the invention]
However, in the conventional acquisition of texture information and image creation from a real subject as described above, image processing is performed assuming that the shooting target is one object, and each object in the shooting target is made of material or color. Due to differences, there is no separation for each part or different image processing for each separated part, and there is a problem that fusion of image processing technology and texture reproduction technology is still insufficient is there.
[0008]
Further, in order to separate conventional images, separation is performed using color information, image structure information, and the like as feature amounts. However, there is also a problem that the separation based only on these viewpoints has insufficient separation.
Therefore, in the reproduction of texture, there are abundant amounts of information to be used, such as distance information and declination spectroscopic information (variable angle spectroscopic information). Therefore, this information is used to perform image processing for texture reproduction. Thus, it is desired to separate the image (object) for each part.
[0009]
The present invention has been made in view of the above-described conventional problems. The subject in the captured image is separated into parts having a predetermined property, and predetermined image processing is performed for each separated part. It is an object to provide an image separation method, an image processing method, and an apparatus.
[0010]
[Means for Solving the Problems]
In order to solve the above-described problem, the first aspect of the present invention obtains two-dimensional distance information at each point on the surface of the subject and obtains declination spectral information when photographing the subject. Then, an image separation method is provided that separates the subject in the captured image into parts based on the acquired distance information and declination spectral information of the subject.
[0011]
Further, it is preferable that the declination spectral information is decomposed into declination information and spectral information and used as information for separating the subject into parts.
[0012]
Moreover, it is preferable to use at least two or more pieces of information among the distance information, the declination information, and the spectral information as information for separating the subject into parts.
[0013]
Similarly, in order to solve the above-described problem, the second aspect of the present invention is characterized in that independent data processing is performed for each part on the image data of the subject separated into parts by the image separation method. An image processing method is provided.
[0014]
The independent data processing is preferably processing for changing the surface reflectance and diffuse reflectance for each part.
[0015]
Similarly, in order to solve the above-mentioned problem, the third aspect of the present invention includes distance information acquisition means for acquiring two-dimensional distance information at each point on the surface of the subject, and each information on the surface of the subject. Declination spectroscopic information acquisition means for acquiring declination spectroscopic information at a point, and image separation means for separating the subject in the captured image into parts based on the distance information and the declination spectroscopic information. An image separation device is provided.
[0016]
The image separation apparatus further includes means for decomposing the declination spectral information into declination information and spectral information, and the image separation unit includes the distance information, the declination information, and the spectroscopic information. Preferably, the subject is separated into parts using at least two pieces of information.
[0017]
Similarly, in order to solve the above-described problem, the fourth aspect of the present invention is an image processing means for subjecting image data of a subject separated into parts by the image separation method to independent data processing for each part. An image processing apparatus is provided.
[0018]
Similarly, in order to solve the above problem, a fifth aspect of the present invention provides a program for executing at least one of the image separation method and the image processing method.
[0019]
DETAILED DESCRIPTION OF THE INVENTION
Hereinafter, an image separation method, an image processing method, and an apparatus according to the present invention will be described in detail based on preferred embodiments shown in the accompanying drawings.
[0020]
FIG. 1 is a block diagram showing a schematic configuration of an embodiment of an image separation apparatus and an image processing apparatus for executing an image separation method and an image processing method according to the present invention.
As shown in FIG. 1, the apparatus of the present embodiment is separated from an image separation apparatus 12 that captures information necessary for image separation while capturing an object 10 and separates the object in the image into parts. And an image processing device 14 that performs independent image processing for each part.
[0021]
This embodiment acquires (two-dimensional) distance information and declination spectral information of each point on the subject in order to realistically reproduce the texture of the object in an image of the object (subject), for example. Thus, the subject in the photographed image is separated into several parts (parts) according to the difference in the material (which affects the difference in how the illumination light is reflected), the difference in color, and the like. Further, different image processing is performed for each separated part to emphasize the texture and reproduce the texture of the object more realistically.
[0022]
The image separation device 12 includes a declination spectral information acquisition unit 16, a distance information acquisition unit 18, a diffuse reflection component estimation unit 20, a spectral reflectance estimation unit 22, and an image separation unit 24.
In addition, the image processing device 14 includes image processing means 26 that performs predetermined image processing for each part of the image separated by the image separation device 12.
[0023]
The declination spectral information acquisition means 16 changes the declination (variation) spectral information of the subject 10, that is, the position (angle) of the light source that illuminates the subject 10 and the light receiving angle for receiving the reflected light from the subject 10. The spectral reflectance for each variable angle is acquired.
Specifically, this is not particularly limited, but for example, a multiband camera capable of capturing a plurality of spectral images with a plurality of channels can be used. That is, using a multiband camera, the angle of the light source with respect to the subject 10 may be changed, and the subject 10 may be photographed from a plurality of directions to obtain a plurality of spectral images.
Alternatively, for example, a spectral distribution for each variable angle may be measured using a variable angle spectrophotometer such as GCMS-3 manufactured by Murakami Color Research Laboratory. In this case, it is necessary to take a separate image and acquire the image data.
[0024]
The distance information acquisition unit 18 measures the shooting distance of each point on the surface of the subject 10. That is, for each point on the subject 10, the distance from the distance information acquisition means 18 to each point is measured, and so-called two-dimensional distance information for each point is obtained. It can also be said that this distance information is information representing the surface shape of the subject 10. The distance information acquisition unit 18 is not particularly limited, and may be, for example, a distance sensor using a light beam, and uses a three-dimensional image measurement method for obtaining a distance to a measurement target based on the principle of the optical triangulation method. It may be a thing.
[0025]
The diffuse reflection component estimation means 20 estimates the diffuse reflection component of the subject 10 from the declination spectral information. The diffuse reflection component estimation means 20 extracts, for each pixel of the subject image, the one having the lowest intensity of the declination spectral image and uses it as the diffuse reflection component.
However, at this time, the distance information (surface shape information) of the subject 10 and the information on the direction of the light source can be used to identify a shadow area in the subject 10, but this shadow region does not contain a diffuse reflection component and is weak in intensity. Therefore, except for this, the diffuse reflection component of the subject 10 is estimated.
[0026]
The spectral reflectance estimation means 22 estimates the spectral reflectance of the subject 10 when the light source direction (incident angle) and the observation direction (light reception angle) change, assuming a Phong model as the spectral reflectance model of the subject 10. Therefore, the parameters α and n in the Phong model are calculated.
Here, the Phong model will be described.
[0027]
FIG. 2 is a diagram for explaining the Phong model. Light emitted from the light source 50 is reflected on the surface 52 and observed (received).
At this time, the unit vector in the normal direction of the surface 52 is N, the incident angle is θ _i , the receiving angle is θ _r , the unit vector in the light source direction is L, the unit vector in the observation direction is V, and the specular reflection direction of the receiving point is If the unit vector to be represented is R _v , the light source spectrum is E (λ), and the diffuse reflectance of the subject (surface 52) is S _d (λ), then the observed reflectance Y (λ) is , Α and n are expressed by the following equation (1), with parameters depending on the material of the subject. In equation (1), (A, B) represents the inner product of vectors A and B.
Y (λ) = α (R _v · L) ⁿ E (λ) + (N · L) S _d (λ) E (λ) (1)
[0028]
Phong's model shows the ratio of the decrease in the specular reflection component due to the deviation of the observation direction from the specular reflection direction, and the cos of the deviation angle (θ _r −θ _i ) (this is equal to the inner product (R _v · L)). Is approximated by the power of n.
The undetermined parameters α and n in the model equation (1) can be uniquely determined by recording declination measurements at two or more angles. However, in practice, it is desirable to calculate an optimum value from a large amount of deviation angle measurement data.
[0029]
The diffuse reflection component estimation means 20 and the spectral reflectance estimation means 22 correspond to means for decomposing the declination spectral information into declination information (diffuse reflection component) and spectral information (spectral reflectance).
The image separating unit 24 separates the subject 10 in the captured image into parts based on the distance information and the declination spectral information. In the present embodiment, in particular, the image separation unit 24 decomposes the declination spectral information into declination information (diffuse reflection component) and spectral information (spectral reflectance), and among the distance information, declination information, and spectral information. The subject 10 in the captured image is separated into parts using at least two pieces of information.
[0030]
The image processing means 26 performs independent data processing for each part of the subject 10 separated by the image separation means 24.
Data processing performed independently for each part is not particularly limited, and various data processing can be performed. For example, the texture of the subject 10 is emphasized by changing the surface reflectance (specular reflectance) or the diffuse reflectance for each part.
[0031]
Hereinafter, as an operation of the present embodiment, an image separation method and an image processing method according to the present invention will be described.
FIG. 3 is a flowchart showing the flow of processing of the image separation method and the image processing method in the present embodiment. Further, in the present embodiment, two triangular pyramids 30 and 40 as shown in FIG. In the triangular pyramid 30, the left surface 32 is a metal surface, the right surface 34 is a non-metal surface, and the left and right surfaces 32 and 34 are the same color. In the triangular pyramid 40, the left surface 42 and the right surface 44 are different colors, and the left and right surfaces 42, 44 are metal surfaces.
[0032]
Hereinafter, description will be given along the flowchart of FIG.
First, in step 100, the distance information acquisition means 18 acquires distance information of each subject (triangular pyramids 30 and 40).
Next, in step 110, multiband image capturing of the triangular pyramid 30 and the triangular pyramid 40 is performed. In this photographing, photographing is performed while moving at least one of the position of the light source and the photographing position of the multiband camera, and a plurality of spectral images are obtained for each subject.
[0033]
The respective declination spectral information is acquired from the captured multiband images of the triangular pyramid 30 and the triangular pyramid 40.
Note that, as described above, the method for capturing the subject and acquiring the distance information, the declination information, and the spectral information is not particularly limited. For example, the conventionally known Non-Patent Document 1 or Non-Patent You may use the method by Tominaga etc. or the method by Manabe etc. which are described in the literature 2.
[0034]
Next, in step 130, the diffuse reflection component estimation means 20 estimates the diffuse reflection component representing the color of the subject from the respective declination spectral images of the triangular pyramid 30 and the triangular pyramid 40. In this estimation, as described above, the one having the lowest intensity of the declination spectral image is extracted for each pixel of the subject, and this is used as the diffuse reflection component.
Next, in step 140, the spectral reflectance estimation means 22 calculates the two parameters α and n in the above-mentioned Phong's model from the respective declination spectral images for the triangular pyramid 30 and the triangular pyramid 40.
[0035]
Next, in step 150, the image separation means 24 separates the subject in the image into parts. This separation method is performed using at least two pieces of information among distance information, declination information, and spectral information, and is not particularly limited to one method.
For example, for the triangular pyramid 30, the left surface 32 and the right surface 34 have the same color, but are different in material. In this case, it can be separated into parts using distance information and spectral information. This method will be described below with reference to the flowchart of FIG.
[0036]
First, in step 200 of FIG. 5, the image separation unit 24 calculates the outward normal vector direction at each point on the subject from the distance information (surface shape information) received from the distance information acquisition unit 18. This outward normal vector direction is equivalent to a value obtained by differentiating the distance information at each point in the X, Y, and Z directions (that is, the vertical, horizontal, and depth directions).
[0037]
Next, in step 210, the image separating means 24 determines whether or not the calculated outward normal vector direction has changed discontinuously at that point, and the XYZ value of the outward normal vector has changed discontinuously. A point to be selected (a point where the differential value in the XYZ directions changes discontinuously) is extracted (selected) as a candidate for a point where the material of the subject changes.
Such a point where the material changes can be a boundary when the subject is separated into parts.
In the case of the triangular pyramid 30 in FIG. 4, as is clear from the drawing, the normal direction of the left surface 32 and the right surface 34 changes greatly at the boundary, and the material of the subject changes at the point on the boundary. Candidate points.
[0038]
Next, in step 220, the image separating unit 24 receives the parameters α and n in the Phong model calculated by the spectral reflectance estimating unit 22, and whether or not the values of the parameters α and n have changed greatly at that point. to decide.
In this determination, when the values of the parameters α and n are greatly changed, the outward normal vector direction is discontinuously changed at that point, and the values of the parameters α and n are Since it has changed greatly, in the next step 230, the point is determined to be a point where the material of the subject changes. The above steps are executed for all pixels, and all points where the material changes are extracted.
[0039]
In step 240, the subject is separated into parts with the point where the material of the subject changes as a boundary. In the case of the triangular pyramid 30 in FIG. 4, it is confirmed that the left surface 32 is a metal and the right surface 34 is a nonmetal, and the materials thereof are different, but the values of the parameters α and n are actually different. It was. The difference in the values of the parameters α and n of the Phong model is due to the difference in the angle-dependent intensity distribution of the regular reflection component and the diffuse reflection component due to the difference in the material in the spectral reflection light data of the subject. In fact, in the triangular pyramid 30, the outward normal vector directions of the left surface 32 and the right surface 34 change greatly, and the values of the parameters α and n are different, so the left surface 32 and the right surface 34 are separated as separate parts. It was done.
[0040]
Further, for example, for the triangular pyramid 40, the left surface 42 and the right surface 44 are made of the same material but different in color. In this case, it can be separated into parts using distance information and declination information (diffuse reflection component). This method will be described below with reference to the flowchart of FIG.
[0041]
First, in step 300 of FIG. 6, the image separation unit 24 calculates the outward normal vector direction at each point on the subject from the distance information (surface shape information) received from the distance information acquisition unit 18. This outward normal vector direction is equivalent to a value obtained by differentiating the distance information at each point in the X, Y, and Z directions (that is, the vertical, horizontal, and depth directions).
[0042]
Next, at step 310, the image separation means 24 determines whether or not the calculated outward normal vector direction has changed discontinuously at that point, and the XYZ value of the outward normal vector has changed discontinuously. A point to be selected (a point where the differential value in the XYZ directions changes discontinuously) is extracted (selected) as a candidate for a point where the material of the subject changes.
Such a point where the material changes can be a boundary when the subject is separated into parts.
In the case of the triangular pyramid 40 in FIG. 4, as is apparent from the drawing, the normal direction of the left surface 42 and the right surface 44 changes greatly at the boundary, and the material of the subject changes at the point on the boundary. Candidate points.
[0043]
Next, at step 320, the image separation means 24 receives the diffuse reflection component estimated by the diffuse reflection component estimation means 20, and determines whether or not this diffuse reflection component has changed significantly at that point.
In this determination, if the diffuse reflection component is significantly changed, the outward normal vector direction is discontinuously changed at that point, and the diffuse reflection component is greatly changed. In the next step 330, the point is determined to be a point where the material of the subject (in this case, the color) changes.
[0044]
Then, in step 340, the subject is separated into parts at the point where the material (color) of the subject changes. In the case of the triangular pyramid 40 in FIG. 4, the left surface 42 and the right surface 44 are made of the same material, but their colors are different. It was confirmed that the diffuse reflection components were actually different. The diffuse reflection component corresponds to a color, and it has been confirmed that the color is actually different between the left surface 42 and the right surface 44. As a result, also in the triangular pyramid 40, the left surface 42 and the right surface 44 are separated as separate parts.
[0045]
In this way, according to the present embodiment, the subject is distance information (outward normal vector direction calculated therefrom), spectral information (Phong's model parameters α and n), and declination information (diffuse reflection component). ) Can be determined to determine the point at which the material (color) changes, thereby separating the subject into parts.
[0046]
Next, returning to step 160 in FIG. 3, the image processing means 26 performs desired image processing on the distance, spectral and declination information for each part obtained by the method up to step 150.
Specifically, each of the separated parts of the triangular pyramid 30 or the triangular pyramid 40 in FIG. 4 was subjected to a color change process based on the characteristics of each part, such as being metallic or non-metallic. For example, if the material is changed from non-metal to metal, the parameters α and n of the Phong model may be changed so that the spectrum of the diffuse reflected light itself is not changed. When only the color of the part is changed, only the spectrum of the diffuse reflected light may be changed without changing the parameters α and n of the Phong model.
Thus, by generating a processed image based on the characteristics of each part, an image with a changed texture could be obtained.
[0047]
The image processing is not particularly limited. For example, by changing the specular reflection component (specular reflection component) for each part, an image with a different metallic feeling can be created, or the part can be cut out. Paste another part, combine several subjects (objects) to create a three-dimensional scene, or perform conventional image processing such as degradation and restoration by multiplying parts by different amounts Various processes are exemplified.
[0048]
In addition, the image separation method and the image processing method described above are assembled and stored in a predetermined recording medium, read from this, or downloaded via a network or the like, and the image separation of the present invention can be performed by an arbitrary computer. The method and the image processing method can be easily executed, and an image with a changed texture can be obtained.
[0049]
The image separation method, the image processing method, and the apparatus according to the present invention have been described in detail above. However, the present invention is not limited to the above-described embodiments, and various kinds of methods can be used without departing from the gist of the present invention. Of course, improvements and changes may be made.
[0050]
【The invention's effect】
As described above, according to the present invention, it is possible to separate a subject in a captured image into parts having a predetermined property and perform predetermined image processing for each of the separated parts. In particular, by representing each object being imaged separately based on information such as distance information related to the texture, declination spectral information, etc., and further performing image processing using these separated information, It is possible to obtain an image that more realistically represents the texture.
[Brief description of the drawings]
FIG. 1 is a block diagram showing a schematic configuration of an embodiment of an image separation apparatus and an image processing apparatus that execute an image separation method and an image processing method according to the present invention.
FIG. 2 is an explanatory diagram for explaining a Phong model;
FIG. 3 is a flowchart illustrating a processing flow of an image separation method and an image processing method according to the present embodiment.
FIGS. 4A and 4B are perspective views showing a triangular pyramid used as a subject in the present embodiment.
FIG. 5 is a flowchart showing a method of separating an image into parts in the present embodiment.
FIG. 6 is a flowchart showing another method for separating an image into parts in the present embodiment.
[Explanation of symbols]
DESCRIPTION OF SYMBOLS 10 Subject 12 Image separation apparatus 14 Image processing apparatus 16 Declination spectral information acquisition means 18 Distance information acquisition means 20 Diffuse reflection component estimation means 22 Spectral reflectance estimation means 24 Image separation means 26 Image processing means 30 and 40 Triangular pyramids 32 and 42 Left side 34, 44 Right side 50 Light source 52 Surface

Claims (9)

When photographing a subject, two-dimensional distance information at each point on the surface of the subject is obtained, and declination spectral information is obtained,
An image separation method, comprising: separating the subject in a captured image into parts based on the acquired distance information and declination spectral information of the subject. The image separation method according to claim 1, wherein the declination spectral information is decomposed into declination information and spectral information and used as information for separating the subject into parts. The image separation method according to claim 2, wherein at least two pieces of information among the distance information, the declination information, and the spectral information are used as information for separating the subject into parts. An image processing method, wherein independent data processing is performed for each part on image data of a subject separated into parts by the image separation method according to claim 1. The image processing method according to claim 4, wherein the independent data processing is processing for changing a surface reflectance and a diffuse reflectance for each part. Distance information acquisition means for acquiring two-dimensional distance information at each point on the surface of the subject;
Declination spectral information acquisition means for acquiring declination spectral information at each point on the surface of the subject;
Image separation means for separating the subject in the captured image into parts based on the distance information and the declination spectral information;
An image separating apparatus comprising: The image separation apparatus according to claim 6, further comprising means for decomposing the declination spectral information into declination information and spectral information, wherein the image separation unit includes the distance information, the declination information, and An image separating apparatus, wherein the subject is separated into parts using at least two or more pieces of the spectral information. Image processing means comprising image processing means for performing independent data processing for each part on image data of a subject separated into parts by the image separation method according to claim 1. apparatus. A program for executing at least one of the image separation method according to claim 1 and the image processing method according to claim 4 or 5.

Images