JP2010002273A - Image processing method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing method can reduce the quantity of data on a three-dimensional image. <P>SOLUTION: This method includes a step S11 for finding a normal line direction concerning a reference point whose distance from a view point is known in a two-dimensional image, and a step S12 for finding a distance from the view point and a normal line direction on the basis of an illumination difference obtained by using the reference point as a reference concerning a non-reference point whose distance from the view point is unknown in the two-dimensional image. Three-dimensional image information of the two-dimensional image is represented by the two-dimensional image, the distance and the normal line direction at the reference point, and the distance and the normal line direction at the non-reference point. <P>COPYRIGHT: (C)2010,JPO&INPIT

Description

  The present invention relates to an image processing method, for example, an image processing method related to a three-dimensional stereoscopic image.

  Conventionally, in order to express a three-dimensional image (an image in a three-dimensional space, meaning a stereoscopic image; the same applies hereinafter), a plurality of images, for example, two obtained from two viewpoints at a distance corresponding to parallax. One image is used (for example, refer to Patent Document 1).

A plurality of two-dimensional images (a two-dimensional space image, which means a planar image; the same applies hereinafter) are stored, and the plurality of two-dimensional images are used when restoring a stereoscopic image. As a result, there is a problem that the data amount of the three-dimensional image becomes very large.
JP 2007-096951 A

  The present invention provides an image processing method capable of reducing the data amount of a stereoscopic image.

  An image processing method according to an aspect of the present invention includes a step of obtaining a normal direction for a reference point whose distance from a viewpoint is known in a two-dimensional image, and the distance from the viewpoint is unknown in the two-dimensional image. Obtaining a distance from a viewpoint and a normal direction based on an illuminance difference with respect to the reference point for a non-reference point, the two-dimensional image, and the distance and normal at the reference point The stereoscopic image information of the two-dimensional image is represented by the direction and the distance and normal direction at the non-reference point.

  According to the present invention, it is possible to provide an image processing method capable of reducing the data amount of a stereoscopic image.

  Embodiments of the present invention will be described below with reference to the drawings. In the description, common parts are denoted by common reference symbols throughout the drawings.

[First Embodiment]
An image processing method according to the first embodiment of the present invention will be described with reference to FIG. FIG. 1 is a flowchart of an image processing method according to this embodiment, and shows a method of including stereoscopic information in a two-dimensional image obtained by photographing with a camera.

  As shown in the figure, first, a reference point and a non-reference point are determined in the two-dimensional image (step S10). The reference point is a point whose distance from the viewpoint (camera) is known, and the non-reference point is another point whose distance is unknown. Further, the reference point and the non-reference point may correspond to each pixel of the two-dimensional image, or may correspond to an area having a set of a plurality of pixels as a unit. Hereinafter, a case corresponding to each pixel will be described as an example.

  To get a reference point, the distance must be known. The distance can be grasped by distance measurement by focusing, distance measurement using infrared rays, or the like, a stereo correspondence point search method using a plurality of two-dimensional images, or motion estimation.

  Then, the normal direction of the reference point is obtained (step S11). The normal direction of the object at the reference point is obtained. The normal direction of the reference point can be obtained from three reference points in the vicinity. That is, the normal direction can be obtained by outer product calculation using the three-dimensional coordinates of the three neighboring points. Further, it can be obtained by the illuminance difference stereo method.

  Further, the distance from the viewpoint of the non-reference point and the normal direction are obtained (step S12). The calculation of the distance and normal direction of the non-reference point obtained in this step is a continuous curved surface or plane that is sufficiently close to the non-reference point to be obtained and is the same as the non-reference point to be obtained. This is done based on a reference point that can be estimated to be above. Hereinafter, a method for calculating the distance and normal direction of the non-reference point will be described with reference to FIG. FIG. 2 is a flowchart of a non-reference point distance and normal direction calculation method.

  As shown in the figure, the angle difference between the normal direction at the reference point and the light source is obtained by the illuminance difference stereo method, and the angle difference between the normal direction at the non-reference point and the light source is obtained by the illuminance difference stereo method (step S20). . At this time, the normal direction of the non-reference point is unknown, but according to the illuminance difference stereo method, it is possible to calculate the angle difference between the normal direction and the light source. The difference in angular difference between the reference point and the non-reference point with respect to the light source can be regarded as an angular difference in the normal direction between the reference point and the non-reference point (step S21). Accordingly, the normal direction of the non-reference point can be estimated based on the angle difference obtained in step S20 and the normal direction of the non-reference point whose reference point or normal has already been estimated (step S22). Subsequently, an angle difference from the center of curvature is calculated when it is assumed that the reference point and the non-reference point are on the same curved surface or plane (step S23). Then, the distance of the non-reference point can be obtained based on the angle difference obtained in step S23 (step S24).

  A method for calculating the distance from the viewpoint and the normal direction at the non-reference point will be described using a specific example. FIG. 3 is a schematic diagram showing an object to be imaged and a light source. As shown in the figure, an object has a cylindrical shape, and light is emitted from a light source L to the object. Note that the position of the light source L with respect to the object can be approximated to infinity. And the image image | photographed with a camera is a two-dimensional image (XY plane) containing X direction and the Y direction orthogonal to this, The depth direction is a Z direction.

  FIG. 4 is a schematic diagram showing the YZ plane in FIG. 3 and particularly showing the relationship between the surface of the cylinder and the light source. By the above step S20, the angle difference θ1 between the normal direction of a certain reference point Q on the cylinder surface (the direction of the normal vector n1) and the light source, and the normal direction of the non-reference point P on the cylinder surface (normal vector n2) The angle difference θ2 between the light source and the light source is obtained by the illuminance difference stereo method. The angle difference θ1 is an angle difference between the straight line L1 connecting the light source L and the reference point Q and the direction of the normal vector n1. The angle difference θ2 is an angle difference between the straight line L2 connecting the light source L and the non-reference point P and the direction of the normal vector n2. When the reference point Q and the non-reference point P are sufficiently close, it can be approximated that the light source L is relatively infinite. If the light source L is at infinity, it can be seen that the straight line L1 and the straight line L2 are parallel. The amount of change Δθ1 (= θ2−θ1) from the angle difference θ1 to the angle difference θ2 can be regarded as a change in angle of the direction of the normal vector n2 with respect to the direction of the normal vector n1 (step S21). .

  For example, assuming that the light source is at infinity and considering an upright cylinder, the angle (vertical component) between the light source of the non-reference point in the vertical direction of the reference point does not change, but the non-reference point in the left-right direction does not change The angle (lateral direction) with the light source changes. The angle can be estimated as a change in the horizontal direction of the normal. In the case of a non-reference point in an oblique direction, the vector sum of changes in the vertical and horizontal components appears as an angular difference.

  Then, since the direction of the normal vector n1 is known, the direction of the normal vector n2 can be obtained based on Δθ1 (step S22). Subsequently, an angle difference Δθ2 from the center of curvature between the reference point and the non-reference point is obtained (step S23). This angle difference Δθ2 can be obtained from θ1 and θ2.

  Next, a method for obtaining the distance from the viewpoint of the non-reference point P from the angle difference Δθ2 will be described with reference to FIG. FIG. 5 also shows the YZ plane as in FIG. Note that it is assumed that the reference point Q and the non-reference point P can be estimated to be on the same continuous curved surface or plane and are sufficiently close in position. As shown in the figure, the angle difference Δθ2 obtained in step S23 is an angle difference from the center of curvature O of the cylindrical object to the reference point Q and the non-reference point P. That is, it corresponds to the angle between the line segment OP and the line segment OQ.

  Then, since the reference point Q and the non-reference point P have the same radius of curvature, considering the line segment QP connecting the reference point Q and the non-reference point P, ∠OQP and ∠OPQ are equal. This is the angle α. A point obtained by orthogonally projecting the non-reference point P onto a plane parallel to the XY plane and having the reference point Q is defined as a point R, and a point obtained by further extending a straight line extending from the center of curvature O to the reference point Q is defined as a point S. Then, since the solid angle of the normal line of the reference point Q is known, the angle β of ∠RQS is also known. As a result, ∠RQP = angle γ = π− (α + β).

  The line segment RQ is obtained as the difference between the Y coordinates of the points R and Q. Therefore, the line segment RP is obtained by RP = RQ / tan γ. That is, if the Z coordinate of the reference point Q is z1, and the Z coordinate of the non-reference point P is z2, z2 = z1 + RP. Then, if the Z coordinate value z2 of the non-reference point P is obtained, the distance from the viewpoint can be easily obtained (step S24). The viewpoint position can be considered as the origin of the XYZ coordinates.

  As described above, the distance from the viewpoint and the direction of the normal line at the reference point and the non-reference point are obtained, and the stereoscopic image is represented by the two-dimensional image, the distance from the viewpoint, and the normal direction (step S13). . Then, the two-dimensional image, distance, and normal direction are compressed and stored in the storage device (step S14). The distance and the normal direction are stored in a storage device in association with a pixel of a two-dimensional image or a set of temporal ranges and / or spatial ranges of pixels (hereinafter referred to as basic pixel units).

<Effect>
As described above, with the image processing method according to the first embodiment of the present invention, the following effects (1) and (2) can be obtained.

(1) The amount of data about a stereoscopic image can be reduced.
In the image processing method according to this embodiment, a three-dimensional image is expressed by a two-dimensional image, distance information, and normal information. That is, the three-dimensional information of each object included in the two-dimensional image is directly used. Accordingly, there is no need to hold a plurality of two-dimensional images taken from different directions for the same scene as in the conventional case, and the data amount of the stereoscopic image can be reduced. Further, by directly using the three-dimensional information, it becomes easy to handle the three-dimensional object included in the two-dimensional image.

(2) A three-dimensional shape can be easily grasped even on a surface with poor texture.
As a method for measuring the distance from the viewpoint, a distance measurement by focusing, a distance measurement using infrared rays, or a stereo corresponding point search method using a plurality of two-dimensional images is generally known. However, an object with poor texture (for example, a plane or curved surface having the same color, no pattern, and almost no change in surface shape) has few feature points. Therefore, it is difficult to estimate the distance by a stereo correspondence point search method or distance measurement by focusing, and even if it can be estimated, the accuracy tends to be low. Therefore, in order to measure the distance of such an object, it is necessary to perform distance measurement using a laser or infrared rays. However, this method has a problem that the apparatus becomes large-scale.

  However, in the method according to the present embodiment, the distance from the viewpoint of the non-reference point and the normal direction are calculated using the illuminance difference stereo method. Therefore, even on the surface of an object with poor texture, in other words, even on a surface with poor pattern or shape change, the distance and normal of each point on the surface can be calculated with high accuracy, and the three-dimensional shape can be easily obtained. I can grasp.

  That is, by using the illuminance difference stereo method, regardless of the texture, for example, if the illuminance difference is constant at a plurality of points, it can be seen that it is a plane. And if it is a plane, it turns out that the normal and distance of the point in the plane are the same. Further, if the difference in illuminance changes at a plurality of points, it is understood that it is a curved surface. Since it is a curved surface, the normal direction also changes gradually. Then, the curvature of the curved surface can be obtained by changing the normal direction, and as a result, the distance from the viewpoint can be obtained for each point of the non-reference point based on any reference point on the curved surface. I can do it.

  FIG. 6 is a diagram schematically illustrating the distance measuring method according to the present embodiment, and illustrates how the distance of a cylindrical surface is obtained. For example, it is assumed that the distance between the boundary between the cylinder and the background portion can be estimated by, for example, a stereo correspondence point search method. Then, this point Q becomes a reference point. The normal vector of the reference point Q is n1.

  Then, the direction of the normal vector n2 of the non-reference point P close to the reference point Q is known by the illuminance difference stereo method. Therefore, the distance of the non-reference point P can be obtained from the normal vectors n1 and n2. Next, when the distance of the non-reference point R close to the non-reference point P is obtained, it can be obtained based on the non-reference point P. That is, as a result of obtaining the normal vector n2 and the distance of the non-reference point P, the non-reference point P can be incorporated into the reference point. Since the normal vector n3 of the non-reference point R can be obtained by the illuminance difference stereo method, if the distance is known, the non-reference point R can also be incorporated into the reference point. Similarly, the distance of the non-reference point S can be obtained based on the non-reference point R.

  Note that the point (non-reference point) for which the distance is obtained by the above method can be estimated to be on the same continuous curved surface or plane as the curved surface or plane on which the reference point exists, and the position is sufficiently close to the reference point. It is a point. Whether they are on the same continuous curved surface or plane, whether the normal direction is the same, is continuously changing, or is in the same feature region surrounded by edges It can be judged by whether or not. Whether or not the positions are close can be determined from the proximity of the feature amount and the position in the XY direction.

  In some cases, the reference point constitutes a vertex or a ridgeline of the object. In such a case, it is desirable to obtain the normal of the non-reference point by using a point in the vicinity of the vertex or ridge line as the reference point. This is because normals cannot be defined for vertices and edges. Furthermore, in FIG. 4 and FIG. 5, the case where the orthogonal coordinate system is used has been described, but this is for ease of explanation, and even in other coordinate systems, the same calculation can be performed by using coordinate transformation. It is.

[Second Embodiment]
Next explained is an image processing method according to the second embodiment of the invention. The present embodiment relates to details of step S14 described in the first embodiment. Accordingly, the processing in the other steps is as described in the first embodiment, and the description thereof is omitted.

<Compression of distance information>
First, the distance information compression method will be described with reference to FIG. FIG. 7 is a flowchart showing a distance information compression method.

  As the distance from the viewpoint increases, the parallax decreases accordingly, and the parallax becomes zero at a position at infinity. Therefore, the accuracy of the distance information expressing the distance decreases as the distance increases. Therefore, the distance information can be expressed by a method in which the amount of information decreases according to the distance from the viewpoint. That is, the distance information is expressed by, for example, a non-linear function (hereinafter referred to as a distance function, step S30). As the distance function, for example, a logarithmic function similarity or a combination of exponential functions can be considered. As an example, if the distance is D (i, j), the distance information can be expressed as δ (i, j) = log (1 / D (i, j)). Note that (i, j) indicates coordinates in the two-dimensional image (XY plane). D (i, j) and δ (i, j) are functions of i and j.

  Also, at a certain distance, it becomes almost indistinguishable from infinity. Therefore, it is also possible to use a distance function that saturates a value, assuming that distance information above a certain level is at infinity. Furthermore, when the distance is a certain distance from the depth of field, the image is not focused and the distance resolution is reduced. Therefore, for a short distance that is a certain degree from the depth of field, a distance function that saturates to a constant value may be used. The distance regarded as infinity and the distance from the depth of field may be changed according to the surrounding conditions such as the depth of field and illuminance.

  Although the compression process may be completed in step S30, the distance information δ (i, j) may be continuously encoded (step S31). An example of encoding is transform encoding, for example. When transform coding is performed, the distance information δ (i, j) obtained in step S30 is subjected to discrete Fourier transform or discrete cosine transform to obtain a coefficient group Fd (u, v). A long code is assigned to a coefficient having a large variance, and a short code is assigned to a coefficient having a small variance. In other words, a long code is assigned to a coefficient having a low appearance frequency, and a short code is assigned to a code having a high appearance frequency. Usually, since the dispersion of the low frequency component is large and the dispersion of the high frequency component is small, a long code is assigned to the coefficient of the low frequency component and a short code is assigned to the coefficient of the high frequency component. Thus, by determining and assigning an appropriate code length for each coefficient, transform-coded distance information Fds (u, v) can be obtained.

  Of course, the encoding in step S30 is not limited to transform encoding, and various methods can be used. At the time of encoding, the allocated code length may be changed according to the visual characteristics and the depth of field in the same manner as the distance function (step S32). That is, since the resolution is high within the depth of field, the assigned code length is increased. Further, the code to be assigned may be reduced as the distance increases. Alternatively, a combination of a distance function and a code length may be selected according to the distance and the depth of field, and many codes may be assigned within the depth of field.

<Compression of normal information>
Next, a normal information compression method will be described with reference to FIG. FIG. 8 is a flowchart showing a normal information compression method.

  First, normal information is expressed by, for example, polar coordinates (r, θ, φ) (step S40). Where r is the length of the normal, θ is the polar angle of the normal, and φ is the azimuth of the normal. In this way, using polar coordinates separates the angle and length, making it easier to handle information. The normal information may be expressed by (θ, φ) without the normal length information. The normal length information may be expressed using the distance function δ (i, j) described in step S30 instead of the length r (i, j). The normal information corresponding to the point (i, j) on the two-dimensional image can be expressed as a function f (i, j) of i and j.

  Although the compression process may be completed in step S40, the normal vector information f (i, j) may be continuously encoded (step S41). An example of encoding is entropy encoding, transform encoding, vector quantization, or the like. These encodings will be described below. Of course, the encoding method of normal line information is not limited to these.

<< Entropy coding >>
First, entropy coding will be described. Entropy coding converts normal information into another symbol with low entropy at the time of coding.

  That is, consider the difference d (i, j) between the normal information f (i, j) of interest and the normal information f (i−1, j) adjacent thereto. That is, d (i, j) = f (i, j) −f (i−1, j). The calculation at this time is a three-dimensional calculation for (r, θ, φ) obtained in step S40, or (δ, θ, φ) using a distance function for the length of the normal, or length information is omitted. This is a two-dimensional calculation for (θ, φ). Furthermore, the calculation for the angle information θ and φ is performed in a 2π cycle.

  The entropy-coded normal line information can be decoded based on d (i, j) and already decoded f (i-1, j). That is, normal information f (i, j) can be decoded by calculating f (i, j) = f (i−1, j) + d (i, j).

  According to entropy coding, only the difference from adjacent normal line information needs to be stored, and the information can be compressed. In the case of a moving image, the same encoding can be performed between frames. That is, the difference of f (i, j) between two temporally continuous two-dimensional images (frames) may be obtained and stored. For example, it is assumed that the normal information at time t1 is f (i, j, t1), and the normal information at time t2 subsequent to time t1 is f (i, j, t2). In this case, the normal information is compressed by obtaining the difference d (i, j) = f (i, j, t1) −f (i, j, t2). In the case of decoding, a calculation of f (i, j, t2) = f (i, j, t1) −d (i, j) may be performed.

<< Transform coding >>
Next, transform coding will be described. Transform coding is the same as in the case of the distance information described above. That is, discrete Fourier transform or discrete cosine transform is performed on f (i, j) obtained in step S40 to obtain a coefficient group Fn (u, v). Then, as in the case of distance information, normal information Fns (u, v) is obtained by assigning a short code to a coefficient having a high appearance frequency and assigning a long code to a coefficient having a low appearance frequency.

  In decoding, Fn (u, v) is obtained from Fns (u, v) based on the code allocation method, and this is further subjected to inverse transform coding. As a result, f (i, j) is obtained.

<< Vector quantization >>
Next, vector quantization will be described. In vector quantization, encoding is not performed for each piece of normal information, but a collection of a plurality of normals is expressed by a vector having normal information as an element and encoded. A vector quantization method will be described with reference to FIG. FIG. 9 is a flowchart showing the flow of vector quantization processing.

  As shown in the figure, first, normal information is divided into partial spaces (step S50). That is, the two-dimensional image is divided into a plurality of partial spaces (blocks) that are a set of a plurality of normal line information. The subspace can be rephrased as a set of a plurality of pixels. Then, an average value (average vector) of normal line information in each block is obtained (step S51). Next, in each partial space, a difference vector representing the difference between the normal information of each pixel unit and the average value obtained in step S51 is obtained (step S52). Subsequently, a representative vector that is most similar to the difference vector obtained in step S52 is extracted from a previously facilitated codebook (step S53). A code book stores a plurality of representative vectors. Then, an index that is information indicating what number representative vector of the code book is a code. And the average vector calculated | required by step S51 and the index extracted by step S53 are output as normal-line information of a two-dimensional image (step S53).

  A specific example of the above will be described with reference to FIG. FIG. 10 is a schematic diagram sequentially illustrating a procedure for vector quantization of normal information for a two-dimensional image having (4 × 4) pixels.

  As shown in the drawing, first, 16 pixels included in the two-dimensional image of (4 × 4) pixels are referred to as pixels P0 to P15, respectively. Further, the normal line information in each pixel is referred to as normal line information f0 to f15, respectively. Then, the two-dimensional image is divided into four partial spaces BLK0 to BLK3 including (2 × 2) pixels. The partial space BLK0 includes pixels P0 to P3, the partial space BLK1 includes pixels P4 to P7, the partial space BLK2 includes pixels P8 to P11, and the partial space BLK3 includes pixels P12 to P15. It is.

  Next, average vectors fave0 to fave3 in the partial spaces BLK0 to BLK3 are obtained. The average vector fave0 is the average value of the normal information f0 to f3, the average vector fave1 is the average value of the normal information f4 to f7, the average vector fave2 is the average value of the normal information f8 to f11, and the average vector fave3 is an average value of the normal line information f12 to f15. An image composed of the average vectors fave0 to fave3 is called an average value image.

  Next, in each of the partial spaces BLK0 to BLK3, difference vectors d0 to d15 between the average vectors fave0 to fave3 and the normal line information of each pixel are obtained. That is, d0 = fave0-f0, d1 = fave0-f1, d2 = fave0-f2, and d3 = fave0-f3. Further, d4 = fave1-f4, d5 = fave1-f5, d6 = fave1-f6, and d7 = fave1-f7. Furthermore, d8 = fave2-f8, d9 = fave2-f9, d10 = fave2-f10, and d11 = fave2-f11. Furthermore, d12 = fave3-f12, d13 = fave3-f13, d14 = fave3-f14, and d15 = fave3-f15. An image composed of the difference vectors d0 to d15 is called a difference image.

  Next, reference is made to the code book. The code book includes, for example, eight differential vectors (representative vectors) as representatives, and indexes 0 to 7 are assigned to each. Each representative vector includes, for example, four difference vectors. This is because each of the partial spaces BLK0 to BLK3 includes four pieces of normal information.

  Then, the optimum representative vector for obtaining the difference image is selected from the code book for each partial space. In the example of FIG. 9, the optimum representative vectors for the blocks BLK0 to BLK3 are the indexes 0, 2, 4, and 6, respectively.

  The average value image and indexes 0, 2, 4, and 6 obtained as described above are output as codes by vector quantization.

  Decoding is performed using the average image and the index. That is, first, a code book is referenced using an index, and a representative vector is extracted for each partial space. Next, using the representative vector as a difference vector, the normal vector information of the subspace is calculated from the average vector included in the average image. Then, the normal information of the entire two-dimensional image is integrated from the normal information of the partial space.

  In the example of FIG. 10, 0, 2, 4, and 6 are given as indexes, so representative vectors corresponding to these indexes are extracted from the codebook. Then, each representative vector and the average vectors fave0 to fave3 are added to obtain normal information f0 to f15. That is, in the partial space BLK0, the average vector fave0 and the difference vectors d0 to d3 are added to obtain normal line information f0 to f3. In the partial space BLK1, the average vector fave1 and the difference vectors d4 to d7 are added to obtain normal line information f4 to f7. The same applies to the partial spaces BLK2 and BLK3. Then, by integrating the obtained normal line information f0 to f15, normal line information of the whole (4 × 4) pixel two-dimensional image can be obtained.

<Extraction of 3D objects>
At the time of encoding, not only the distance information and normal line information at each pixel is encoded, but also a solid object as an object is grasped and extracted from the subject, and this is associated with the two-dimensional image. It may be encoded.

<< Plain object extraction >>
First, a planar object extraction method will be described with reference to FIGS. 11 and 12. FIG. 11 is a flowchart showing an extraction method, and FIG. 12 is a schematic diagram showing a rectangular parallelepiped object.

  First, a region having a similar normal direction in the two-dimensional image is extracted (step S60). For example, in the example of FIG. 12, in the plane F1 in front of the rectangular parallelepiped (the hatched area in FIG. 12), the normals face the front at all points. Accordingly, the surface F1 is extracted as a region having a similar normal direction. Thereby, boundary information indicating the shape of the surface F1 is obtained. That is, in the case of FIG. 12, two sides in contact with the other two surfaces F2 and F3 and two sides in contact with the background are obtained as boundary information. The boundary information is expressed by coordinates in a two-dimensional image, for example.

  Next, the position of the representative point and the normal information on the surface extracted in step S60 are selected (step S61). In the example of FIG. 12, for example, one of the points included in the surface F1 is selected as the representative point. The representative point is preferably a characteristic point. The plane F1 is expressed by the representative point position and the normal information at the representative point (these are referred to as representative point information).

  When storing a two-dimensional image, the image information of the region extracted in step S60 and the viewpoint position (shooting position) are associated with the boundary information and representative point information obtained in steps S60 and S61 (step S62). . In FIG. 12, the case where the surface F1 is extracted has been described as an example, but the same applies to the case where the surfaces F2 and F3 are extracted.

<< Extraction of curved surface object >>
Next, a method for extracting a curved object will be described with reference to FIGS. FIG. 13 is a flowchart showing an extraction method, and FIG. 14 is a schematic diagram showing a semi-cylindrical object.

  First, a region in which the radius of curvature is similar in the two-dimensional image and the center of curvature is a continuous curve or point is extracted (step S70). In the example of FIG. 14, the curved surface in front of the semi-cylindrical is a straight line L1 having a continuous center of curvature and the same radius of curvature r. Accordingly, the curved surface F4 is extracted, and boundary information indicating the shape of the curved surface F4 is obtained. That is, in the case of FIG. 14, one side in contact with the upper surface F5 of the half cylinder and three sides in contact with the background are obtained as boundary information.

  Next, the curve formed by the curvature center of the surface extracted in step S70 and the curvature radius are stored (step S71). In the example of FIG. 14, the curve (or point) formed by the straight line L1 at the center of curvature and the radius of curvature r are stored. The radius of curvature is obtained from the distance between two points on the curve and the angle difference between the normals, and the center of curvature is obtained by solving simultaneous equations of curvature circles from the coordinates of the two points and the radius of curvature. The curved surface F4 is expressed by the curve and the radius of curvature (these are called curved surface information) thus obtained.

  When storing the two-dimensional image, the image information of the region extracted in step S70 and the viewpoint position (shooting position) are associated with the boundary information and curved surface information obtained in steps S70 and S71 (step S72). Note that the upper surface F5 in FIG. 14 can be extracted by the method described with reference to FIGS.

<Effect>
As described above, the image processing method according to the second embodiment of the present invention has the following effect (3) in addition to the effects (1) and (2) described in the first embodiment. Is obtained.

(3) The amount of data about the stereoscopic image can be further reduced.
In the method according to the present embodiment, distance information and normal information are compressed by various methods. Therefore, the data amount of the 3D image information can be further reduced. In addition, by extracting a three-dimensional object, in the case of a moving image, once the background image is recorded, it is not necessary to record it again. Only a three-dimensional object (target object) with a small area needs to be stored, and a high compression rate can get.

  Note that encoding is not limited to the above method. For example, the distance information may be provided with a difference in gradation (range of one gradation for the distance information) according to the sense of distance. That is, for example, the background in a two-dimensional image is usually a very long distance, and the distance is often not well understood by human eyes. Therefore, the gradation is roughened in such a region. Conversely, a close-up view is easy for humans to grasp the sense of distance. Therefore, the gradation is made finer in such a region. Thus, the total amount of information can also be compressed by making a difference for each distance in the gradation of distance information.

  Moreover, when extracting a three-dimensional object, you may pay attention to the difference in distance. That is, the point where the change in distance is large is regarded as the boundary of the object. Alternatively, a point where the distance is discontinuous is a boundary, and if it is continuous, it is regarded as a surface. Thereby, the outline of the three-dimensional object can be grasped, and the surface shape can be grasped from the normal line information. When extracting a three-dimensional object, the object is handled as a set of minute planes. If the distance function is linear or the directions of the normals are similar, it can be seen that the region is a plane.

  In addition, when there is motion in the screen between frames, that is, in the case of a moving image, motion compensation coding may be performed. That is, the motion of the three-dimensional object in the frame is estimated, a prediction value is obtained using the estimated motion, and a difference between the prediction result and the actual result is obtained. Then, the obtained difference is stored. More specifically, a three-dimensional object is extracted for every two frames. Next, a similar three-dimensional object is searched between frames. If the object is a plane, the plane information is searched using the representative point as a mark, and the change is calculated. In the case of a curved surface, the curved surface information is searched and the change is calculated. At this time, the search is performed using the feature points on the curved surface and the center of curvature thereof as landmarks. Also, the texture change is the projection of image information from the viewpoint direction on the 3D object (texture pasting), and the position change (change in physical positional relationship such as movement, rotation, enlargement, reduction, etc.) A difference from the actual texture image information is obtained as a predicted value. That is, the background portion overlapping the three-dimensional object is interpolated from the past or future frame, and the background in the state where the object does not exist can be predicted. The amount of information can be compressed by holding the difference between these frames. In this case, the handling is facilitated by considering the difference in units of minute areas rather than the entire object. Using the change information between the two frames obtained in this way, a change in the next frame is predicted, and a difference from the actual value is obtained.

  Furthermore, the prediction of the part that cannot be seen from the field of view of the three-dimensional object is also possible by the following method. That is, the extracted object is compared with a typical shape (basic solid such as a rectangular parallelepiped or a sphere, a straight line, a plane, an infinite circle, etc.), and a similar shape is searched. Then, a typical shape similar to the object is used for predictive image calculation. Alternatively, a change in the distance or normal direction of the surface of the three-dimensional object may be expressed as a function, and a typical shape to be compared may be selected from the functional property. For example, if the change in the distance between adjacent pixels is linear over a certain area, the set can be estimated as a plane. If it is linear over a range with no area, it can be estimated that it is a straight line.

  In addition, when storing the shape and texture for each 3D object, the 3D environment (positional relationship and angle between the viewpoint and the object) where the imaging device is located is also stored, so that the imaging device moves. It is also possible to predict what objects will appear in the field of view when the angle changes.

  Further, the distant view has less change than the close view. Therefore, the time difference between the reference images for prediction may be changed depending on the distance. Further, the expansion / reduction of the object may be predicted in accordance with the distance change (approach or separation) in the time series of the extracted three-dimensional object. In addition, the positional relationship between the time before and after that can be predicted from the transition of the positional relationship between the extracted three-dimensional object and the viewpoint. Then, the predicted three-dimensional object may be rendered, and the difference from the actual image may be stored as information.

[Third Embodiment]
Next explained is an image processing method according to the fourth embodiment of the invention. The present embodiment relates to an imaging device and a playback device that realize the image processing method described in the first and second embodiments.

<Imaging device>
First, the imaging device will be described. The imaging apparatus generates and compresses stereoscopic information of a two-dimensional image according to the image processing method according to the first and second embodiments. FIG. 15 is a block diagram of an imaging apparatus, and an example thereof is a camera. As illustrated, the imaging device 1 includes an image input unit 2, a distance measuring unit 3, a calculation unit 4, and a transmission / storage unit 5.

  The image input unit 2 includes an imaging unit, and captures a target object by the imaging unit to obtain a two-dimensional image. Note that the image input unit 2 itself does not necessarily include the imaging unit, and a two-dimensional image captured by a camera outside the imaging device 1 may be input to the image input unit 2.

  The distance measuring unit 3 measures the distance of the target object. That is, as described in the first embodiment, the distance is measured using focusing, infrared rays, laser, or the like, or a stereo corresponding point search method using a plurality of two-dimensional images given from the image input unit 2 or the like To estimate the distance.

  The calculation unit 4 receives a two-dimensional image from the image input unit 2 and receives a distance from the distance measurement unit 3. Then, the calculation unit 4 generates and compresses stereoscopic information of the captured two-dimensional image. Details of the calculation unit 4 will be described later.

  The transmission / storage unit 5 is a semiconductor memory such as a hard disk drive or a flash memory, for example. In this case, the transmission / accumulation unit 5 includes a communication device such as a LAN, and transmits the stored data to the playback device through these communication devices. Further, the transmission / storage unit 5 may be a memory card that can be inserted into and removed from the imaging apparatus. In this case, the memory card serves as data transmission means and storage means.

<< Details of Calculation Unit 4 >>
Next, the detail of the said calculating part 4 is demonstrated. As shown in the figure, the calculation unit 4 includes a two-dimensional image compression unit 6, an additional information calculation unit 7, and an additional information compression unit 8.

  The two-dimensional image compression unit 6 compresses the two-dimensional image given from the image input unit 2 by a known method. Then, the compressed two-dimensional image is stored in the transmission / accumulation unit 5 as two-dimensional image compression data.

  The additional information calculation unit 7 calculates the distance and the normal direction for the pixels constituting the two-dimensional image. The additional information calculation unit 7 includes a first additional information calculation unit 10, an illuminance difference calculation unit 11, a light source angle difference calculation unit 12, and a second additional information calculation unit 13.

  The first additional information calculation unit 10 calculates the normal direction at the reference point. That is, a reference point is determined based on the distance information given from the distance measuring unit 3, and the process of step S11 described in the first embodiment is performed on the reference point to calculate the normal direction at the reference point.

  The illuminance difference calculation unit 11 calculates the illuminance difference for the two-dimensional image given from the image input unit 2. The illuminance difference calculation unit 12 calculates the illuminance difference between adjacent pixels for each pixel constituting the two-dimensional image.

  The light source angle difference calculation unit 12 obtains an angle difference with the light source at the non-reference point. That is, the pixels constituting the two-dimensional image are sequentially selected, and if the point is a non-reference point, the angle difference from the light source in the normal direction is obtained based on the illuminance difference calculated by the illuminance difference calculation unit 11. Information about whether the selected pixel is a reference point or a non-reference point is given from the first additional information calculation unit 10.

  Based on the distance and normal direction at the reference point given from the first additional information calculation unit 10 and the angle difference given from the light source angle difference calculation unit 12, the second additional information calculation unit 13 Calculate the normal direction. The calculation method of the distance and the normal direction at the non-reference point is as shown in the flow of FIG. 2 described in the first embodiment.

  The additional information compression unit 8 encodes and compresses the distance and normal direction at the reference point and the non-reference point obtained by the additional information calculation unit 7 by the various methods described in the second embodiment. Then, the compressed distance and normal direction are stored in the transmission / accumulation unit 5 as additional information compressed data.

<Reproducing device>
Next, the playback device will be described. The reproduction device expands and reproduces the stereoscopic information of the two-dimensional image according to the image processing method according to the first and second embodiments. FIG. 16 is a block diagram of a playback apparatus, and an example thereof is a personal computer (PC). As illustrated, the playback device 1 includes a transmission / storage unit 21, a calculation unit 22, a display unit 23, and an input unit 24.

  The transmission / storage unit 21 receives and stores the two-dimensional image compressed data and the additional information compressed data transmitted from the transmission / storage unit 5 of the imaging device 1.

  The calculation unit 22 receives the compressed two-dimensional image data and the additional information compressed data from the transmission / storage unit 21, reproduces the two-dimensional image, and generates this stereoscopic image. Details of the calculation unit 22 will be described later.

  The display unit 23 displays the two-dimensional image and the stereoscopic image obtained by the calculation unit.

  The input unit 24 is a mouse or a keyboard, for example, and receives information (for example, a viewpoint, a gazing point, a depth of field, etc.) necessary for reproducing a stereoscopic image.

<< Details of Calculation Unit 22 >>
Next, the detail of the said calculating part 22 is demonstrated. As shown in the figure, the calculation unit 22 includes a two-dimensional image expansion unit 30, an additional information expansion unit 31, a three-dimensional shape restoration unit 32, and a rendering unit 33.

  The two-dimensional image expansion unit 30 decodes the two-dimensional image compression data given from the transmission / accumulation unit 21 by a known method to obtain a two-dimensional image. This two-dimensional image is displayed on the display unit 23.

  The additional information decompression unit 31 decodes the additional information compressed data provided from the transmission / accumulation unit 21 by the method described in the second embodiment, and determines the distances at the reference points and the non-reference points and the directions of the normals. obtain.

  The three-dimensional shape restoration unit 32 generates the three-dimensional shape of the two-dimensional image by associating the distance and the normal direction obtained by the additional information decompression unit 31 with the two-dimensional image obtained by the two-dimensional image decompression unit 30. To do.

  The rendering unit 33 renders the three-dimensional shape obtained by the three-dimensional shape restoration unit 32 according to the information received by the input unit 24 to generate a target object. The object obtained by the rendering unit 33 is displayed on the display unit 23.

  With the above apparatus, the first and second embodiments can be realized. Note that the arithmetic units 4 and 22 in the present embodiment may be realized by software. That is, the imaging device 1 may include a calculation device such as a CPU, and the calculation device may function as the two-dimensional image compression unit 6, the additional information calculation unit 7, and the additional information compression unit 8 by software. The calculation unit 22 is the same.

  Furthermore, it is preferable to separate the imaging device 1 and the playback device 20 as in this embodiment. This is because the camera as the imaging apparatus 1 is normally required to be portable and the camera rarely has a high image processing capability. In this case, since it is necessary to transmit data from the imaging apparatus 1 to the reproduction apparatus 20, it is important to compress the data amount of the three-dimensional image.

  Further, the imaging device 1 and the playback device 20 can be applied as a three-dimensional map creation device. In this case, the calculation unit 4 of the imaging device 1 further includes a 3D map creation unit and a 3D map compression unit. The three-dimensional map creation unit receives distances and normal directions at the reference point and the non-reference point from the second additional information calculation unit 13. That is, stereoscopic information about a two-dimensional image is received. The 3D map creation unit then projects the 2D image into 3D to create 3D map information. At this time, the texture is also integrated. Then, the 3D map compression unit compresses the 3D map information created by the 3D map creation unit and stores it in the transmission / accumulation unit 5 as 3D map information compressed data.

  The calculation unit 22 of the playback device 20 further includes a 3D map decompression unit and a 3D map creation unit. The 3D map decompression unit decompresses the 3D map information compressed data received from the transmission / accumulation unit 21, and based on this, the 3D map creation unit creates a 3D map. Then, in the three-dimensional shape restoration unit 32, the three-dimensional shape in the three-dimensional map is restored.

  As described above, according to the image processing method according to the first to third embodiments of the present invention, in the two-dimensional image, the step of obtaining the normal direction of the reference point and the illuminance difference based on the reference point are obtained. And determining the distance and normal direction of the non-reference point based on the step. Then, the stereoscopic image information of the two-dimensional image is expressed by the two-dimensional image and the distance and normal direction at the reference point and the non-reference point. That is, it is possible to handle a three-dimensional image by directly using stereoscopic information. Therefore, only one two-dimensional image is necessary, and the data amount of the three-dimensional image can be reduced.

  This method will be schematically described again with reference to FIGS. FIG. 17 is a schematic diagram of a two-dimensional image. As shown in the figure, a cylindrical foreground object exists on the background. The hatching in the figure indicates a shadow (brightness). For such a two-dimensional image, a point at which the distance can be measured is set as a reference point. FIG. 18 shows the reference points of the two-dimensional image in FIG. 17 by cross marks. As shown in the figure, the outline of the background and the foreground object is easy to estimate the distance, and easily becomes a reference point. All other areas without cross marks are non-reference points. Then, based on the illuminance difference and the normal direction of the reference point, the normal direction of the non-reference point is obtained. This is shown in FIG. As shown in the figure, it can be seen from the difference in illuminance that the background is three planes of the floor and two walls, and the directions of these normal vectors n1 to n3 can be obtained. It can also be seen from the difference in illuminance that the foreground object has a cylindrical shape, the direction of the normal vector n4 on the upper surface of the cylinder, and the direction of the normal vectors n5, n6, n7,. By knowing the normal line, the distance between the non-reference points can be obtained by the method shown in FIG. 5 described in the first embodiment.

  The following method can be used to calculate the distance and normal direction of the non-reference point. That is, as shown in FIGS. 5 and 6, first, a first angle difference θ1 which is an angle difference between the normal direction of the reference point and the light source is obtained by the illuminance difference stereo method. Further, the second angle difference θ2, which is the angle difference between the normal direction of the non-reference point that can be estimated to be sufficiently close to the reference point and on the same continuous curved surface or plane as the reference point, and the light source is expressed as illuminance. Obtained by the difference stereo method. Next, the normal direction at the non-reference point is obtained based on the normal direction of the reference point and the third angle difference Δθ1 that is the difference between the first angle difference θ1 and the second angle difference θ2. Further, based on the first angle difference θ1 and the second angle difference θ2, a fourth angle difference Δθ2 that is an angle difference from the center of curvature between the reference point and the non-reference point is obtained. Accordingly, the distance between the non-reference points can be obtained based on the distance between the reference points and the fourth angle difference Δθ2.

  As described above, by using the illuminance difference, it is easy to obtain the normal line and distance even for an object with little change on the surface. In the above method, the angle difference between the normal direction of the two points (the reference point and the non-reference point for which the distance and the normal direction are to be obtained based on the reference point) and the light source is expressed as the angle difference in the normal direction. Approximate. Therefore, the angle difference between the line segments connecting the light source and the two points becomes the normal angle error. Therefore, it is sufficient that the two points are close to each other so that this error is within an allowable range. That is, it is possible to determine whether or not the two points are sufficiently close based on the distance from the light source and the accuracy requirement of the normal angle.

  Note that the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the scope of the invention in the implementation stage. Furthermore, the above embodiments include inventions at various stages, and various inventions can be extracted by appropriately combining a plurality of disclosed constituent elements. For example, even if some constituent requirements are deleted from all the constituent requirements shown in the embodiment, the problem described in the column of the problem to be solved by the invention can be solved, and the effect described in the column of the effect of the invention Can be extracted as an invention.

1 is a flowchart of an image processing method according to the first embodiment of the present invention. 6 is a flowchart showing a method for calculating a distance and a normal direction of a non-reference point in the image processing method according to the first embodiment of the present invention. A schematic diagram of a three-dimensional object. The conceptual diagram which shows the calculation method of the distance of a non-reference | standard point and normal direction which concerns on 1st Embodiment of this invention. The conceptual diagram which shows the distance calculation method of the non-reference | standard point which concerns on 1st Embodiment of this invention. It is a schematic diagram of a three-dimensional object, and is a diagram for explaining a method for calculating the distance and normal direction of a non-reference point according to the first embodiment of the present invention. The flowchart of the data compression method which concerns on 2nd Embodiment of this invention. The flowchart of the data compression method which concerns on 2nd Embodiment of this invention. The flowchart of the data compression method which concerns on 2nd Embodiment of this invention. The schematic diagram for demonstrating the data compression method which concerns on 2nd Embodiment of this invention. The flowchart of the object extraction method which concerns on 2nd Embodiment of this invention. It is a schematic diagram of a three-dimensional object, and is a flowchart of the object extraction method according to the second embodiment of the present invention. The flowchart of the object extraction method which concerns on 2nd Embodiment of this invention. It is a schematic diagram of a three-dimensional object, and is a flowchart of the object extraction method according to the second embodiment of the present invention. The block diagram of the imaging device which concerns on 3rd Embodiment of this invention. The block diagram of the reproducing | regenerating apparatus concerning 3rd Embodiment of this invention. The schematic diagram of a two-dimensional image. The schematic diagram which shows a mode that the direction of the normal line in a reference point is calculated | required with the image processing method which concerns on 1st thru | or 3rd Embodiment of this invention. The schematic diagram which shows a mode that the distance in a non-reference | standard point and the direction of a normal line are calculated | required with the image processing method which concerns on 1st thru | or 3rd Embodiment of this invention.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 ... Imaging device, 2 ... Image input part, 3 ... Distance measuring part, 4, 22 ... Operation part, 5, 21 ... Transmission / accumulation part, 6 ... Two-dimensional image compression part, 7 ... Additional information calculation part, 8 ... Additional information compression unit, 10 ... first additional information calculation unit, 11 ... illuminance difference calculation unit, 12 ... light source angle difference calculation unit, 13 ... second additional information calculation unit, 20 ... reproduction device, 23 ... display unit, 24 ... Input unit 30... 2D image decompression unit 31. Additional information decompression unit 32. Solid shape restoration unit 33 33 Rendering unit

Claims (5)

Obtaining a normal direction for a reference point whose distance from the viewpoint is known in a two-dimensional image;
Obtaining a distance from the viewpoint and a normal direction based on an illuminance difference with reference to the reference point for a non-reference point whose distance from the viewpoint is unknown in the two-dimensional image, and 3D image information of the two-dimensional image is represented by a three-dimensional image, the distance and normal direction at the reference point, and the distance and normal direction at the non-reference point. . Obtaining the distance and the normal direction for the non-reference point, obtaining a first angle difference that is an angle difference between the normal direction and the light source at the reference point by an illuminance difference stereo method;
Obtaining a second angle difference which is an angle difference between the normal direction at the non-reference point and the light source by the illuminance difference stereo method;
Obtaining the normal direction at a non-reference point based on the normal direction at the reference point and a third angle difference that is a difference between the first angle difference and the second angle difference;
Obtaining a fourth angle difference that is an angle difference from the center of curvature between the reference point and the non-reference point based on the first angle difference and the second angle difference;
The image processing method according to claim 1, further comprising: calculating the distance of the non-reference point based on the distance of the reference point and the fourth angle difference. The image processing method according to claim 1, wherein the non-reference point for which the distance is obtained is a new reference point. Compressing the two-dimensional image;
Compressing the normal direction at the reference point and the non-reference point by expressing them in polar coordinates;
The image processing method according to claim 1, further comprising: compressing the distance between the reference point and the non-reference point by taking a reciprocal number as a logarithm. After determining the distance and the normal direction at the reference point and the non-reference point,
Extracting a region in which the direction of the normal is similar in the two-dimensional image;
When the extracted area is a plane, a representative point in the plane is determined, the plane is expressed by the distance and the normal direction of the representative point, and when the extracted area is a curved surface, the curved surface The image processing method according to claim 1, further comprising: expressing the curved surface by a curve composed of a radius of curvature and a center of curvature.

Images