JP5834712B2 - Image generation apparatus, image generation method, and program for image generation apparatus - Google Patents

Description

  The present invention relates to an image generation apparatus, an image generation method, and a program for an image generation apparatus that generate an image for a stereoscopic image.

  Conventionally, in order to enable a realistic autostereoscopic view, using two images (stereo images) taken with two cameras, an intermediate position when taken from a position between the two cameras An image is generated and multi-viewed. For example, Patent Document 1 discloses that a first image acquisition unit including a first imaging lens unit that forms an image of a subject and a first imaging element, a second imaging lens unit, and an image formed thereby. A second image acquisition unit having a first lens array unit in which a plurality of lenses that receive the captured image are arranged in an array and a second imaging element, and the second image acquisition unit includes a first image acquisition unit A stereoscopic video imaging device is disclosed that is disposed away from the subject in the horizontal direction when viewed from the subject.

JP 2010-78768 A

  However, in the prior art, a special camera in which a plurality of lenses are arranged in an array is used, and this camera is a camera that is difficult to manufacture, and the cost of equipment is high and difficult to realize. Furthermore, in the prior art, it is necessary to perform clustering of objects with high pixel correlation, and since drawing must be performed from the back when drawing, sorting processing in the depth direction is necessary. The amount was increasing.

  Therefore, the present invention has been made in view of the above-described problems, and an example of the problem is to provide an image generation apparatus that generates an image with a smaller amount of calculation at a low cost. .

In order to solve the above-described problem, the invention according to claim 1 is an image that receives input of a first image captured from a first position and a second image captured from a second position by a plurality of imaging means. From the input means, the hue conversion means for performing a hue conversion for obtaining the hue of each pixel of the first image and the second image, the first image and the second image from the first image and the second image that have been subjected to the hue conversion. Feature point extraction means for extracting feature points of the image, correspondence point search means for finding corresponding feature points in the first image and the second image, and calculating a disparity vector having a distance between the corresponding feature points Disparity vector calculating means, position setting means for setting a third position between the first position and the second position, a positional relationship of the third position with respect to the first position and the second position, and the disparity vector And according to the above From the first image and the second image, and an image generating means for generating a third image when it is captured from said third position, said hue conversion means, according to the hue in the range of values of the hue And converting the image into the hue image having a hue value as an index for the group, and the feature point extracting means determines the direction connecting the first position and the second position as the first image and The scan direction of each image of the second image is set as the feature point, and the corresponding point is extracted in order according to the scan direction. It is characterized in that corresponding feature points are sequentially searched by collating the original image before hue conversion at the position of the feature points .

The invention according to claim 2, in the image generating apparatus according to claim 1, wherein the image generating means, the position of the corresponding feature point in the third image, the positional relationship of the third position It is determined from the corresponding ratio and the parallax vector.

According to a third aspect of the present invention, in the image generation device according to the first or second aspect , the image generation unit is configured to select the third position out of the first position and the second position according to the parallax vector. A third image is generated by using pixel information of the first image or the second image at a position closer to the position.

According to a fourth aspect of the present invention, in the image generating apparatus according to the third aspect , when the image generating means has no pixel according to the parallax vector, the first position and the second position are A third image is generated by using pixel information of the first image or the second image at a position farther from the third position.

According to a fifth aspect of the present invention, in the image generating apparatus according to any one of the first to fourth aspects, the position setting means sets a plurality of third positions, and the image generating means A plurality of third images are generated.

The invention according to claim 6 is an image generation method in which the image generation apparatus generates an image, and the first image captured from the first position and the second position are captured by a plurality of imaging means. An image input step for receiving an input with the second image, a hue conversion step for performing a hue conversion for obtaining a hue of each pixel of the first image and the second image, and the first and second images subjected to the hue conversion A feature point extracting step for extracting feature points of the first image and the second image, a corresponding point searching step for obtaining corresponding feature points in the first image and the second image, and A disparity vector calculating step of calculating a disparity vector having a distance between, a position setting step of setting a third position between the first position and the second position, and the first position and the second position Serial positional relationship between the third position and in accordance with the disparity vector from the first image and the second image, have a, an image generation step of generating a third image when it is captured from said third position, In the hue conversion step, the hue is classified into groups according to a range of the hue value, and converted into the hue image having an index for the group as a hue value. In the feature point extraction step, the hue conversion step The direction connecting the first position and the second position is set as the scan direction of each image of the first image and the second image, the portion where the hue change occurs is extracted as the feature point, and in the corresponding point search step The feature points that appear in order according to the scan direction are ordered, and the original image before hue conversion is collated at the position of the feature points to sequentially correspond. Wherein the searching for the characteristic point.

According to a seventh aspect of the present invention, there is provided an image input means for receiving input of a first image captured from a first position and a second image captured from a second position by a plurality of imaging means. Hue conversion means for performing hue conversion to obtain the hue of each pixel of the first image and the second image, and feature points of the first image and the second image are extracted from the hue-converted first image and second image Feature point extracting means, corresponding point search means for obtaining corresponding feature points in the first image and the second image, disparity vector calculating means for calculating a disparity vector having a distance between the corresponding feature points, Position setting means for setting a third position between the first position and the second position, and the first image according to the positional relationship of the third position with respect to the first position and the second position and the disparity vector, And a second image, to function as an image generating means for generating a third image when it is captured from said third position, said color conversion means are classified into groups according to the hue in the range of values of the hue The feature point extracting means converts the direction connecting the first position and the second position to each of the first image and the second image by converting the index for the group to a hue image having a hue value. The scan direction of the image is extracted, and the portion where the hue change occurs is extracted as the feature point, and the corresponding point search means orders the feature points that appear in order according to the scan direction, and at the position of the feature point , we search for a feature point successively corresponding by performing matching of the original image before color conversion.

  According to the present invention, the input of the first image picked up from the first position by the plurality of image pickup means and the second image picked up from the second position is received, and the hue of each pixel of the first image and the second image is received. The feature points of the first image and the second image are extracted from the first image and the second image subjected to the hue conversion, the corresponding feature points in the first image and the second image are obtained, A disparity vector having a distance between corresponding feature points is calculated, a third position is set between the first position and the second position, and the positional relationship and disparity of the third position with respect to the first position and the second position According to the vector, a third image is generated from the first image and the second image when captured from the third position, so that a special camera is not used and the position of the third position is obtained without depth information. Image data between feature points according to the relationship and the disparity vector. Since draw, at low cost, it can produce images with less amount of calculation.

1 is a block diagram illustrating a schematic configuration example of an image generation system according to an embodiment of the present invention. FIG. 2 is a block diagram illustrating an example of a schematic configuration of image generation in FIG. 1. FIG. 2 is a schematic diagram illustrating an example of a positional relationship between the position of the imaging device in FIG. 1 and the position of a virtual imaging device. 3 is a flowchart illustrating an operation example of the image generation apparatus in FIG. 1. It is a schematic diagram which shows an example of the image imaged with the imaging device of FIG. It is a schematic diagram which shows an example of the scanning direction with respect to the imaged image. It is a schematic diagram which shows an example of the line data which carried out area | region classification by the hue and was indexed. It is a schematic diagram which shows an example of the image by which the hue conversion was carried out. It is a schematic diagram which shows an example of the feature point (reference | standard point) and parallax vector of the imaged image. It is a schematic diagram which shows an example of the reference point of the imaged image, and the reference point of the image when imaged from a virtual imaging device. It is a schematic diagram which shows an example of the relationship of the pixel value around a reference point. It is a schematic diagram which shows an example of the relationship of the pixel value around a reference point.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiment described below is an embodiment when the present invention is applied to an image generation system.

[1. Overview of image generation system configuration and functions]
(1.1 Configuration and Function of Image Generation System 1)

  First, the configuration and outline function of an image generation system according to an embodiment of the present invention will be described with reference to FIG.

  FIG. 1 is a block diagram illustrating a schematic configuration example of an image generation system 1 according to the present embodiment.

  As shown in FIG. 1, the image generation system 1 captures an image of the subject 5 at a predetermined position and an image generation device 10 that generates an image when the image is captured from a virtual position from an image of the subject 5 captured at the predetermined position. Imaging device (an example of imaging means) 20.

  The image generation device 10 includes an image input unit 10a that receives input of image data of a plurality of images captured by the imaging device 20, a hue conversion unit 10b that performs conversion to obtain the hue of each pixel from image data of RGB signals, A feature point extracting unit 10c for extracting feature points of images from the image data subjected to hue conversion, a corresponding point searching unit 10d for obtaining corresponding feature points between images, and a disparity vector having a distance between the corresponding feature points A parallax vector calculating means 10e for calculating a position, a position setting means 10f for setting a virtual position, an image generating means 10g for generating an image captured at the virtual position, and a display unit 13 for displaying the generated image. Have.

  The imaging device 20 includes a left-eye imaging device 20a and a right-eye imaging device 20b, which are examples of imaging means, and constitutes a stereo camera. For example, the imaging device 20a and the imaging device 20b are installed at a predetermined distance from the subject 5 in the horizontal direction. The position where the imaging device 20a is installed is an example of the first position, and the position where the imaging device 20b is installed is an example of the second position. The imaging devices 20a and 20b are a still camera that captures still images or a video camera that captures moving images, and have the same performance with respect to resolution, color gradation, and the like. The outputs of the imaging devices 20a and 20b are digital outputs, but an analog output imaging device has an A / D converter and outputs a digital signal.

  The imaging device 20a captures the subject 5 from the left side and outputs an image for the left eye (an example of a first image captured at the first position). The imaging device 20b captures the subject 5 from the right side and outputs an image for the right eye (an example of a second image captured at the second position).

  The imaging device 20 a and the imaging device 20 b are connected to the image input unit 10 a of the image generation device 10.

(1.2 Configuration and Function of Image Generation Device 10)
Next, the configuration and function of the image generation apparatus 10 will be described with reference to FIG.
FIG. 2 is a block diagram illustrating an example of a schematic configuration of the image generation apparatus 10.

  As illustrated in FIG. 2, the image generation device 10 is, for example, a personal computer or a server device, and includes a communication unit 11, a storage unit 12, a display unit 13, an operation unit 14, and an input / output interface unit 15. The system control unit 16 is provided. The system control unit 16 and the input / output interface unit 15 are connected via a system bus 17.

  The communication unit 11 transmits and receives data and signals to and from the outside of the image generation apparatus 10. For example, it is used for network connection.

  The storage unit 12 includes, for example, a hard disk drive and stores an operating system, a control program, an image processing program, and the like.

  The display unit 13 is configured by, for example, a liquid crystal display element or an EL (Electro Luminescence) element. The display unit 13 displays the image of the subject 5 captured by the imaging device 20 and the generated image.

  The operation unit 14 is configured by, for example, a keyboard and a mouse.

  The input / output interface unit 15 is a part directly connected to the output ends of the imaging device 20a and the imaging device 20b, and is an interface that realizes the function of the image input means 10a. This is an interface between the imaging device 20 (20a, 20b), the communication unit 11, the storage unit 12, and the system control unit 16.

  The system control unit 16 includes, for example, a CPU 16a, a ROM 16b, and a RAM 16c. In the system control unit 16, the CPU 16 a reads out and executes various programs stored in the ROM 16 b, the RAM 16 c, and the storage unit 12. For example, the system control unit 16 executes an image processing program, and includes a hue conversion unit 10b, a feature point extraction unit 10c, a corresponding point search unit 10d, a parallax vector calculation unit 10e, a position setting unit 10f, an image generation unit 10g, and the like. Realize the function.

(1.3 Position of virtual imaging device)
Next, the position (virtual position) of the virtual imaging device will be described with reference to FIG.
FIG. 3 is a schematic diagram illustrating an example of a positional relationship between the positions of the imaging devices 20a and 20b and the position of the virtual imaging device.

  As shown in FIG. 3, the imaging device 20a and the imaging device 20b are installed at a distance L in the x direction (for example, the horizontal direction). Three virtual imaging devices 25 are set between the imaging device 20a and the imaging device 20b. The imaging device 25a, the imaging device 25b, and the imaging device 25c are set at equal intervals. Assuming that the first position 0 of the installation position of the imaging device 20a is the second position L of the installation position of the imaging device 20b, as an example of the third position, in the case of the imaging device 25a, the position L / 4, the imaging device 25b. In this case, the position L / 2 is set, and in the case of the imaging device 25c, the position 3L / 4 is set. Here, an image captured by the virtual imaging device 25 is an intermediate parallax image (an example of a third image).

[2. Operation of image generation system]
Next, the operation of the image generation system according to the embodiment of the present invention will be described with reference to FIGS.
FIG. 4 is a flowchart illustrating an operation example of the image generation apparatus 10. FIG. 5 is a schematic diagram illustrating an example of an image captured by the imaging device 20. FIG. 6 is a schematic diagram illustrating an example of a scan direction with respect to a captured image. FIG. 7 is a schematic diagram illustrating an example of line data that has been clustered and indexed. FIG. 8 is a schematic diagram illustrating an example of a hue-converted image. FIG. 9 is a schematic diagram illustrating an example of feature points (reference points) and parallax vectors of a captured image. FIG. 10 is a schematic diagram illustrating an example of a reference point of a captured image and a reference point of an image when captured from a virtual imaging device. 11 and 12 are schematic diagrams illustrating an example of the relationship between pixel values around the reference point.

Here, the principle of the method for generating the intermediate parallax image will be briefly described.
In the proposed method, in generating an intermediate parallax image, instead of calculating and interpolating the pixel value, the movement distance of the object by the movement of the viewpoint is used, and the trajectory of the movement between the images is traced in reverse. An intermediate parallax image is generated using only the pixels present in the two original images 30a and 30b.

  Also, feature points corresponding to each other are extracted from the two original images 30a and 30b, and a disparity vector is calculated from the feature points. In the generated intermediate parallax image, it is considered that pixels having a value approximate to the feature point of the original image always exist between the feature points. For example, if virtual imaging devices 25a, 25b, and 25c are installed at equal intervals as shown in FIG. 3, the positional relationship is obtained by dividing the movement distance (the length of the parallax vector) by 4, and the intermediate It can be seen at what interval the pixels having the same value as the feature points in the parallax image are arranged. Therefore, a reference point corresponding to the feature point of the original images 30a and 30b in the intermediate parallax image is obtained according to the positional relationship and the parallax vector. By arranging pixels around the reference point, an intermediate parallax image can be generated. The reference point is an example of a corresponding feature point in the third image.

  First, as an operation of the image generation system, the system control unit 16 of the image generation apparatus 10, as illustrated in FIG. 3, uses the positions of virtual imaging devices 25 a, 25 b, and 25 c (for example, L / 4, L / 2, 3L / 4) are set in advance. Thus, the system control unit 16 of the image generation device 10 functions as an example of the position setting unit 10f that sets the third position between the first position and the second position.

  Then, the system control unit 16 of the image generation apparatus 10 sets in advance in the storage unit 12 a storage area for generating an intermediate parallax image corresponding to the positions of the virtual imaging devices 25a, 25b, and 25c.

  Next, as illustrated in FIG. 4, the image generation apparatus 10 receives input of left and right images (step S <b> 1). Specifically, as illustrated in FIG. 5, the system control unit 16 of the image generation apparatus 10 receives an image 30a (an example of a first image) obtained by imaging the subject 5 from the left side, and captures the image from the right side. The received image 30b (an example of the second image) is received from the imaging device 20b. The image 30a is an image captured from the position of the imaging device 20a (an example of the first position), and the image 30b is an image captured from the position of the imaging device 20b (an example of the second position). The image data is RGB signal data. As described above, the system control unit of the image generation apparatus 10 receives the input of the first image captured from the first position by the plurality of imaging units and the second image captured from the second position. Functions as an example.

  Next, the image generation device 10 acquires image data for one line of each image (step S2). Specifically, as illustrated in FIG. 6, the system control unit 16 of the image generation apparatus 10 scans the captured images 30 a and 30 b in the x direction and acquires image data for one line. The x direction in the scan direction is an example of a direction connecting the first position and the second position.

Next, the image generation device 10 performs HSV conversion on the RGB signals (step S3). Specifically, the system control unit 16 of the image generation apparatus 10 obtains the hue Mxy of each pixel for each pixel of one line of the data of each image of the RGB signal according to the following formula.

  Here, the R, G, and B values in the image data of the RGB signal are in the range of 0.0 to 1.0, with 0.0 as the minimum amount and 1.0 as the maximum value. Among the three values of R, G, and B, the maximum value is MAX and the minimum value is MIN. When R = G = B (black, white, gray), Mxy is not calculated and a specific value is assigned. As described above, the system control unit 16 of the image generation apparatus 10 functions as an example of the hue conversion unit 10b that performs the hue conversion for obtaining the hue of each pixel of the first image and the second image.

Next, the image generation apparatus 10 performs hue grouping (step S4). Specifically, the system control unit 16 of the image generation apparatus 10 segments the hue Mxy values by grouping them, obtains Mxy at each point, and performs hue grouping. A grouping of Mxy is defined as Axy.

  In this example, the number of hue groupings is six. In addition, since there are groups other than colors such as R = G = B (black, white, gray), there are a total of 7 groups. Then, as shown in FIG. 7, in each of the images 30a and 30b, cluster index information is input to the line data subjected to HSV conversion. The index is a numerical value from 0 to 6. “0” for colors other than R = G = B, “1” for group HY, “2” for group HG, “3” for group HC, group HB “4” is assigned to each pixel, “5” is assigned to each group HM, and “6” is assigned to each pixel. As described above, the system control unit 16 of the image generation apparatus 10 classifies the hues into groups according to the range of the hue values, and an example of a hue conversion unit that converts the hues into an image of hues using the indices for the groups as hue values. Function as. Note that the number of groups can be further subdivided.

  Here, when hues are obtained for all lines and the hues are grouped, hue images 32a and 32b are obtained for the images 30a and 30b, as shown in FIG.

  Next, the image generation apparatus 10 performs a boundary search of hue groups (step S5). Specifically, as shown in FIG. 7, the system control unit of the image generation apparatus 10 searches for the position of the boundary where the index value changes in the direction of the arrow in each of the images 30a and 30b. Thereby, the edge information of each image 30a, 30b is obtained.

  Next, the image generation device 10 specifies the position of the feature point (step S6). Specifically, as illustrated in FIG. 7, the system control unit 16 of the image generation apparatus 10 includes, in each image 30a, the pixel 40a at the boundary position where the hue index has changed from “0” to “2”. The pixel 41a at the boundary position changed from “2” to “4” and the pixel at the boundary position changed from “4” to “1” are specified as the feature point pixels. Similarly, the system control unit 16 of the image generation apparatus 10 changes the pixel 40b at the boundary position where the hue index changes from “0” to “2” in each image 30b, and changes from “2” to “4”. The pixel 41b at the boundary position is specified as the pixel of the feature point. In this way, a portion having a certain difference in hue value is selected as a feature point. Further, the system control unit 16 of the image generation apparatus 10 functions as an example of a feature point extraction unit 10c that extracts feature points of the first image and the second image from the first image and the second image that have been subjected to hue conversion. In addition, the system control unit 16 of the image generation apparatus 10 functions as an example of a feature point extraction unit that extracts a portion where a hue change has occurred as a feature point.

  Next, the image generation device 10 searches for a corresponding feature point (step S7). Specifically, the system control unit 16 of the image generation apparatus 10 orders the feature points that appear in order according to the scan direction in the line data of the image 30a and the line data of the image 30b, and sets the feature point position ( The corresponding feature points are searched sequentially by performing pattern matching of the original image (before hue conversion). For example, the feature point of the pixel 40a and the feature point of the pixel 40b correspond to each other, and the feature point of the pixel 41a and the feature point of the pixel 41b correspond to each other. Further, as shown in FIG. 9, it is assumed that the feature point of the pixel 42a corresponds to the feature point of the pixel 42b. As described above, the system control unit 16 of the image generation apparatus 10 functions as an example of the corresponding point search unit 10d that obtains corresponding feature points in the first image and the second image. If there is no corresponding point, the feature point is not handled as a feature point.

  Next, the image generation device 10 calculates a parallax vector (step S8). Specifically, the system control unit 16 of the image generation apparatus 10 calculates a parallax vector 50 in the scan direction (x direction) as illustrated in FIG. 9. If the parallax vector 50 is a feature point pixel 42a of the image 30a and a corresponding feature point pixel 42b of the image 30b, the (left eye) image 30a is used as a reference, for example, the feature point pixel 42a of the image 30a in the x direction With reference to the position “4”, the distance (12 pixels) (the length of the parallax vector) to the position “16” in the x direction of the pixel 42b of the feature point of the (right eye) image 30b is provided. In the process of calculating the parallax vector 50, the system control unit 16 of the image generation apparatus 10 calculates the distance (length corresponding to 12 pixels) between the corresponding feature points based on the extracted feature points. As described above, the system control unit 16 of the image generation apparatus 10 functions as an example of the disparity vector calculation unit 10e that calculates the disparity vector having the distance between the corresponding feature points.

  In FIG. 9, for convenience, it is assumed that only line data of 1 to 4 lines of the image 30a and the image 30b are shown, and the second line is subjected to data processing. Furthermore, in the intermediate parallax image 35a captured from the virtual imaging device 25a, the intermediate parallax image 35b captured from the virtual imaging device 25b, and the intermediate parallax image 35c captured from the virtual imaging device 25c. Also, only line data of 1 to 4 lines are shown. In FIG. 9, an arrow indicates the moving direction of the feature point when the viewpoint moves from the left side to the right side or from the right side to the left side.

  Next, the image generation device 10 determines the position of the reference point of the intermediate parallax image from the relationship between the parallax vector and the parallax (step S9). As shown in FIG. 10, the system control unit 16 of the image generation apparatus 10 includes the positions of the virtual imaging devices 25a, 25b, and 25c (for example, L / 4, L / 2, and 3L / 4) and the imaging device 20a. Of the reference point in the second line according to the ratio (the positional relationship of the third position with respect to the first position and the second position) between the distance L and the imaging device 20b and the length of the parallax vector 50 (12 pixels) 45a, 45b and 45c are obtained. As described above, the system control unit 16 of the image generation apparatus 10 linearly calculates the positions of the feature points of the left and right images using the disparity vector 50, so that the position of the feature point of the intermediate parallax image, that is, the reference point Determine the position.

  For example, the system control unit 16 of the image generation apparatus 10 sets the disparity vector 51 (length 3 pixels) obtained by multiplying the disparity vector 50 by the ratio of the length L / 4 to the length L, and the feature point pixel 42a. Based on the position, the position of the pixel 45a of the reference point of the intermediate parallax image 35a is obtained. In addition, the system control unit 16 of the image generation apparatus 10 sets the disparity vector 52 (6 pixels in length) obtained by multiplying the disparity vector 50 by the ratio of the length L / 2 and the length L, and the feature point pixel 42a. Based on the position, the position of the pixel 45b of the reference point of the intermediate parallax image 35b is obtained. Further, the system control unit 16 of the image generation apparatus 10 has a parallax vector 53 (length 3 pixels) obtained by reversing the direction by multiplying the parallax vector 50 by the ratio of the length 3L / 4 and the length L, and the characteristics. Based on the position of the point pixel 42b, the position of the reference point pixel 45c of the intermediate parallax image 35c is obtained. As described above, the system control unit 16 of the image generation device 10 functions as an example of an image generation unit that determines the position of the corresponding feature point in the third image from the ratio and the disparity vector corresponding to the positional relationship of the third position. To do.

  Next, the image generation device 10 generates a one-line intermediate parallax image (step S10). The system control unit 16 of the image generation apparatus 10 basically has an image with a smaller shift amount of feature points due to movement of the viewpoint, that is, the image pickup apparatuses 20a and 20b closer to the virtual image pickup apparatuses 25, 25b, and 25c. The intermediate parallax images 35a, 35b, and 35c are generated using the pixel values. For example, as illustrated in FIG. 10, the system control unit 16 of the image generation apparatus 10 uses the pixel value “p” of the image 30a of the left eye, the pixel value of the pixel 45a of the second line of the intermediate parallax image 35a, Then, the pixel value of the pixel 45b of the second line of the intermediate parallax image 35b is obtained and rendered. On the other hand, the system control unit 16 of the image generation apparatus 10 obtains and renders the pixel value of the pixel 45c of the second line of the intermediate parallax image 35c using the pixel value “p ′” of the image 30b of the right eye. Note that the shift amount of the pixel 45a from the pixel 42a of the left-eye image 30a is three pixels, the shift amount of the right-eye image 30b from the pixel 42b is nine pixels, and the shift from the left-eye image 30a is greater. The shift amount is small.

  Then, the system control unit 16 of the image generation apparatus 10 obtains pixel values other than the reference point pixels 45a, 45b, and 45c. The system control unit 16 of the image generation apparatus 10 stores the obtained pixel values in the corresponding pixel locations in the storage area for generating the intermediate parallax images 35a, 35b, and 35c, and the intermediate parallax images 35a and 35b. , 35c.

  Here, as a property of the stereo image, the right image and the left image are obtained by translating the same image. Considering the pixels around the reference point, since the reference point is a feature point, the pixels adjacent to the feature point are considered to have an approximate relationship with the left eye image 30a and the right eye image 30b. As shown in FIG. 11, when focusing on the pixel 43a (pixel value “o”) adjacent to the reference point pixel 42a (pixel value “p”) of the left-eye image 30a, the pixel of the reference point of the right-eye image 30b. The pixel 43b (pixel value “o ′”) adjacent to 42b (pixel value “p ′”) has an approximate pixel value with respect to the pixel 43a.

  By using this relationship, one of the pixel value “o” and the pixel value “o ′” is used for the pixel adjacent to the reference point pixel in the intermediate parallax images 35a, 35b, and 35c. Thus, the pixel values of the pixels around the reference point can be interpolated. Whether to use one of the pixel value “o” and the pixel value “o ′” is determined depending on which of the feature points of the left-eye image 30a or the right-eye image 30b has a smaller shift amount.

  Accordingly, the system control unit 16 of the image generation apparatus 10 sets the pixel value of the pixel 46a of the intermediate parallax image 35a to the pixel value “o” of the pixel 43a of the left-eye image 30a according to the parallax vector 53, as shown in FIG. Ask from. Then, the system control unit 16 of the image generation device 10 obtains the pixel value of the pixel 46b of the intermediate parallax image 35b from the pixel value “o” of the pixel 43a of the left-eye image 30a, and the pixel 46c of the intermediate parallax image 35c. The value is obtained from the pixel value “o ′” of the pixel 43b of the image 30b of the right eye. Further, as shown in FIG. 12, the system control unit 16 of the image generation apparatus 10 interpolates the pixels of the line by applying to the pixels adjacent to the pixels 46 a, 46 b, and 46 c.

  As shown in FIG. 12, in the intermediate parallax image, as in the case of the pixel 47a of the intermediate parallax image 35a, when attention is paid to the end of the image, even if the pixel exists in the right eye image 30b, There is a pixel area that does not exist. In this case, the system control unit 16 of the image generation apparatus 10 uses the pixel value “l ′” of the pixel 44b of the right-eye image 30b, and follows the disparity vector 54 starting from the right-eye image 30b side to generate an intermediate parallax image. The pixel value “l ′” of the pixel 47a of 35a is obtained. The system control unit 16 of the image generation apparatus 10 similarly obtains the pixel values of the pixel 47b of the intermediate parallax image 35b and the pixel 47c of the intermediate parallax image 35c. As described above, when there is no pixel of the images 30a and 30b that are closer to each other according to the parallax vector, the system control unit 16 of the image generation device 10 determines the pixel values of the pixels of the images 30b and 30a that have the larger shift amount. Using this, the pixel value of the pixel of the intermediate parallax image is obtained.

  Then, the system control unit 16 of the image generation device 10 obtains pixel values for all the pixels of the predetermined line data of the intermediate parallax images 35a, 35b, and 35c, and the intermediate parallax images 35a and 35b in the storage unit 12. , 35c are stored in the storage area.

  Next, the image generating apparatus 10 determines whether or not it can move to the next line (step S11). Specifically, the system control unit 16 of the image generation apparatus 10 determines whether or not it is the last line of the images 30a and 30b. When the processing of the second line is completed, the system control unit 16 of the image generation apparatus 10 moves the processing target to the next line because it can move to the next third line (step S11; YES). Return to step S2.

  As described above, the system control unit 16 of the image generation device 10 is captured from the first position and the second image according to the positional relationship of the third position with respect to the first position and the second position and the disparity vector, from the third position. Functions as an example of the image generation means 10g for generating the third image in the case of Further, the system control unit 16 of the image generation device 10 uses the information on the pixels of the first image or the second image at the position closer to the third position out of the first position and the second position according to the parallax vector. It functions as an example of image generation means for generating an image. Further, the system control unit 16 of the image generation device 10, when there is no pixel according to the parallax vector, the pixel of the first image or the second image at a position farther from the third position among the first position and the second position. It functions as an example of an image generation unit that generates the third image using the information.

  When the processing of the last line is finished (step S11; NO), the image generation device 10 displays an intermediate parallax image (step S12). Specifically, the system control unit 16 of the image generation apparatus 10 reads the intermediate parallax images from the storage unit 12 and displays the intermediate parallax images 35a, 35b, and 35c on the display unit 13.

  As described above, according to the present embodiment, the first image 30a captured from the first position (x = 0) and the second position (x = L) by the imaging devices 20a and 20b which are examples of the plurality of imaging units. From the first image and the second image obtained by receiving the input to the second image 30b captured from the first image and performing a hue conversion for obtaining the hue of each pixel of the first image and the second image. A feature point of the second image is extracted, a corresponding feature point is obtained in the first image and the second image, a disparity vector 50 having a distance between the corresponding feature points is calculated, and the first position and the second position Are set to the third position (position L / 4, position L / 2, position 3L / 4), and the first image is determined according to the positional relationship of the third position with respect to the first position and the second position and the disparity vector. And the second image, the third image taken from the third position By generating 5a, 35b, and 35c, a special camera is not used, and image data between feature points is drawn according to the positional relationship of the third position and the disparity vector without depth information. An image can be generated with a smaller amount of calculation.

  In addition, since the imaging devices 20a and 20b used in the present embodiment are ordinary still cameras or video cameras, the present embodiment is a technique that is easy to realize. In the present embodiment, the imaging devices 20a and 20b may perform synchronous shooting when shooting moving images.

  Further, when extracting feature points according to the change in hue, the feature points are obtained from the hue, so that there is no influence of luminance.

  In addition, when hues are classified into groups according to the range of hue values and converted into hue images with the index for the group as a hue value, hue conversion using HSV conversion is performed to perform hue conversion. By doing so, the boundary of the hue group is easily extracted. In addition, since processing is performed for each line, clustering at the pixel matrix level is not necessary, and processing is easy. In the image generation apparatus 10, system requirements such as necessary memory and calculation amount can be reduced, and hardware can be easily realized.

  When the scan direction (x direction) of each of the first image 30a and the second image 30b is a direction connecting the first position (x = 0) and the second position (x = L), the scan direction; The direction of the parallax vector 50 (installation phrase of the imaging device 20a and the imaging device 20b) is parallel, and the parallax vector 50 is easily obtained. That is, the system control unit 16 of the image generation apparatus 10 can obtain the disparity vector 50 only by obtaining the distance between the corresponding feature points (the length of the disparity vector).

  The system control unit 16 of the image generation device 10 determines the position of the corresponding feature point in the third image (each intermediate parallax image 35a, 35b, 35c) from the ratio and the parallax vector corresponding to the positional relationship of the third position. In this case, it becomes easy to determine a reference point as a reference for drawing the intermediate parallax images 35a, 35b, and 35c.

  In accordance with the parallax vector 50, the system control unit 16 of the image generation device 10 obtains the third image from the pixels of the first image 30a or the second image 30b at a position closer to the third position among the first position and the second position. In the case of generation, since drawing is performed using pixels of an image that is positioned closer to the intermediate parallax image, a more natural image can be generated.

  When there is no pixel according to the parallax vector 50, the system control unit 16 of the image generation device 10 has the first image or the second image at a position farther from the third position among the first position and the second position. When the third image is generated using the pixel information, one of the imaging devices can generate an image that is an edge of the image and is not captured.

  When the system control unit 16 of the image generation device 10 sets a plurality of third positions and generates a plurality of third images (intermediate parallax images 35a, 35b, and 35c), a plurality of intermediate parallaxes can be easily obtained from multiple eyes. Images 35a, 35b, and 35c can be generated.

  When the image 30a and the image 30b are not at the same resolution, the system control unit 16 of the image generation apparatus 10 may perform the processing from step S1 after matching the resolutions of the image 30a and the image 30b.

  The direction for obtaining the feature points is not limited to the x direction of the scan direction, and may be the y direction.

  Moreover, the imaging device 20a and the imaging device 20b may be installed in directions other than the horizontal direction (x direction). For example, when installed in the vertical direction (y direction), the scan direction is preferably the y direction.

  In addition, when the system control unit 16 of the image generation apparatus 10 obtains feature points, an edge may be obtained using a differential mask with respect to an image subjected to hue conversion.

  In addition, the system control unit 16 of the image generation apparatus 10 evaluates the degree of matching between feature points by using a matching method such as SSD (Sum of Squared Difference), SAD (Sum of Absolute Difference), and the like. You may ask for it. For example, the system control unit 16 of the image generation apparatus 10 performs pixel color matching in the line direction of the first image 30a and the second image 30b. In addition, the system control unit 16 of the image generation apparatus 10 may obtain an image region of the second image 30b that matches an image region including the feature point of the first image 30a, and obtain a corresponding feature point. Pattern matching may be performed on the original image before hue conversion.

  Furthermore, the present invention is not limited to the above embodiments. Each of the embodiments described above is an exemplification, and any configuration that has substantially the same configuration as the technical idea described in the claims of the present invention and has the same operational effects can be used. It is included in the technical scope of the present invention.

1: Image generation system 10: Image generation device 20 (20a, 20b): Imaging device (imaging means)
30 (30a, 30b): image (first image, second image)
35 (35a, 35b, 35c): intermediate parallax image (third image)
40a, 40b, 41a, 41b, 42a, 42b, 45a, 45b, 45c: feature point pixels (feature points)
50: Disparity vector

Claims (7)

Image input means for receiving inputs of a first image picked up from a first position and a second image picked up from a second position by a plurality of image pickup means;
Hue conversion means for performing hue conversion to obtain the hue of each pixel of the first image and the second image;
Feature point extracting means for extracting feature points of the first image and the second image from the hue-converted first image and second image;
Corresponding point search means for obtaining corresponding feature points in the first image and the second image;
Disparity vector calculating means for calculating a disparity vector having a distance between the corresponding feature points;
Position setting means for setting a third position between the first position and the second position;
According to the positional relationship of the third position with respect to the first position and the second position and the parallax vector, a third image when captured from the third position is generated from the first image and the second image. Image generating means;
Equipped with a,
The hue conversion means classifies the hue into groups according to a range of the hue value, converts the hue into an image of the hue with the index for the group as a hue value ,
The feature point extracting means sets a direction connecting the first position and the second position as a scan direction of each image of the first image and the second image, and a portion where the hue change occurs as the feature point Extract and
The corresponding point search means sequentially orders the feature points that appear in order according to the scan direction, and sequentially searches for corresponding feature points by collating the original image before hue conversion at the position of the feature points . An image generation apparatus characterized by that. The image generation apparatus according to claim 1,
The image generation apparatus, wherein the image generation unit determines the position of the corresponding feature point in the third image from a ratio corresponding to the positional relationship of the third position and the parallax vector. In the image generation device according to claim 1 or 2 ,
In accordance with the parallax vector, the image generation unit uses the pixel information of the first image or the second image at a position closer to the third position out of the first position and the second position to generate a third image. An image generation apparatus characterized by generating. The image generation apparatus according to claim 3 .
When there is no pixel according to the parallax vector, the image generation unit obtains information on the pixel of the first image or the second image at a position farther from the third position among the first position and the second position. An image generation apparatus characterized in that a third image is generated using the third image. In the image generation device according to any one of claims 1 to 4 ,
The position setting means sets a plurality of third positions;
The image generation device, wherein the image generation means generates a plurality of third images. An image generation method in which an image generation device generates an image,
An image input step for receiving input of a first image captured from a first position and a second image captured from a second position by a plurality of imaging means;
A hue conversion step for performing a hue conversion to obtain a hue of each pixel of the first image and the second image;
A feature point extracting step of extracting feature points of the first image and the second image from the first image and the second image subjected to the hue conversion;
A corresponding point search step for obtaining corresponding feature points in the first image and the second image;
A disparity vector calculating step of calculating a disparity vector having a distance between the corresponding feature points;
A position setting step of setting a third position between the first position and the second position;
According to the positional relationship of the third position with respect to the first position and the second position and the parallax vector, a third image when captured from the third position is generated from the first image and the second image. An image generation step;
I have a,
In the hue conversion step, the hue is classified into a group according to a range of the hue value, and converted into an image of the hue with the index for the group as a hue value;
In the feature point extraction step, a direction connecting the first position and the second position is a scan direction of each image of the first image and the second image, and a portion where the hue change occurs is the feature point. Extract and
In the corresponding point search step, the feature points that appear in order according to the scanning direction are ordered, and corresponding feature points are sequentially searched by collating the original image before hue conversion at the position of the feature points . An image generation method characterized by the above. Computer
Image input means for receiving inputs of a first image picked up from a first position and a second image picked up from a second position by a plurality of image pickup means;
Hue conversion means for performing hue conversion to obtain the hue of each pixel of the first image and the second image;
Feature point extraction means for extracting feature points of the first image and the second image from the hue-converted first image and second image;
Corresponding point search means for obtaining corresponding feature points in the first image and the second image;
Disparity vector calculating means for calculating a disparity vector having a distance between the corresponding feature points;
Position setting means for setting a third position between the first position and the second position, and the first position according to the positional relationship of the third position with respect to the first position and the second position and the parallax vector, From the image and the second image, function as image generation means for generating a third image when captured from the third position ,
The hue converting means classifies the hue into groups according to a range of the hue value, converts the hue into an image of the hue having an index for the group as a hue value,
The feature point extracting means sets a direction connecting the first position and the second position as a scan direction of each image of the first image and the second image, and a portion where the hue change occurs as the feature point Extract and
The corresponding point search means sequentially orders the feature points that appear in order according to the scan direction, and searches for the corresponding feature points sequentially by collating the original image before hue conversion at the position of the feature points. A program for an image generation apparatus.