JP2018010359A - Information processor, information processing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information processor capable of finding corresponding points between images with high precision and high density while suppressing calculation costs.SOLUTION: The present invention relates to an information processor having: acquisition means of acquiring a first single-viewpoint image and a second single-viewpoint image obtained by imaging the same subject from different viewpoints; generation means for generating a plurality of feature quantity maps respectively for the first single-viewpoint image and second single-viewpoint image by performing processing for applying a filter for detecting a specific structure in an image on the first single-viewpoint image and second single-viewpoint image in stages while changing filters; and search means for searching for corresponding points of the first single-viewpoint image and second single-viewpoint image based upon the plurality of generated feature quantity maps, filters that the generation means applies in the respective stages being a plurality of filters which are different from one another.SELECTED DRAWING: Figure 3

Description

  The present invention relates to a technique for searching for corresponding points between different viewpoint images.

  There is a technique for acquiring information about the distance and shape of a subject using a plurality of images (multi-viewpoint images) when the same subject is viewed from different viewpoints. There is also a technique for estimating the position and orientation of a camera using a multi-viewpoint image. Furthermore, there is a technique for synthesizing a plurality of images for the purpose of creating a panoramic image, reducing noise, and super-resolution. In these techniques, it is essential to search for corresponding points (corresponding points) between a plurality of images.

  Patent Document 1 discloses a method for searching for corresponding points between two input images using a neural network. In Patent Document 1, each of the two input images is divided into a plurality of rectangular regions, a feature amount vector is calculated for each of the divided rectangular regions, and the first input image is calculated based on the calculated feature amount vector. The process of searching for the rectangular area of the second input image corresponding to the rectangular area is repeated.

JP 2009-205553 A

  In Patent Document 1, when a feature vector for each divided rectangular area is calculated for each of two input images, learning using a neural network is performed for each input image, so that the calculation cost is high. In addition, since the corresponding points are searched in units of divided rectangular areas, the density of the corresponding points is low. If the density of corresponding points is low, the matching accuracy of corresponding points is inevitably lowered. In order to increase the density of corresponding points, it is necessary to divide the input image more finely and repeat the generation of feature values by learning using a neural network, which increases the calculation cost even more. As described above, Patent Document 1 has a problem that it is difficult to obtain corresponding points between input images with high accuracy and high density while suppressing calculation cost.

  Therefore, an object of the present invention is to provide an information processing apparatus capable of obtaining corresponding points between images with high accuracy and high density while suppressing calculation cost.

  The present invention provides acquisition means for acquiring a first single-viewpoint image and a second single-viewpoint image obtained by imaging the same subject from different viewpoints, the first single-viewpoint image, and the second single-viewpoint image. A process of applying a filter for detecting a specific structure in the image to each of the viewpoint images is performed step by step by changing the filter, so that the first single-viewpoint image and the second single-viewpoint image are processed. A creation unit that creates a plurality of feature quantity maps for each of the viewpoint images, and searches for corresponding points between the first single-viewpoint image and the second single-viewpoint image based on the created feature quantity maps The information processing apparatus is characterized in that the filter applied by the creating means at each stage is a plurality of different filters.

  According to the present invention, it is possible to obtain corresponding points between images with high accuracy and high density while suppressing calculation cost.

1 is a block diagram illustrating a hardware configuration of an information processing apparatus according to a first embodiment. 1 is a block diagram illustrating a functional configuration of an information processing apparatus according to a first embodiment. 7 is a flowchart showing the flow of processing by the information processing apparatus according to the first embodiment. Block diagram showing the functional configuration of the feature quantity generator Flow chart showing the flow of processing by the feature quantity generator Block diagram showing the functional configuration of the feature quantity generator Flow chart showing the flow of processing by the feature quantity generator The figure which shows the input image used in Example 1 The figure which shows the filter used in Example 1 The figure which shows the result of parallax estimation FIG. 2 is a block diagram illustrating a functional configuration of an information processing apparatus according to a second embodiment. 7 is a flowchart showing a flow of processing by the information processing apparatus according to the second embodiment.

  Hereinafter, exemplary embodiments of the present invention will be described by way of example with reference to the drawings. However, the relative arrangement of the constituent elements described below, the device shape, and the like are merely examples, and are not intended to limit the scope of the present invention only to them. It should be understood that within the scope of the present invention, the embodiments described below are appropriately modified and improved within the scope of the present invention based on the ordinary knowledge of those skilled in the art. It is.

[Example 1]
In this embodiment, a case where a parallax map is created from a multi-viewpoint image (a plurality of images) will be described. As a camera for acquiring multi-viewpoint images, a camera (such as a plenoptic camera or a multi-lens camera) that can acquire multi-viewpoint images by simultaneously capturing a subject with a single camera, or a properly installed camera Multiple cameras may be used. Alternatively, a multi-viewpoint image may be acquired by capturing a subject while moving one camera. In the following, each viewpoint image included in the multi-viewpoint image is referred to as a single viewpoint image.

<Configuration of information processing device>
Hereinafter, the configuration of the information processing apparatus according to the first embodiment will be described. FIG. 1 is a block diagram illustrating an example of a hardware configuration of the information processing apparatus according to the first embodiment. The information processing apparatus 100 (hereinafter abbreviated as processing apparatus 100) in the first embodiment includes a CPU 101, a RAM 102, a ROM 103, a secondary storage device 104, an input interface (hereinafter abbreviated as IF) 105, and an output IF 106. Composed. These components are connected to each other by a system bus 107. The processing device 100 is connected to the external storage device 108 and the operation unit 110 via the input IF 105, and is connected to the external storage device 108 and the display device 109 via the output IF 106.

  The CPU 101 executes a program stored in the ROM 103 using the RAM 102 as a work memory, and comprehensively controls each component of the processing apparatus 100 via the system bus 107. Thereby, various processes described later are executed. The secondary storage device 104 is a device for storing various data handled by the processing device 100, and an HDD is used in this embodiment. The CPU 101 writes data to the secondary storage device 104 and reads data stored in the secondary storage device 104 via the system bus 107. As the secondary storage device 104, various storage devices such as an optical disk drive and a flash memory can be used in addition to the HDD.

  The input IF 105 includes, for example, a serial bus IF such as USB or IEEE1394, and input of data, commands, and the like from an external device to the processing device 100 is performed via the input IF 105. Specifically, the processing apparatus 100 acquires data from the external storage device 108 via the input IF 105. As the external storage device 108, for example, a hard disk, a memory card, a CF card, an SD card, a USB memory, or the like can be used. Further, the processing device 100 acquires a command input by the user using the operation unit 110 via the input IF 105. The operation unit 110 is a device for inputting user instructions to the processing device 100, and includes, for example, a mouse and a keyboard.

  The output IF 106 includes, for example, a video output terminal such as DVI or HDMI (registered trademark) in addition to a serial bus interface such as USB and IEEE1394 similar to the input IF 105. Output of data and the like from the processing apparatus 100 to the external apparatus is performed via the output IF 106. The processing device 100 displays the image by outputting the processed image or the like to the display device 109 (liquid crystal display or the like) via the output IF 106. In addition, although the component of the processing apparatus 100 exists besides the thing mentioned above, since it is not the main point of this invention, description is abbreviate | omitted.

<Outline of processing executed by information processing apparatus>
Hereinafter, an outline of a process (hereinafter, this process) for creating a parallax map based on a multi-viewpoint image, which is executed by the processing apparatus 100 in the present embodiment, will be described.

  First, the CPU 101 reads a filter (data) from the external storage device 108. This filter is acquired prior to this processing, and may be learned in advance using a training image different from the multi-viewpoint image to be processed. The training image to be used is preferably an image obtained by capturing a subject that is the same or similar to the subject of the multi-viewpoint image to be processed or a scene that is the same or similar to the scene of the multi-viewpoint image. The following method exists as a first specific example of the filter learning method. That is, a large number of partial images are extracted from the training image, a covariance matrix is generated for each of the extracted partial images, and an average (average covariance matrix) of these covariance matrices is calculated. Then, this average covariance matrix is a method for obtaining a filter by performing principal component analysis or eigenvalue analysis by singular value decomposition. As a second specific example, there is a method of learning a filter from a training image using a learning algorithm of a known convolutional neural network. However, the filter learning method is not limited to these specific examples. Further, as a filter, an image to be processed or a function given analytically (for example, a discrete cosine transform base) may be used in addition to what is learned using a training image.

  Next, as shown in Expression (1), a plurality of first filters are convolved (applied) to each single-viewpoint image included in the multi-viewpoint image to obtain a first feature map. In the present specification, a filter that is primarily folded into a single viewpoint image is referred to as a first filter, and there are a plurality of first filters.

In Expression (1), F _1i (x, y) represents a coefficient at the coordinates (x, y) of the i-th first filter. I _k (x, y) represents a pixel value at the coordinates (x, y) of the k-th single-viewpoint image. T _ik (x, y) represents a pixel value at the coordinates (x, y) of the first feature map obtained by convolving the i-th first filter with the k-th single-viewpoint image. The pixel value of the single viewpoint image is a value such as luminance, color difference (that is, a component other than luminance in a color space such as YUV or Lab), a color channel (for example, RGB), and the like.

  By convolving the filter with the input image, a specific structure in the single viewpoint image can be detected. For example, when a Sobel filter having a gradient in the horizontal direction and a distribution of uniform values in the vertical direction is convoluted with the input image, the output image (feature map) is output at the position of the vertical edge included in the input image. The pixel value takes a large value. Thus, the feature map represents the spatial distribution of the structure corresponding to the convolved filter.

  Next, as shown in Expression (2), the second feature value map is obtained by convolving the second filter with the first feature value map. In this specification, a filter that is secondarily convolved with a single-viewpoint image (that is, after the first filter) is referred to as a second filter, and there are a plurality of second filters.

In Expression (2), F _2j (x, y) represents a coefficient at the coordinates (x, y) of the j-th second filter. O _ijk (x, y) represents a pixel value at the coordinates (x, y) of the second feature map.

  Note that non-linear transformation may be performed on the obtained feature map. Nonlinear transformation is performed for the purpose of improving the search accuracy of corresponding points by enhancing edges included in an image and amplifying differences between pixels. Nonlinear conversion includes statistical filter processing, threshold value processing, and conversion of values for each pixel by a nonlinear function. Hereinafter, a case where non-linear transformation is not performed unless otherwise specified will be described.

  In the above example, the second feature map is finally acquired by convolving the first filter and the second filter with each single-viewpoint image. However, the feature map to be finally acquired is It is not limited to the second feature amount map. In other words, the n + 1-th feature quantity is obtained by performing the same process as described above to obtain the (n + 1) -th feature map by convolving the (n + 1) -th filter with the n-th (n is a natural number) feature map. It is good also as a feature-value map which acquires a map finally. Although the filter may be folded once, increasing the number of times makes it possible to extract a more complicated structure of the image.

  As general feature quantities used in this technical field, various local feature quantities such as SIFT (Scale-Invariant Feature Transform) are known. Since the position where these local feature amounts are calculated is limited to a position satisfying a specific condition in the image, when searching for corresponding points using the local feature amounts, the density of the corresponding points is the resolution of the image (pixel Very low compared to (density). On the other hand, in the present invention, since the feature amount can be calculated for all pixels in principle, the density of corresponding points can be made the same as the resolution of the image. In addition, since the present invention can calculate the feature amount only by the filter processing, the calculation cost of the feature amount per point can be extremely reduced.

In this embodiment, after acquiring the second feature value map, the processing device 100 searches for corresponding points between single viewpoint images based on the acquired second feature value map. In the following description, a case where a parallax map is created based on two single-view images acquired by a stereo camera, that is, the first single-view image I ₁ and the second single-view image I ₂ will be described as an example. Here, the parallax map in the present embodiment is image data in a bitmap format having a parallax value corresponding to each pixel position as a pixel value. A stereo camera is a camera that can acquire information on its depth direction by simultaneously capturing images of a subject from a plurality of different directions. For two single-viewpoint images acquired by a stereo camera, the horizontal direction of each image is the same. is there.

To estimate the parallax in the single viewpoint image I ₁ of the coordinates (x, y), first single-view image I second feature map O _ij1 of coordinates for ₁ (x, y) values in i and j of the order To obtain a feature vector V ₁ (x, y). For example, the feature vector V ₁ (x, y) is expressed as V ₁ (x, y) = (O ₁₁₁ (x, y), O ₁₂₁ (x, y), O ₁₃₁ (x, y),. ). When there are M first filters and N second filters, the dimension of the feature vector V ₁ (x, y) is M × N.

Next, the feature vector V ₂ (x ′, y ′) is obtained by arranging the values at the coordinates (x ′, y ′) of the second feature map O _ij2 for the single viewpoint image I _{2 in} the order of i and j. . Here, the order in which the values in the coordinates (x ′, y ′) of the second feature map O _ij2 are arranged is determined when the feature vector V ₁ (x, y) is acquired. _This is the same as the order of i and j in which the values at the coordinates (x, y) of _ij1 are arranged. The feature vector V ₂ (x ′, y ′) is repeatedly acquired by changing the coordinates (x ′, y ′). However, in this embodiment, since the multi-viewpoint image is acquired by the stereo camera, the movement range of the coordinates (x ′, y ′) at this time passes the coordinates (x, y) on the single-viewpoint image I _2. Can be limited to the horizon.

Next, the similarity between the feature vector V ₁ (x, y) and the feature vector V ₂ (x ′, y ′) is quantified and derived. Examples of the similarity include various commonly used distances (Euclidean distance, Manhattan distance, Hamming distance, etc.), a cross-correlation coefficient, and the like.

Next, the coordinate (x ′, y ′) that maximizes the similarity is derived, and the distance between the derived coordinate (x ′, y ′) and the coordinate (x, y) is output as an estimated parallax value. To do. A parallax map is obtained by executing the above-described processing on all coordinates of the single viewpoint image I ₁ .

In the above example, the case where a multi-viewpoint image is acquired by a stereo camera has been described. However, the means for acquiring a multi-viewpoint image is not limited to a stereo camera. When the multi-viewpoint image acquisition means is not a stereo camera, the coordinates (x ′, y ′) are used when acquiring the feature vector V ₂ (x ′, y ′) while moving the coordinates (x ′, y ′). The same processing as described above is performed by expanding the movement range. Alternatively, when information about the position and orientation of the camera that captured the subject is obtained when each single-viewpoint image is obtained, the coordinates (x) are obtained when the feature vector V ₂ (x ′, y ′) is obtained. The movement range of ', y') can be limited to epipolar lines that are uniquely determined from this information.

<About processing executed by information processing apparatus>
Hereinafter, specific processing executed by the processing apparatus 100 according to the present embodiment will be described with reference to FIGS. 2 and 3. FIG. 2 is a block diagram illustrating a functional configuration of the processing apparatus 100 according to the present embodiment. As illustrated, the processing apparatus 100 includes an acquisition unit 201, a feature amount generation unit 202, a corresponding point search unit 203, and an output unit 204. These components are realized by the CPU 101 of the processing apparatus 100 expanding a control program stored in the ROM 103 in the RAM 102 and executing the expanded program. Alternatively, the processing apparatus 100 may be configured to include a dedicated processing circuit corresponding to each component.

  The acquisition unit 201 acquires a multi-viewpoint image and outputs the acquired multi-viewpoint image to the feature amount generation unit 202. In the present embodiment, the acquisition unit 201 includes a first single viewpoint image that is an image when the subject is viewed from the first viewpoint, and a second viewpoint that is different from the first viewpoint. It is assumed that a second single-viewpoint image that is an image is acquired. The first single viewpoint image is acquired by imaging the subject from the first viewpoint, and the second single viewpoint image is acquired by imaging the subject from the second viewpoint. The first single-viewpoint image and the second single-viewpoint image may be data input from an external device or data stored in the secondary storage device 104.

  The feature value generation unit 202 creates a plurality of feature value maps corresponding to the first single-viewpoint image based on the first single-viewpoint image by using a filter acquired in advance, and outputs it to the corresponding point search unit 203. . In addition, the feature quantity generation unit 202 creates a plurality of feature quantity maps corresponding to the second single viewpoint image based on the second single viewpoint image by using the pre-acquired filter, and the corresponding point search section 203. Output to.

  The corresponding point search unit 203 determines whether the first single viewpoint image and the second single viewpoint image are based on a plurality of feature amount maps corresponding to the first single viewpoint image and a plurality of feature amount maps corresponding to the second single viewpoint image. Search for corresponding points between.

  The output unit 204 outputs a parallax map corresponding to the first single-viewpoint image and the second single-viewpoint image based on the search result by the corresponding point search unit 203.

  FIG. 3 is a flowchart of processing executed by the processing device 100 in this embodiment. In step S <b> 301, the acquisition unit 201 acquires a multi-viewpoint image to be processed via the input interface 105 or from the secondary storage device 104. Then, the acquisition unit 201 outputs the acquired multi-viewpoint image to the feature amount generation unit 202. In this embodiment, a case where the multi-viewpoint image acquired by the acquisition unit 201 is two single-viewpoint images is described as an example. However, the number of single-viewpoint images included in the multi-viewpoint image is not limited to two and may be three or more. When there are three or more single-viewpoint images included in the multi-viewpoint image, one or a plurality of sets of two single-viewpoint images are created, and a subsequent process is performed on each set to create a parallax map . Note that the form of the parallax map is not limited to image data in the bitmap format, and the parallax map may be output in a table format that defines the relationship between the pixel position and the parallax value. In addition, when a some parallax map is produced with respect to one single viewpoint image, they are synthesize | combined and finally one parallax map is output. As a method of combining a plurality of parallax maps, a method of averaging pixel values of each parallax map for each coordinate or a method of weighting and adding pixel values of each parallax map for each coordinate may be used.

  In step S <b> 302, the feature amount generation unit 202 creates a plurality of feature amount maps for each single-viewpoint image by sequentially convolving each of the single-viewpoint images input from the acquisition unit 201 with a plurality of filters. . Hereinafter, such processing is referred to as sequential or stepwise filter processing. The filter used in this step is read from the external storage device 108. FIG. 4 is a functional block diagram of the feature quantity generation unit 202 when the first filter and the second filter are sequentially convoluted with the single viewpoint image. As shown in FIG. 4, the feature value generation unit 202 includes a filter processing unit 211 that convolves the first filter with each single-viewpoint image, and a filter processing unit 213 that convolves the second filter with the output of the filter processing unit 211. Have. FIG. 5 shows a flowchart of processing executed by the feature quantity generation unit 202 shown in FIG. As illustrated in FIG. 5, in step S <b> 311, the filter processing unit 211 convolves the first filter with the single viewpoint image (that is, executes the first filter processing). Next, in step S313, the filter processing unit 213 convolves the second filter with the output of the filter processing unit 211 (ie, executes the second filter processing). In addition, although the case where the 1st filter and the 2nd filter are convolved is demonstrated here, the number of the filters which convolve sequentially is not limited to 2, and may be 3 or more. For example, when the third filter is further convoluted, the third filter process is additionally executed after the second filter process.

  Note that the nonlinear conversion process described above may be executed after the filter process. FIG. 6 shows a functional block diagram of the feature quantity generation unit 202 when the nonlinear transformation process is executed in addition to the two-stage filter process. FIG. 7 shows a flowchart of processing executed by the feature quantity generation unit 202 shown in FIG. Specifically, as the nonlinear conversion process in step S312 or S314 in FIG. 7, a known conversion process such as tanh, sigmoid, or ReLU used in the neural network may be used.

  Returning to the description of FIG. In step S303, the feature quantity generation unit 202 has completed creation of a plurality of feature quantity maps for each single viewpoint image, that is, feature quantities corresponding to all combinations of the first filter and the second filter. It is determined whether a map has been created for each single-viewpoint image. If the determination result in step S303 is true, the process proceeds to step S304. If the determination result is false, the process returns to step S302.

  In step S304, the corresponding point search unit 203 searches for a corresponding point between the first single viewpoint image and the second single viewpoint image based on the feature amount map created in step S302. Here, the corresponding point search unit 203 may adaptively change the corresponding point search range of the target position (target pixel position) based on the corresponding point search result of the neighboring position. For example, when a parallax map is acquired in advance by rough sampling (low resolution), and then a parallax value is calculated at a position between the sampling positions (corresponding points are searched), the parallax value already calculated at a neighboring position is A candidate value is determined, and a parallax value is calculated within the range of the candidate value. As another example, when the parallax value is sequentially calculated by scanning the sampling position, the parallax value at the sampling position with the highest similarity is selected from the parallax values between the new sampling position and the neighboring sampling positions. There is a method of adopting the parallax value of the target position. Yet another example is a method of calculating a disparity value that minimizes a cost function based on a Markov random field.

  In step S305, the output unit 204 converts the corresponding point search result into a format such as a parallax map and outputs the result.

  The above is the processing for obtaining corresponding points between single-viewpoint images in the present embodiment. According to the present embodiment, in the parallax estimation between single-viewpoint images, the complicated structure of each single-viewpoint image can be extracted effectively, so that the accuracy of the parallax estimation is improved and the parallax estimation result is stabilized.

<About the effects of this embodiment>
In order to explain the effects of the present embodiment, an example in which the above processing is actually performed will be shown below. In this example, two images having parallax of 5 pixels only in the horizontal direction and parallel optical axes are used as input images. FIG. 8A and FIG. 8B show input images used in this example. The two input images shown in FIG. 8 are a pair of artificially created parallax images, and these images uniformly give parallax to the same original image, and further have different blur and luminance modulation. It is obtained by giving.

  FIG. 9A shows the extraction of 80,000 5 × 5 partial images from a large number of natural images prepared as training images, calculation of an average covariance matrix, and principal component analysis for the calculated average covariance matrix. It is a figure which shows eight 1st filters obtained by (1). Further, FIG. 9B shows that 640,000 partial images are obtained by convolving each of the first filters (eight) with the extracted 80,000 partial images, and an average covariance matrix is calculated. It is a figure which shows eight 2nd filters obtained by the principal component analysis with respect to the calculated average covariance matrix. As shown in the figure, each filter has a size of 5 × 5.

  In this example, as a parallax estimation error, a mean square of a difference from a true value in an area excluding an image end portion (an area having a width of 5 pixels in the upper, lower, left, and right) where a convolution error occurs is evaluated. Also, the Euclidean distance (sum of squared differences) is used for the similarity of the feature vector, and the block size is 5 × 5.

  Each diagram of FIG. 10 is a disparity map of the corresponding point search result. In the parallax map shown in FIG. 10, the estimated parallax value at each pixel position is represented by gradation expression. FIG. 10A is a disparity map derived by conventional block matching based on the sum of squared differences of pixel values. The processing time required to derive this parallax map is 0.4 seconds, and the error of the parallax estimation value is 8.23 pixels. FIG. 10B is a disparity map derived when only the first filter is used. The processing time required to derive this parallax map is 0.2 seconds, and the error of the parallax estimation value is 1.56 pixels. FIG. 10C is a disparity map derived when the first filter and the second filter are used. The processing time required for deriving this parallax map is 1.8 seconds, and the error of the parallax estimation value is 0.10 pixels. Thus, the accuracy of parallax estimation is improved by increasing the number of convolutions according to the method of the present embodiment.

  Depending on the input image, the accuracy of the parallax estimation is improved by increasing the size of the filter and the block. In the above example, the size of both the filter and the block is 5 × 5, but the above processing may be performed using, for example, a 15 × 15 size filter and block. The processing time required to derive the disparity map in this case is 3.2 seconds in the case of block matching, 0.2 seconds when only the first filter is used, and the first filter and the second filter are When used, it is 1.9 seconds. Thus, according to the present embodiment, not only the accuracy of the parallax estimation can be improved but also the processing speed can be increased. The reason for this is as follows. That is, in the case of block matching, the corresponding point search is performed by comparing vectors of the number of pixels included in the block (in the above example, 225 (= 15 × 15) dimensions). On the other hand, in this embodiment, the corresponding point search is performed by comparing vectors of the number of filters (8 or 64 in the above example), and the number of dimensions of the comparison target vector can be small. As described above, in this embodiment, the calculation cost mainly depends on the number of filters, not the filter size. Therefore, even if the filter size is changed depending on the image, the processing time is substantially constant.

  Furthermore, the present embodiment is also superior to the prior art in terms of robustness, and it is possible to accurately obtain corresponding points between input images even when the brightness of the images is changed.

[Example 2]
In the second embodiment, a case where a filter is created based on a multi-viewpoint image to be processed will be described with reference to FIGS. 11 and 12. In addition, description is abbreviate | omitted about the content same as Example 1. FIG.

  FIG. 11 is a block diagram illustrating a functional configuration of the processing apparatus 100 according to the present embodiment. As illustrated, the processing apparatus 100 includes an acquisition unit 201, a feature amount generation unit 202, a corresponding point search unit 203, an output unit 204, and a filter creation unit 205. The filter creation unit 205 creates a filter based on the multi-viewpoint image.

  FIG. 12 is a flowchart of processing executed by the processing device 100 in the present embodiment. In step S <b> 1201, the acquisition unit 201 acquires a multi-viewpoint image to be processed via the input interface 105 or from the secondary storage device 104. Then, the acquisition unit 201 outputs the acquired multi-viewpoint image to the filter creation unit 205.

  In step S <b> 1202, the filter creation unit 205 creates a plurality of filters based on the multi-viewpoint image input from the acquisition unit 201. The method for creating the filter is the same as the method described in the first embodiment. Note that a filter may be created using an image other than the input multi-viewpoint image. In that case, the image used for creating the filter, the calculated mean covariance matrix, the created filter, and the like are stored in an external storage device. Read from 108.

  In step S <b> 1203, the feature value generation unit 202 performs sequential (stepwise) filter processing on each single-viewpoint image acquired by the acquisition unit 201 using the filter generated by the filter generation unit 205. By this processing, a feature amount map for each single viewpoint image is created.

  In step S1204, the feature quantity generation unit 202 completes the creation of a plurality of feature quantity maps for each single viewpoint image, that is, the feature quantity map corresponding to all combinations of filters that are sequentially convoluted is displayed as a single viewpoint. It is determined whether each image has been created. If the determination result in step S1204 is true, the process proceeds to step S1205. If the determination result is false, the process returns to step S1203.

  In step S1205, the corresponding point search unit 203 searches for a corresponding point between single viewpoint images based on the feature amount map created in step S1203.

  In step S1206, the output unit 204 converts the corresponding point search result into a format such as a parallax map and outputs the result.

<Other embodiments>
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program This process can be realized. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

100 Information Processing Device 201 Acquisition Unit 202 Feature Quantity Generation Unit 203 Corresponding Point Search Unit

Claims (9)

An acquisition means for acquiring a first single-viewpoint image and a second single-viewpoint image obtained by imaging the same subject from different viewpoints;
For each of the first single-view image and the second single-view image, a process of applying a filter for detecting a specific structure in the image is performed step by step by changing the filter. Creating means for creating a plurality of feature amount maps for each of the first single-viewpoint image and the second single-viewpoint image;
Search means for searching for corresponding points between the first single-viewpoint image and the second single-viewpoint image based on the plurality of created feature amount maps;
The information processing apparatus characterized in that the filter applied by the creating means at each stage is a plurality of different filters.   The information processing apparatus according to claim 1, further comprising an output unit that outputs a parallax map based on a search result obtained by the search unit. The search means includes
Deriving a first feature vector at the target pixel position of the first single-view image based on a plurality of feature maps for the first single-view image,
Based on a plurality of feature amount maps for the second single viewpoint image, a second feature amount vector is derived for each pixel position in the search range of the second single viewpoint image;
Deriving the similarity between the first feature quantity vector and the second feature quantity vector for each pixel position in the search range;
3. The pixel position of the second single viewpoint image corresponding to the target pixel position is a pixel position having the highest similarity among the derived similarities. Information processing device.   The information processing apparatus according to claim 3, wherein the search unit changes the search range based on a corresponding point search result in the vicinity of the target pixel position. The first single-view image and the second single-view image are image data of the same size,
The feature quantity map is bitmap format data having a feature quantity as a pixel value, and has the same size as the first single-viewpoint image. The information processing apparatus described.   6. The apparatus according to claim 1, further comprising execution means for executing nonlinear transformation each time a filter is applied when the creation means performs the process of applying the filter stepwise. Information processing device.   The information processing according to any one of claims 1 to 6, further comprising a creation unit that creates the plurality of filters based on the first single-viewpoint image and the second single-viewpoint image. apparatus. Acquiring a first single-viewpoint image and a second single-viewpoint image obtained by imaging the same subject from different viewpoints;
For each of the first single-view image and the second single-view image, a process of applying a filter for detecting a specific structure in the image is performed step by step by changing the filter. Creating a plurality of feature amount maps for each of the first single-viewpoint image and the second single-viewpoint image;
Searching for corresponding points between the first single-viewpoint image and the second single-viewpoint image based on the plurality of created feature amount maps;
An information processing method characterized in that the filters applied at each stage in the creating step are a plurality of different filters.   A program for causing a computer to execute the method according to claim 8.