WO2016027524A1 - Information processing apparatus, information processing method, program, and electronic optical device - Google Patents

Abstract

[Problem] To quantitatively and quickly measure a change of optical status that may occur in an optical system formed in an optical device. [Solution] An information processing apparatus of the present disclosure comprises: an image capture unit that captures a plurality of random pattern images, which are outputted from an optical device having a given optical system, in respective situations where the optical system exhibits respective different states, thereby generating a plurality of pieces of image capture data corresponding to the respective states of the optical system; a corresponding position determination unit that calculates, by use of the plurality of pieces of image capture data generated by the image capture unit, an optical flow among the plurality of captured images with respect to at least three arbitrary points in the captured images corresponding to the plurality of pieces of image capture data, thereby determining positions corresponding to each other among the plurality of captured images; and a feature quantity calculation unit that calculates, on the basis of a determination result obtained by the corresponding position determination unit, a feature quantity representative of the difference of the optical system among the plurality of captured images of interest.

Description

Information processing apparatus, information processing method, program, and electro-optical device

The present disclosure relates to an information processing apparatus, an information processing method, a program, and an electro-optical device.

In recent years, with the development of information processing technology, stereoscopic display devices such as so-called head-mounted displays that allow a user to visually recognize a stereoscopic image by separately outputting a left-eye image and a right-eye image to the user have become widespread. Has come to do. Since the stereoscopic display device is a device that perceives the image stereoscopically by the user recognizing the left-eye image and the right-eye image, the stereoscopic image provided by the stereoscopic display device is It is important to evaluate whether it is appropriate from the viewpoint of comfort.

Therefore, Patent Document 1 below discloses a technique for evaluating a stereoscopic video obtained by fusing a left-eye image and a right-eye image using image processing for obtaining an optical flow.

JP 2005-142819 A

Here, in the technique disclosed in Patent Document 1, the stereoscopic image is evaluated by calculating the optical flow. However, it is preferable to calculate the optical flow by paying attention to characteristic points existing in the image to be evaluated, and depending on the image to be evaluated, there is a possibility that the optical flow having sufficient accuracy cannot be calculated. There is.

In addition, in the stereoscopic display device, due to the change of the optical system provided in the device, the image projected on the left and right eyes of the user is theoretically shifted in the vertical and horizontal directions, the size is different, There is a possibility of tilting from side to side. However, although the technique disclosed in Patent Document 1 can evaluate the image itself, it cannot quantitatively and rapidly measure a change that can occur in the optical system provided in the stereoscopic display device. . Therefore, when evaluating an optical system provided in a stereoscopic display device, an inspector has to make subjective judgments based on his sense and experience in the production line of the stereoscopic display device.

In addition, the situation as described above is also necessary in the case where it is desired to compare the wide-angle side image and the telephoto side image in an optical apparatus having a zoom optical system, or in the case where it is desired to compare the standard optical system and the optical system of the inspection product Similarly, there has been a demand for a technique for quantitatively and rapidly measuring a state change that can occur in an optical system.

Therefore, in the present disclosure, in view of the above circumstances, an information processing apparatus, an information processing method, and a program capable of quantitatively and rapidly measuring a change in an optical state that may occur in an optical system provided in an optical apparatus. suggest.

Also, the present disclosure proposes an electro-optical device in which an optical system provided in the optical device is adjusted based on a feature amount that quantitatively represents a change in an optical state that can occur in the optical system.

According to the present disclosure, a plurality of random pattern images output from an optical device having a predetermined optical system are captured in a situation where the state of the optical system is different, and a plurality of imaging data corresponding to the state of the optical system, respectively. And an optical flow between the plurality of captured images at an arbitrary point of at least three or more points in the captured image corresponding to the captured image data. By calculating, a corresponding position specifying unit that specifies positions corresponding to each other between the plurality of captured images, and between the plurality of captured images being focused on based on a specifying result obtained by the corresponding position specifying unit An information processing apparatus is provided that includes a feature amount calculation unit that calculates a feature amount that represents a difference in the optical system.

In addition, according to the present disclosure, a plurality of random pattern images output from an optical apparatus having a predetermined optical system are captured in a situation where the state of the optical system is different, and a plurality of images corresponding to the state of the optical system are respectively obtained. Generating imaging data, and using the plurality of captured image data, calculating an optical flow between the plurality of captured images at an arbitrary point of at least three points in the captured image corresponding to the captured data. Thus, specifying the positions corresponding to each other among the plurality of captured images and representing the difference in the optical system between the plurality of captured images of interest based on the identification result of the positions corresponding to each other Calculating an amount, and an information processing method is provided.

In addition, according to the present disclosure, a plurality of random pattern images output from an optical apparatus having a predetermined optical system are captured in a situation where the state of the optical system is different, and a plurality of images corresponding to the state of the optical system are respectively obtained. A computer having an imaging unit that generates imaging data uses a plurality of imaging data captured by the imaging unit, and a plurality of captured images at arbitrary points of at least three points in a captured image corresponding to the imaging data. Based on the corresponding position specifying function for specifying positions corresponding to each other among the plurality of captured images and the specifying result obtained by the corresponding position specifying function by calculating the optical flow between There is provided a program for realizing a feature amount calculation function for calculating a feature amount representing a difference in the optical system between a plurality of captured images.

Further, according to the present disclosure, an optical system that displays a predetermined image on a predetermined display screen and a random pattern image output from an optical device having the predetermined optical system are in a state where the state of the optical system is different. An imaging unit that captures a plurality of images and generates a plurality of imaging data respectively corresponding to the state of the optical system, and a captured image corresponding to the imaging data using the plurality of imaging data captured by the imaging unit A corresponding position specifying unit that specifies positions corresponding to each other among the plurality of captured images by calculating an optical flow between the plurality of captured images at an arbitrary point of at least three points, and the corresponding position specifying unit A feature amount calculation unit that calculates a feature amount representing a difference in the optical system between the plurality of captured images of interest based on the obtained specific result. It acquires feature amount information based on the feature amount calculated by the electronic optical apparatus and an adjustment unit for adjusting the optical system based on the feature amount information is provided.

According to the present disclosure, optical data between a plurality of captured images is used at any point of at least three or more points in a captured image corresponding to the captured image data using a plurality of captured image data captured in different situations of the optical system. By calculating the flow, positions corresponding to each other between the plurality of captured images are specified. Further, based on the obtained specific result, a feature amount representing a difference in the optical system between the plurality of captured images of interest is calculated.

As described above, according to the present disclosure, it is possible to quantitatively and rapidly measure a change in an optical state that may occur in an optical system provided in an optical apparatus.

Note that the above effects are not necessarily limited, and any of the effects shown in the present specification or other things that can be grasped from the present specification together with the above effects or instead of the above effects. The effect of may be produced.

3 is a block diagram illustrating an example of a configuration of an information processing device according to a first embodiment of the present disclosure. FIG. It is the block diagram which showed an example of the structure of the arithmetic processing part which the information processing apparatus which concerns on the embodiment has. It is explanatory drawing which showed an example of the production | generation process of the random pattern image which concerns on the embodiment. It is explanatory drawing which showed an example of the production | generation process of the random pattern image which concerns on the embodiment. It is explanatory drawing which showed an example of the production | generation process of the random pattern image which concerns on the embodiment. It is explanatory drawing which showed an example of the calculation process of the optical flow which concerns on the embodiment. It is explanatory drawing which showed an example of the calculation process of the optical flow which concerns on the embodiment. It is explanatory drawing which showed an example of the feature-value calculation process which concerns on the embodiment. It is explanatory drawing which showed an example of the feature-value calculation process which concerns on the embodiment. It is explanatory drawing which showed an example of the feature-value calculation process which concerns on the embodiment. It is explanatory drawing which showed an example of the state determination process of the optical system which concerns on the same embodiment. It is the flowchart which showed an example of the flow of the information processing method which concerns on the embodiment. It is the block diagram which showed an example of the structure of the electro-optical apparatus which concerns on the same embodiment. It is the block diagram which showed an example of the structure of the electro-optical apparatus which concerns on the same embodiment. FIG. 3 is a block diagram illustrating an example of a hardware configuration of an information processing apparatus and an electro-optical device according to an embodiment of the present disclosure.

Hereinafter, preferred embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

The description will be made in the following order.
1. 1. First embodiment 1.1. Configuration of information processing apparatus 1.2. Flow of information processing method 1.3. Electro-optical equipment About hardware configuration

(First embodiment)
<Configuration of information processing device>
First, the configuration of the information processing apparatus according to the first embodiment of the present disclosure will be described in detail with reference to FIGS. 1 to 10.
FIG. 1 is a block diagram illustrating an example of the configuration of the information processing apparatus according to the present embodiment. FIG. 2 is a block diagram illustrating an example of a configuration of an arithmetic processing unit included in the information processing apparatus according to the present embodiment. 3 to 4B are explanatory diagrams showing an example of the random pattern image generation process according to the present embodiment. 5 to 6 are explanatory diagrams showing an example of the optical flow calculation process according to the present embodiment. 7 to 9 are explanatory diagrams showing an example of the feature amount calculation processing according to the present embodiment. FIG. 10 is an explanatory diagram showing an example of the state determination process of the optical system according to the present embodiment.

[Overall configuration]
First, the overall configuration of the information processing apparatus according to the present embodiment will be described with reference to FIG.
The information processing apparatus 10 according to the present embodiment captures a random pattern image output from the optical system 3 provided in the optical apparatus 1 and calculates a feature amount that characterizes the optical state of the optical system 3. And functions as a measuring device for measuring the optical state of the optical system 3.

Here, the optical apparatus 1 to be measured by the information processing apparatus 10 according to the present embodiment is not particularly limited, and a known optical system 3 capable of realizing a plurality of different optical states is provided. What is necessary is just to have. In the optical apparatus 1, various images are output to the outside through the optical system 3.

Examples of the optical device 1 include an optical device having a zoom optical system, an optical device having a stereo optical system, or an optical device that generates a stereo image. Examples of the optical apparatus having the zoom optical system include a camera, a camcorder, and a telescope. Examples of the optical apparatus having a stereo optical system include a microscope and binoculars. Optical devices that generate stereo images include stereoscopic display devices such as head-mounted displays.

The information processing apparatus 10 according to the present embodiment mainly includes an imaging unit 101, an arithmetic processing unit 103, and a storage unit 105, as illustrated in FIG.

The imaging unit 101 includes various lenses and various imaging elements such as a charge coupled device (CCD) and a complementary metal oxide semiconductor (CMOS). Here, the lens and the image sensor constituting the imaging unit 101 are not particularly limited, and any known one can be used as long as it can appropriately capture an image output from the optical device 1. it can.

In addition to the lens and the image sensor, the imaging unit 101 uses a known drive mechanism such as various motors and actuators in order to change the installation position of the lens and the image sensor relative to the optical device 1. You may have.

The imaging unit 101 according to the present embodiment captures a plurality of random pattern images output from the optical device 1 having the predetermined optical system 3 under different conditions of the optical system 3, and changes the state to the state of the optical system 3. A plurality of corresponding imaging data is generated.

Here, the random pattern image is a pattern image in which the randomness of the pattern arrangement is mathematically guaranteed, and there is no portion having the same pattern in a certain pattern image. Therefore, in such a random pattern image, the value of the autocorrelation coefficient at the position of interest is 1, and the value of the cross-correlation coefficient between the position of interest and other positions is 0. Such a random pattern image can be created by overlaying and synthesizing images with fine textures, but can also be created by using an M-sequence signal composed of random numbers consisting of 0 and 1. It is.

The random pattern image captured by the imaging unit 101 may be a random pattern image created as described above, but it is more preferable to use a synthetic random pattern image generated by the arithmetic processing unit 103 described later.

Using the random pattern image as described above as an image to be output from the optical device 1 and performing an arithmetic process on the captured image obtained by capturing the random pattern image, all the pixels constituting the captured image are subjected to an optical flow described later. It can be used as a feature point when calculating. Therefore, when calculating the optical flow for such a captured image, the optical flow can be calculated at an arbitrary point constituting the image.

In addition, the “situation in which the state of the optical system 3 is different” refers to, for example, one optical system that outputs a left-eye image and one optical system that outputs a right-eye image in a head-mounted display (HMD). The optical device 1 may be realized by the presence of a plurality of optical systems 3 having different states. Further, “the situation in which the state of the optical system 3 is different” refers to an optical system in which one type of optical system 3 exists in the optical device 1 like an optical device having a zoom optical system and a wide-angle image is output. It may be realized by changing the state of various optical elements in the optical system 3 like an optical system when a telephoto image is output. In addition, “the situation where the state of the optical system 3 is different” means that the optical system 3 included in the standard optical device 1 and the optical system 3 included in the optical device 1 to be inspected have a plurality of It may be realized by a plurality of optical systems 3 that may be in different states between the optical devices 1.

A plurality of pieces of imaging data generated by the imaging unit 101 and corresponding to the state of the optical system 3 are output to the arithmetic processing unit 103.

The arithmetic processing unit 103 is implemented by, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory), a RAM (Random Access Memory), an input device, an output device, a communication device, and the like. The arithmetic processing unit 103 controls the imaging process performed by the imaging unit 101 and performs various arithmetic processes on a plurality of imaging data generated by the imaging unit 101, thereby providing an optical system provided in the optical apparatus 1. The feature quantity characterizing the state 3 is calculated. The arithmetic processing unit 103 can also determine the state of the optical system 3 provided in the optical device 1 based on the calculated feature amount. Further, the arithmetic processing unit 103 can generate a random pattern image used in the optical device 1 and output it to the optical device 1.

The detailed configuration of the arithmetic processing unit 103 will be described in detail later.

The storage unit 105 is realized by, for example, a RAM or a storage device included in the information processing apparatus 10 according to the present embodiment. The storage unit 105 performs various programs including various databases used for processing in the imaging unit 101 and the arithmetic processing unit 103, various programs including applications used for various arithmetic processes executed by these processing units, and some processing. Various parameters that need to be saved at the time, the progress of the process, and the like may be recorded as appropriate.

The storage unit 105 can be freely accessed by each processing unit such as the imaging unit 101 and the arithmetic processing unit 103 to write and read data.

The overall configuration of the information processing apparatus 10 according to the present embodiment has been described in detail above with reference to FIG.

[Configuration of arithmetic processing unit 103]
Next, the configuration of the arithmetic processing unit 103 included in the information processing apparatus 10 according to the present embodiment will be described in detail with reference to FIGS.

As illustrated in FIG. 2, the arithmetic processing unit 103 according to the present embodiment includes an imaging control unit 111, a random pattern image generation unit 113, a data output unit 115, a data acquisition unit 117, and a corresponding position specifying unit 119. And a feature amount calculation unit 121 and an optical system state determination unit 123.

The imaging control unit 111 is realized by, for example, a CPU, a ROM, a RAM, a communication device, and the like. The imaging control unit 111 comprehensively controls imaging processing in the imaging unit 101. In addition, the imaging control unit 111 cooperates with a data output unit 115 described later to output a composite random pattern image generated by a random pattern image generation unit 113 described later to the optical device 1 to be imaged. Is also possible. Accordingly, the imaging unit 101 included in the information processing apparatus 10 according to the present embodiment can capture a random pattern image output from the optical device 1 including the predetermined optical system 3 at an appropriate timing.

The random pattern image generation unit 113 is realized by a CPU, a ROM, a RAM, and the like, for example. The random pattern image generation unit 113 generates a random pattern image used in the optical device 1 based on the M-sequence signal. For example, as schematically illustrated in FIG. 3, the random pattern image generation unit 113 may generate a combined random pattern image by combining random pattern images based on at least two types of M-sequence signals.

An M-sequence signal (Maximum sequence code) is a random signal sequence of length n = 2 ^m −1 generated using a prime irreducible polynomial h (m) of degree m, and the numbers constituting the sequence are It consists only of 0 and 1. A random pattern image can be obtained by assigning such M-sequence signals to the pixels constituting the image. For example, a random pattern image generated by assigning an M-sequence signal for each pixel is hereinafter referred to as an “M1 image” for convenience. A random pattern image generated by assigning an M-sequence signal to each pixel region defined by p × p (p ≧ 2) pixels will be referred to as an “Mp image” for convenience.

The degree m of the irreducible polynomial h (m) used when generating the random pattern image is preferably determined according to the number of pixels of the image (image corresponding to the imaging data) generated by the imaging unit 101. Specifically, when an image (captured image) corresponding to the captured data is composed of P pixels, it is preferable to select an integer m that satisfies m ≧ log ₂ (P). By selecting such an integer m, it is possible to prevent a repeated pattern from being present in the image.

Here, the random pattern image used for generating the synthesized random pattern image may be selected according to the window size that is a processing unit in the optical flow calculation process performed by the corresponding position specifying unit 119 described later. preferable. That is, the random pattern image generation unit 113 includes a first random pattern image in which the size of the dots constituting the random pattern image is smaller than the window size, and a second random size in which the size of the dots is larger than the window size. It is preferable to select at least a pattern image.

By using the second random pattern image, the pattern constituting the random pattern image is not completely crushed even if the optical system 3 as the imaging target is blurred, and the corresponding position in the subsequent stage The specifying unit 119 can perform more accurate calculation processing. Further, by using the first random pattern image, the optical flow can be calculated even when the window size in the optical flow calculation is relatively small. Accordingly, it is possible to realize a highly accurate optical flow calculation process while suppressing an increase in the calculation amount.

Based on the knowledge as described above, the random pattern image generation unit 113 has an odd dot size such as M1, M3, M5, M7, M9,..., As shown in FIG. A synthetic random pattern image may be generated using only the random pattern image. Further, the random pattern image generation unit 113, for example, as shown in FIG. 4B, only random pattern images with an even dot size such as M2, M4, M6, M8, M10, and so on. Utilizing this, a synthetic random pattern image may be generated.

Here, when the random pattern image has a high frequency component exceeding the limit determined by the frequency characteristics of the imaging unit 101, aliasing noise is generated in the captured image, and the original signal is completely restored. Will not be able to. Therefore, it is preferable to remove a high frequency component by applying a low pass filter (LPF) to a random pattern image having such a high frequency component. By using the random pattern image obtained in this way (indicated as “M # F image” in FIGS. 4A and 4B), the influence of blur and moire interference of the imaging unit 101 can be suppressed. It becomes possible. Note that the low-pass filter to be used is not particularly limited, and a known filter can be used. For example, a Fourier transform such as a fast Fourier transform (FFT) or an inverse Fourier transform and a frequency filter may be combined. An averaging filter may be used.

Further, as a result of further studies by the present inventors, in order to calculate the optical flow more reliably over almost the entire captured image obtained by capturing the random pattern image, the window size at the time of optical flow calculation is set to It has become clear that it is more preferable to set the dot size to 2 to 3 times. Therefore, when generating a composite random pattern image, the random pattern image generation unit 113 replaces the processing shown in FIG. 3 with a plurality of random pattern images whose dot size is ½ or less of the window size. It is preferable to use it. In addition, it is more preferable that the random pattern image generation unit 113 uses a plurality of random pattern images whose dot size is 1/3 or less of the window size when generating the composite random pattern image.

The random pattern image generation unit 113 outputs the composite random pattern image generated in this way to the data output unit 115.

The data output unit 115 is realized by, for example, a CPU, a ROM, a RAM, a communication device, and the like. The data output unit 115 outputs the combined random pattern image generated by the random pattern image generation unit 113 to the optical device 1 that is the imaging target of the imaging unit 101. As a result, in the optical apparatus 1, the composite random pattern image is output through the optical system 3.

The data acquisition unit 117 is realized by, for example, a CPU, a ROM, a RAM, a communication device, and the like. The data acquisition unit 117 acquires a plurality of pieces of imaging data generated by the imaging unit 101 and outputs them to a corresponding position specifying unit 119 described later.

The corresponding position specifying unit 119 is realized by, for example, a CPU, a ROM, a RAM, and the like. The corresponding position specifying unit 119 uses a plurality of imaging data captured by the imaging unit 101, and at least three or more arbitrary points (preferably, three or more points not on a straight line) in the captured image corresponding to the imaging data. ), The optical flow between the plurality of captured images is calculated. Thereafter, the corresponding position specifying unit 119 specifies positions corresponding to each other among the plurality of captured images based on the calculated optical flow. Here, the point for calculating the optical flow may be any point in the captured image, and may be a lattice point in the captured image or may not be a lattice point.

In the captured image generated by the imaging unit 101, a plurality of points can be considered as schematically illustrated in FIG. Now, one of the plurality of captured images generated by the imaging unit 101 will be referred to as an image A, and the other one will be referred to as an image B. Further, and be expressed points of the image A and _(M x, M _y), a point image B _(P x, P _y) will be denoted as.

The corresponding position specifying unit 119 sets an appropriate window size in cooperation with the random pattern image generation unit 113 prior to the calculation of the optical flow. The specific size of the window size to be set is not particularly limited, and may be set as appropriate according to the required optical flow calculation accuracy.

The corresponding position specifying unit 119 sets the number of points for calculating the optical flow to at least 3 points. The feature amount calculation unit 121 described later performs a process of specifying a geometric conversion matrix that associates the image A and the image B of interest. The geometric conversion that can occur between the two images will be described later. There are two types: affine transformation and projective transformation. Here, as described in detail below, in the case of adopting a model in which the geometric transformation characterizing two images is an affine transformation, the number of unknowns in the affine transformation matrix of 3 rows × 3 columns is six. Therefore, when the affine transformation model is adopted in the feature amount calculation unit 121 described later, the corresponding position specifying unit 119 may be at least three points (the number of values applied to the process is six). Further, when adopting a model in which the geometric transformation characterizing two images is a projective transformation, the number of unknowns in the projection transformation matrix of 3 rows × 3 columns is eight. Therefore, when the projective transformation model is adopted in the feature amount calculation unit 121 described later, the corresponding position specifying unit 119 may be at least 4 points (the number of values applied to the process is 8).

As a prior verification, the present inventor calculated an optical flow for 4 to 8000 points using a random pattern image and an image obtained by rotating, translating, and enlarging / reducing the random pattern image. Tried to process. As a result, when a converted image with almost no distortion was used for verification, no significant difference was found in the accuracy of optical flow calculation. Accordingly, in the actual measurement of the optical system 3, if accurate affine transformation or projective transformation is considered between two images to be compared, the number of points for calculating the optical flow is three (in the case of affine transformation). Alternatively, even if there are four points (in the case of projective transformation), the optical flow calculation result is considered to have sufficient accuracy. In addition, there may be a case where optical distortion or blur that cannot be expressed only by affine transformation or projective transformation exists between two images. In this case, the number of points for calculating the optical flow is set to 1000 points or less (for example, about several tens to several hundred points), and the optical flow with sufficient accuracy can be obtained by arranging the points uniformly over the entire image. It was found that it was possible to calculate.

The corresponding position specifying unit 119 calculates the optical flow between the two images of interest in this manner, and thereby, at the point A (M _x , M _y ) in the image A as schematically shown in FIG. Corresponding points are identified from the points constituting the image B. Thereby, the corresponding position specifying unit 119 can specify that the point of the image B corresponding to the point A (M _x , M _y ) in the image A is the point B (P _x , P _y ). The corresponding position specifying unit 119 performs such a corresponding position specifying process on the point where the optical flow is calculated. In addition, when the corresponding position specifying unit 119 calculates the optical flow for a sufficiently large number of points, the number of points for which sufficient accuracy can be obtained in the feature amount calculation processing in the feature amount calculation unit 121 described later. However, the corresponding position specifying process may be performed, or the corresponding position specifying process may be performed for all points for which optical flows have been calculated.

Corresponding position specifying unit 119, when specifying the correspondence between the two points of interest based on the optical flow calculation result, outputs the obtained specifying result to feature amount calculating unit 121.

The feature amount calculation unit 121 is realized by, for example, a CPU, a ROM, a RAM, and the like. The feature amount calculation unit 121 calculates a feature amount representing a difference in the optical system between a plurality of focused images based on the specification result obtained by the corresponding position specification unit 119.

In more detail, the feature quantity calculation unit 121 uses the identification result of the point correspondence obtained by the corresponding position identification unit 119 and the first image (image A) corresponding to the first imaging data. Then, a transformation matrix representing the geometric correspondence between the second image (image B) corresponding to the second image data is calculated. In addition, the feature quantity calculation unit 121 uses the calculated transformation matrix to calculate a feature quantity (hereinafter referred to as an optical system shift, an image size change caused by the optical system, an optical system tilt, and an optical system distortion). , Also referred to as “optical system feature amount”).

In the feature quantity calculation unit 121 according to the present embodiment, as schematically shown in FIG. 7, as the geometric transformation representing the geometric correspondence between the image A and the image B, two of affine transformation and projective transformation are used. Consider the type. Hereinafter, the arithmetic processing performed by the feature amount calculation unit 121 for both the affine transformation model and the projective transformation model will be specifically described.

[Affine transformation model]
First, the feature amount calculation processing in the feature amount calculation unit 121 when the affine transformation model is adopted will be specifically described.
When the point A (M _x , M _y ) and the point B (P _x , P _y ) are related by affine transformation, the relationship can be expressed as in the following expression 101. Here, in the following equation 101, the matrix A is a transformation matrix representing an affine transformation as represented by the following equation 103. As is apparent from Equation 103, the affine transformation matrix A has six unknowns a ₁₁ to a ₂₃ in its matrix elements.

Here, the affine transformation includes a translation operation represented by the following equation 105, a rotation operation represented by equation 107, a skew (shearing) operation represented by equation 109, and an enlargement / reduction operation represented by equation 111. , Is defined as a combination. In the following equation 105, matrix elements T _x and T _y correspond to the position shift amounts in the x-axis direction and the y-axis direction, respectively, the angle θ in equation 107 corresponds to the rotation angle, and the angle in equation 109 φ corresponds to the skew angle, and the matrix elements S _x and S _y in Equation 111 correspond to the size change amounts in the x-axis direction and the y-axis direction, respectively.

Specifically, the feature amount calculation unit 121 is an affine transformation matrix A represented by the following Expression 103 ′, which is a transformation matrix when an enlargement / reduction operation, a skew operation, a rotation operation, and a translation operation are performed in order. As a result, feature amount calculation processing is performed.

The feature amount calculation unit 121 calculates the affine transformation matrix A represented by the equation 103 ′ based on the following equations 113a and 113b using the identification result of the point correspondence obtained by the corresponding position identification unit 119. . Here, in the following formulas 113a and 113b, there is a process of calculating an inverse matrix of a matrix of 3 rows × 3 columns, and the inverse matrix is calculated using a general solution of an inverse matrix of 3 rows × 3 columns. can do.

Here, in the above formulas 113a and 113b, the sigma symbol represents that the sum of the values represented by the respective elements is calculated for all the points to be used, and N represents the number of points to be used for the calculation. .

The feature quantity calculation unit 121 performs the operations represented by the above formulas 113a and 113b based on the calculation result of the optical flow, whereby the matrix elements a ₁₁ to a _{23 of the} affine transformation matrix A represented by the formulas 103 and 103 ′. Can be specified.

As is clear from the comparison between Expression 103 and Expression 103 ′, when the values of the matrix elements of the affine transformation matrix A can be specified, the size change amounts S _x , S _y , and position are obtained by combining Expression 103 and Expression 103 ′. The shift amounts T _x , T _y , the rotation angle θ, and the skew angle φ can be calculated. Therefore, the feature quantity calculation unit 121 uses the value of the obtained affine transformation matrix A to calculate the optical system feature quantity based on the following formulas 115 to 125.

Further, the feature quantity calculation unit 121 calculates a standard deviation σ of a conversion error represented by the following expression 127 as a feature quantity that represents a distortion that cannot be expressed by affine transformation or a degree of image collapse. Here, in the following expression 127, (P _x , P _y ) is the value of the point specified by the corresponding position specifying unit 119, and (P _x , P _y ) with a hat symbol is obtained. It is an estimated value when (M _x , M _y ) is affine transformed using the affine transformation matrix A. When the value of the standard deviation σ is small, it indicates that the amount of distortion or collapse is small, and when the value of the standard deviation σ is large, it indicates that the amount of distortion or collapse is large.

The feature amount calculation unit 121 calculates an optical system feature amount that represents a difference in the state (optical state) of the optical system 3 as expressed by the above formulas 115 to 127 by performing the above arithmetic processing. .

Since the transformation matrix A in the affine transformation model is defined as a combination of the four transformation matrices represented by the above formulas 105 to 111, the following 24 permutations of the four transformation matrices (= ₄ P ₄ ) Can be considered.

TRKS, TSRK, TSKR, TRSK, TKSR, TKRS,
RTKS, RSTK, RSKT, RTSK, RKST, RKTS,
KRTS, KSRT, KSTR, KRST, KTSR, KTRS,
SRKT, STRK, STKR, SRTK, SKTR, SKRT

Of the 24 permutations, affine transformation models other than “TRKS”, “TSRK”, “TSKR”, “TRSK”, “TKSR”, and “TKRS” that perform the translation operation last are actually the above 4 It is clear that two matrix operations are performed, but the elements (a ₁₃ , a ₂₃ ) indicating the parallel movement amount (position shift amount) of the affine transformation model do not become T _x , T _y . For this reason, T _x and T _y obtained as a result of the above calculation cannot be handled as the vertical shift amount and the horizontal shift amount.

Of the above six types of affine transformation models, for three types of “TSRK”, “TRSK”, and “TKRS”, S _x , S _y , θ, φ, T _x , T _y are obtained by solving simultaneous equations. The operation is relatively complicated and difficult to handle.

In addition, among “TRKS”, “TSKR”, and “TKSR” among the above six types of affine transformation models, S _x , S _y , θ, φ, T _x , and T _y can be found relatively easily. it can. However, since the “TSKR” and “TKSR” models are the models that perform the rotation transformation R first, the enlargement / reduction directions of the enlargement / reduction parameters S _x and S _y are respectively in the X-axis direction. And it will not coincide with the Y-axis direction. Therefore, in such a model, S _x and S _y obtained as a calculation result cannot be handled as a size change amount in the X-axis direction and a size change amount in the Y-axis direction.

From the above, what is useful as an affine transformation model is one model of “TRKS”. Therefore, the feature amount calculation unit 121 according to the present embodiment handles the expression 103 ', which is an affine transformation model of "TRKS", as the transformation matrix A of the affine transformation model.

[Projective transformation model]
Next, the feature amount calculation processing in the feature amount calculation unit 121 when the projective transformation model is employed will be specifically described.
When the point A (M _x , M _y ) and the point B (P _x , P _y ) are related by projective transformation, the relationship can be expressed as in the following Expression 131. Here, in Equation 131 below, λ is a scalar coefficient, and the matrix H is a transformation matrix representing projective transformation as represented by Equation 133 below. As is clear from Expression 133, the projective transformation matrix H has eight unknowns h ₁₁ to h ₃₂ in its matrix elements.

Here, the projective transformation is performed in addition to the translation operation represented by the above equation 105, the rotation operation represented by the equation 107, the skew (shearing) operation represented by the equation 109, and the enlargement / reduction operation represented by the equation 111. , Defined as a combination of projection operations represented by the following Expression 135. In Expression 135 below, matrix elements α and β correspond to projection coefficients in the x-axis direction and the y-axis direction, respectively.

Specifically, the feature quantity calculation unit 121 is a projection matrix represented by the following expression 133 ′, which is a transformation matrix when an enlargement / reduction operation, a skew operation, a rotation operation, a parallel movement operation, and a projection operation are performed in order. Assuming the transformation matrix H, a feature amount calculation process is performed. Note that the projective transformation matrix H represents exactly the same transformation even if all elements are multiplied by a constant, and thus the feature quantity calculation unit 121 assumes a projective transformation matrix H ′ represented by the following expression 133 ″, and features An amount calculation process is performed.

The feature quantity calculation unit 121 calculates the projective transformation matrix H ′ represented by the expression 133 ″ based on the following expression 135 using the identification result of the point correspondence obtained by the corresponding position specifying unit 119. Here, in the following Expression 135, there is a process of calculating an inverse matrix of a matrix of 8 rows × 8 columns. Such an inverse matrix can be calculated using a known numerical operation program.

Here, in the above equation 135, the sigma symbol indicates that the sum of the values represented by each element is calculated for all of the points to be used, and N indicates the number of points to be used for the calculation.

The feature quantity calculation unit 121 identifies each matrix element h ₁₁ to h ₃₂ of the projective transformation matrix H expressed by Expression 133, 133 ″ by performing the calculation expressed by Expression 135 based on the calculation result of the optical flow. can do.

As is clear from the comparison between the expression 133 and the expression 133 ″, when the values of the matrix elements of the projective transformation matrix H can be specified, the size change amounts S _x , S _y , The shift amounts T _x and T _y , the rotation angle θ, the skew angle φ, and the projection coefficients α and β can be calculated. Therefore, the feature quantity calculation unit 121 calculates the optical system feature quantity based on the following formulas 137 to 151 using the value of the obtained projective transformation matrix H.

When the projective transformation model is adopted, the feature amount calculation unit 121 performs conversion when the enlargement / reduction operation, the skew operation, the rotation operation, the translation operation, and the projection operation are sequentially performed as in the above-described equation 133 ′. Instead of the matrix, a projection transformation matrix H represented by the following expression 153 may be assumed to perform the feature amount calculation process. A projective transformation matrix H represented by the following Expression 153 is a transformation matrix when an enlargement / reduction operation, a skew operation, a rotation operation, a projection operation, and a parallel movement operation are sequentially performed.

Also in this case, the feature amount calculation unit 121 performs the calculation represented by the above equation 135 based on the calculation result of the optical flow, and each matrix element h ₁₁ to h of the projective transformation matrix H represented by the equations 133 and 153. ₃₂ is specified. After that, the feature quantity calculation unit 121 can calculate the optical system feature quantity in the same manner as in the case of using the expression 133 ″ by combining the expression 133 and the expression 153.

Since the transformation matrix H in the projective transformation model is defined as a combination of the five transformation matrices represented by the above formulas 105 to 111 and 135, there are 120 permutations of the five transformation matrices (= ₅ P ₅ ) Is considered.

However, as with the affine transformation model, it is preferable that the translation operation and the projection operation are performed as much as possible (that is, one of the operations is performed last and the other operation is performed second from the last). Conceivable. Therefore, as the projective transformation model according to the present embodiment, it is preferable to use the “BTRKS” model represented by the above-described equations 133 ′ and 133 ″ or the “TBRKS” model represented by the above-described equation 153.

As described above, the affine transformation model and the projective transformation model employed in the feature amount calculation unit 121 according to the present embodiment have been specifically described.

By performing the feature amount calculation process as described above, the feature amount calculation unit 121 according to the present embodiment calculates, for example, an optical system feature amount as illustrated in FIG. The optical system feature amount is a quantification of the state of the optical system 3, and the state of the optical system 3 can be objectively determined by referring to the feature amount.

[Selection of geometric transformation model]
Next, with reference to FIG. 9, the selection policy of two types of geometric transformation models, an affine transformation model and a projective transformation model, will be briefly described.

The feature quantity calculation unit 121 according to the present embodiment can use two types of geometric transformation models, an affine transformation model and a projective transformation model. These two types of models are as shown in FIG. It is preferable to use properly from the viewpoint.

In other words, when the optical system 3 to be imaged by the imaging unit 101 is an optical system in which there is no tilt of the display image or the tilt of the display image is negligible, when calculating the optical system feature quantity, It is preferable to adopt a conversion model. As described above, the affine transformation model has a small number of unknowns, and it is possible to obtain a highly accurate feature amount calculation result while suppressing the calculation cost.

On the other hand, when the optical system 3 to be imaged by the imaging unit 101 is an optical system in which there is a negligible amount of tilting, it is preferable to employ a projective transformation model when calculating the optical system feature amount. Since the projective transformation model is a model that also takes into account the projection operation, it is possible to accurately estimate the amount of collapse, but since the number of unknowns is larger than the affine transformation model, the computational cost is higher than that of the affine transformation model. Become.

Here, the fall of the display image is a situation where the display image is not directly facing the imaging unit 101. Specifically, the tilt of the display image means that the upper end of the display image is located on the back side with respect to the lower end along the optical axis connecting the imaging unit 101 and the optical system 3 (or located on the near side). And the case where the right end portion of the display image is located on the back side (or located on the near side) with respect to the left end portion.

As specific examples when the affine transformation model is advantageous, for example, as shown in FIG. 9, (a) when measuring the zoom amount (magnification) in the zoom optical system, or (b) only the feature amount excluding the tilt And (c) when measuring an HMD having an optical system that generates a stereo image from a single display, and (d) when measuring a stereo optical system such as a microscope and binoculars.

Further, as a specific example when the projective transformation model is advantageous, for example, as shown in FIG. 9, (a) when measuring an optical system whose imaging surface can be inclined with respect to the optical system, or (b) 2 When measuring HMD which has an optical system which generates a stereo picture from two displays, etc. are mentioned.

In the feature amount calculation unit 121, which model is to be adopted is set in advance according to the above policy according to the characteristics of the optical system 3 to be measured, etc., prior to the measurement of the optical system 3. Just keep it.

The feature amount calculation unit 121 outputs information regarding the optical system feature amount calculated as described above as shown in FIG. 8 to the optical system state determination unit 123. Further, the feature amount calculation unit 121 may output information about the optical system feature amount calculated as described above to various information processing apparatuses such as various computers and various servers existing outside the information processing apparatus 10. Good.

The optical system state determination unit 123 is realized by, for example, a CPU, a ROM, a RAM, and the like. The optical system state determination unit 123 determines the state of the optical system 3 based on the optical system feature amount calculated by the feature amount calculation unit 121. Specifically, in the optical system state determination unit 123, a determination threshold is set in advance for each optical system feature amount as shown in FIG. 8, and the magnitude of the calculated optical system feature amount and the determination threshold value is small. The state of the optical system 3 is determined by comparison.

The determination result of the state of the optical system determined by the optical system state determination unit 123 may be output to the user of the information processing apparatus 10 by various methods, or may be directly fed back to the optical device 1. Good. By using such a determination result, the state of the optical system 3 provided in the optical apparatus 1 can be specifically grasped, and the optical system 3 can be adjusted more easily.

FIG. 10 simply shows an example of an optical system state determination process when a head-mounted display having a stereo optical system is a processing target.

For example, as illustrated in FIG. 10, the optical system state determination unit 123 refers to the feature amount related to the calculated size (S _x , S _y ) and the feature amount is not approximately 1 (that is, the feature amount). Can be determined that the magnification is different between the right and left optical systems.

Further, the optical system state determination unit 123 pays attention to the rotation angle (θ), and the optical system is tilted when the feature amount is not almost 0 (that is, when the feature amount exceeds a predetermined threshold). (That is, it can be determined that the optical system is rotating).

Similarly, the optical system state determination unit 123 pays attention to the skew angle (φ), and when the feature amount is not almost zero (that is, when the feature amount exceeds a predetermined threshold), the optical system is distorted. It can be determined that there is.

The optical system state determination unit 123 focuses on the shift amount (T _x , T _y ), and when the feature amount does not become almost zero (that is, when the feature amount exceeds a predetermined threshold), the optical system state determination unit 123 focuses on the shift amount (T _x , T _y ). It can be determined that there are cases where the mounting position and angle are not correct or the left and right optical systems are mounted in reverse. In particular, in the case of a stereo optical system, the shift amount T _y is a feature amount representing the parallax in the optical system 3, and thus the determination result is an important guideline when evaluating the stereo optical system.

Further, the optical system state determination unit 123 pays attention to the projection coefficients (α, β), and the optical system is tilted when the feature amount is not almost zero (that is, when the feature amount exceeds a predetermined threshold). It can be determined that the display has fallen or the display has fallen.

In addition, the optical system state determination unit 123 focuses on the standard deviation (σ) of the conversion error, and when the feature amount is not almost zero (that is, when the feature amount exceeds a predetermined threshold), It can be determined that there is distortion.

As described above, by knowing in advance how the optical system feature amount behaves depending on the state of the optical system 3, the optical system state determination unit 123 can calculate the optical system feature amount that is sequentially calculated. Based on this, the state of the optical system 3 can be determined.

The configuration of the arithmetic processing unit 103 included in the information processing apparatus 10 according to the present embodiment has been described in detail above with reference to FIGS.

Heretofore, an example of the function of the information processing apparatus 10 according to the present embodiment has been shown. Each component described above may be configured using a general-purpose member or circuit, or may be configured by hardware specialized for the function of each component. In addition, the CPU or the like may perform all functions of each component. Therefore, it is possible to appropriately change the configuration to be used according to the technical level at the time of carrying out the present embodiment.

It should be noted that a computer program for realizing each function of the information processing apparatus according to the present embodiment as described above can be produced and installed in a personal computer or the like. In addition, a computer-readable recording medium storing such a computer program can be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Further, the above computer program may be distributed via a network, for example, without using a recording medium.

<Flow of information processing method>
Next, an example of a flow of an information processing method performed by the information processing apparatus 10 according to the present embodiment will be briefly described with reference to FIG. FIG. 11 is a flowchart illustrating an example of the flow of the information processing method according to the present embodiment.

In the information processing method according to the present embodiment, first, a random pattern image is generated by the random pattern image generation unit 113 by the method described above (step S101). Thereafter, the generated random pattern image is output to the optical device 1 via the data output unit 115 (step S103).

When the random pattern image is output via the optical system 3 of the optical device 1, the imaging unit 101 captures the random pattern image output from the optical device 1 under the control of the imaging control unit 111 (step S105). ) To generate imaging data. The imaging data generated by the imaging unit 101 is transmitted to the corresponding position specifying unit 119 via the data acquisition unit 117.

The corresponding position specifying unit 119 specifies the corresponding position of the point of interest by using the two captured images by referring to the transmitted imaging data and using the method using the optical flow as described above (step S107). ). Thereafter, the corresponding position specifying unit 119 outputs information representing the result of specifying the corresponding position of the point to the feature amount calculating unit 121.

Based on the affine transformation model or projective transformation model described above, the feature amount calculation unit 121 uses the information on the corresponding positions of the points by the corresponding position specifying unit 119 to specify a conversion matrix between two captured images ( Step S109). After that, the feature quantity calculation unit 121 calculates various optical system feature quantities as shown in FIG. 8 based on the identified conversion matrix (step S111). When the feature amount calculation unit 121 calculates the optical system feature amount, the feature amount calculation unit 121 outputs information regarding the calculated optical system feature amount to the optical system state determination unit 123.

The optical system state determination unit 123 determines the state of the focused optical system 3 based on the optical system feature amount calculated by the feature amount calculation unit 121 (step S113). As a result, in the information processing method according to the present embodiment, it is possible to calculate a feature amount representing a difference in the optical system 3 and to determine the state of the optical system 3 based on the feature amount.

The flow of the information processing method according to the present embodiment has been briefly described above with reference to FIG.

<About electro-optical equipment>
Next, the electro-optical device according to the present embodiment will be briefly described with reference to FIGS. 12 and 13. 12 and 13 are block diagrams illustrating an example of the configuration of the electro-optical device according to the present embodiment.

By using the optical system characteristic amount calculated by the information processing apparatus 10 according to the present embodiment, the optical system determination result, and the like, it is possible to realize the following electro-optical device.

That is, as schematically shown in FIG. 12, the electro-optical device 20 according to the present embodiment mainly includes a predetermined optical system 201, an adjustment unit 203, and a storage unit 205.

The optical system 201 is a mechanism for outputting various images, such as still images and moving images, held or acquired by the electro-optical device 20 to the outside of the electro-optical device 20. The optical system 201 is configured using various optical elements such as a lens, a mirror, and a filter.

The type of the optical system 201 included in the electro-optical device 20 is not particularly limited, and examples thereof include known optical systems such as a zoom optical system, a stereo optical system, and an optical system that generates a stereo image. Can do. By having such an optical system 201, the electro-optical device 20 functions as, for example, a stereoscopic display device such as a camera, a camcorder, a telescope, a microscope, binoculars, and a head-mounted display.

In addition, in the optical system 201 according to the present embodiment, in order to change the state (optical state) of the optical system 201 by changing the installation position, the installation angle, and the like of various optical elements constituting the optical system 201, Various drive mechanisms (not shown) are provided. Such a drive mechanism is not particularly limited, and examples thereof include a voice coil motor and a piezo element.

The adjustment unit 203 is realized by a CPU, a ROM, a RAM, a communication device, and the like. The adjustment unit 203 acquires feature amount information based on the optical system feature amount generated by the information processing apparatus 10 according to the present embodiment capturing a random pattern image output from the optical system 201, and the acquired feature amount information. The optical system 201 is adjusted based on the above. Here, the feature amount information acquired by the adjustment unit 203 from the information processing device 10 is not only the optical system feature amount generated by the information processing device 10 but also the state of the optical system 201 obtained based on the optical system feature amount. Information indicating the determination result is also included.

The adjustment unit 203 adjusts the state of the optical system 201 to an appropriate state by controlling the drive mechanism provided in the optical system 201 based on the feature amount information acquired from the information processing apparatus 10. The control method of the drive mechanism is not particularly limited, and feedback control according to the acquired feature amount information may be performed, or the optical system feature amount stored in the storage unit 205 or the like described later. The drive mechanism may be controlled by referring to a database associated with the drive mechanism control method.

The adjustment unit 203 automatically adjusts the optical system 201 based on the feature amount information acquired from the information processing apparatus 10, so that the state of the optical system 201 in the electro-optical device 20 always maintains an appropriate state. It becomes possible.

The storage unit 205 is realized by, for example, a RAM or a storage device provided in the electro-optical device 20 according to the present embodiment. The storage unit 205 needs to store entity data of various images output from the optical system 201, various databases and various programs used for processing in the adjustment unit 203, and some processing. Various parameters, progress of processing, and the like may be recorded as appropriate.

The storage unit 205 can be freely accessed by each processing unit such as the optical system 201 and the adjustment unit 203 to write and read data.

The overall configuration of the electro-optical device 20 according to the present embodiment has been described in detail above with reference to FIG.

In FIG. 12, the aspect in which the feature amount information is acquired from the information processing apparatus 10 provided outside the electro-optical device 20 and the adjustment unit 203 adjusts the optical system 201 has been described. The device itself may have the function of the information processing apparatus 10.

For example, as illustrated in FIG. 13, the electro-optical device 20 having the function of the information processing apparatus 10 includes an optical system 201, an adjustment unit 203, a storage unit 205, an imaging unit 207, and an arithmetic processing unit 209. Prepare mainly.

Here, the optical system 201 and the adjustment unit 203 have the same functions as the optical system 201 and the adjustment unit 203 included in the electro-optical device 20 described with reference to FIG. Therefore, detailed description is omitted below.

Further, the storage unit 205 has both the function of the storage unit 205 in the electro-optical device 20 described with reference to FIG. 12 and the function of the storage unit 105 in the information processing apparatus 10, and will be described in detail below. The detailed explanation is omitted.

Furthermore, the imaging unit 207 and the arithmetic processing unit 209 have the same functions as the imaging unit 101 and the arithmetic processing unit 103 included in the information processing apparatus 10 according to the present embodiment, respectively, and have the same effects. Detailed description is omitted below.

In the electro-optical device 20 in this example, the random pattern image generated by the arithmetic processing unit 209 is output from the optical system 201, and the output random pattern image is captured by the imaging unit 207, thereby corresponding to the random pattern image. Imaging data is generated. The arithmetic processing unit 209 performs the arithmetic processing as described above based on the generated imaging data, thereby generating information representing the optical system feature amount and the optical system state determination result. The adjustment unit 203 can appropriately maintain the state of the optical system 201 by adjusting the optical system 201 based on the feature amount information generated by the arithmetic processing unit 209.

Heretofore, an example of the function of the electro-optical device 20 according to the present embodiment has been shown. Each component described above may be configured using a general-purpose member or circuit, or may be configured by hardware specialized for the function of each component. In addition, the CPU or the like may perform all functions of each component. Therefore, it is possible to appropriately change the configuration to be used according to the technical level at the time of carrying out the present embodiment.

It should be noted that a computer program for realizing each function of the electro-optical device according to the present embodiment as described above can be produced and mounted on a personal computer or the like. In addition, a computer-readable recording medium storing such a computer program can be provided. The recording medium is, for example, a magnetic disk, an optical disk, a magneto-optical disk, a flash memory, or the like. Further, the above computer program may be distributed via a network, for example, without using a recording medium.

(About hardware configuration)
Next, the hardware configurations of the information processing apparatus 10 and the electro-optical device 20 according to the embodiment of the present disclosure will be described in detail with reference to FIG. FIG. 14 is a block diagram for explaining a hardware configuration of the information processing apparatus 10 and the electro-optical device 20 according to the embodiment of the present disclosure. In FIG. 14, the imaging unit 101 included in the information processing apparatus 10 and the imaging unit 207 that can be included in the electro-optical device 20 are not illustrated.

The information processing apparatus 10 and the electro-optical device 20 mainly include a CPU 901, a ROM 903, and a RAM 905. The information processing apparatus 10 and the electro-optical device 20 further include a host bus 907, a bridge 909, an external bus 911, an interface 913, an input device 915, an output device 917, a storage device 919, and a drive 921. A connection port 923, and a communication device 925.

The CPU 901 functions as an arithmetic processing device and a control device, and in accordance with various programs recorded in the ROM 903, the RAM 905, the storage device 919, or the removable recording medium 927, the entire operation in the information processing device 10 and the electro-optical device 20 or one of them. Control part. The ROM 903 stores programs used by the CPU 901, calculation parameters, and the like. The RAM 905 primarily stores programs used by the CPU 901, parameters that change as appropriate during execution of the programs, and the like. These are connected to each other by a host bus 907 constituted by an internal bus such as a CPU bus.

The host bus 907 is connected to an external bus 911 such as a PCI (Peripheral Component Interconnect / Interface) bus via a bridge 909.

The input device 915 is an operation means operated by the user such as a mouse, a keyboard, a touch panel, a button, a switch, and a lever. Further, the input device 915 may be, for example, remote control means (so-called remote control) using infrared rays or other radio waves, or a mobile phone or PDA corresponding to the operation of the information processing device 10 and the electro-optical device 20. The external connection device 929 such as the above may be used. Furthermore, the input device 915 includes an input control circuit that generates an input signal based on information input by a user using the above-described operation means and outputs the input signal to the CPU 901, for example. A user of the information processing apparatus 10 and the electro-optical device 20 operates the input device 915 to input various data or instruct a processing operation to the information processing apparatus 10 and the electro-optical device 20. Can do.

The output device 917 is a device that can notify the user of the acquired information visually or audibly. Examples of such devices include CRT display devices, liquid crystal display devices, plasma display devices, EL display devices and display devices such as lamps, audio output devices such as speakers and headphones, printer devices, mobile phones, and facsimiles. The output device 917 outputs results obtained by various processes performed by the information processing device 10 and the electro-optical device 20, for example. Specifically, the display device displays results obtained by various processes performed by the information processing apparatus 10 and the electro-optical device 20 as text or images. On the other hand, the audio output device converts an audio signal composed of reproduced audio data, acoustic data, and the like into an analog signal and outputs the analog signal.

The storage device 919 is a data storage device configured as an example of a storage unit of the information processing device 10 and the electro-optical device 20. The storage device 919 includes, for example, a magnetic storage device such as an HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, or a magneto-optical storage device. The storage device 919 stores programs executed by the CPU 901, various data, various data acquired from the outside, and the like.

The drive 921 is a reader / writer for a recording medium, and is built in or externally attached to the information processing apparatus 10 and the electro-optical device 20. The drive 921 reads information recorded on a removable recording medium 927 such as a mounted magnetic disk, optical disk, magneto-optical disk, or semiconductor memory, and outputs the information to the RAM 905. In addition, the drive 921 can write a record on a removable recording medium 927 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory. The removable recording medium 927 is, for example, a DVD medium, an HD-DVD medium, a Blu-ray (registered trademark) medium, or the like. The removable recording medium 927 may be a compact flash (registered trademark) (CompactFlash: CF), a flash memory, or an SD memory card (Secure Digital memory card). Further, the removable recording medium 927 may be, for example, an IC card (Integrated Circuit card) on which a non-contact IC chip is mounted, an electronic device, or the like.

The connection port 923 is a port for directly connecting a device to the information processing apparatus 10 and the electro-optical device 20. Examples of the connection port 923 include a USB (Universal Serial Bus) port, an IEEE 1394 port, a SCSI (Small Computer System Interface) port, and the like. As another example of the connection port 923, there are an RS-232C port, an optical audio terminal, an HDMI (High-Definition Multimedia Interface) port, and the like. By connecting the external connection device 929 to the connection port 923, the information processing apparatus 10 and the electro-optical device 20 directly acquire various data from the external connection device 929 or provide various data to the external connection device 929. To do.

The communication device 925 is a communication interface configured with, for example, a communication device for connecting to the communication network 931. The communication device 925 is, for example, a communication card for a wired or wireless LAN (Local Area Network), Bluetooth (registered trademark), or WUSB (Wireless USB). The communication device 925 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), or a modem for various communication. The communication device 925 can transmit and receive signals and the like according to a predetermined protocol such as TCP / IP, for example, with the Internet or other communication devices. The communication network 931 connected to the communication device 925 is configured by a wired or wireless network, and may be, for example, the Internet, a home LAN, infrared communication, radio wave communication, satellite communication, or the like. .

Heretofore, an example of a hardware configuration capable of realizing the functions of the information processing apparatus 10 and the electro-optical device 20 according to the embodiment of the present disclosure has been shown. Each component described above may be configured using a general-purpose member, or may be configured by hardware specialized for the function of each component. Therefore, it is possible to change the hardware configuration to be used as appropriate according to the technical level at the time of carrying out this embodiment.

The preferred embodiments of the present disclosure have been described in detail above with reference to the accompanying drawings, but the technical scope of the present disclosure is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field of the present disclosure can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that it belongs to the technical scope of the present disclosure.

In addition, the effects described in this specification are merely illustrative or illustrative, and are not limited. That is, the technology according to the present disclosure can exhibit other effects that are apparent to those skilled in the art from the description of the present specification in addition to or instead of the above effects.

The following configurations also belong to the technical scope of the present disclosure.
(1)
An imaging unit that captures a plurality of random pattern images output from an optical device having a predetermined optical system in a situation where the state of the optical system is different, and generates a plurality of imaging data respectively corresponding to the state of the optical system; ,
By using the plurality of imaging data captured by the imaging unit and calculating the optical flow between the plurality of captured images at any point of at least three or more points in the captured image corresponding to the imaging data, A corresponding position specifying unit for specifying positions corresponding to each other between the captured images;
A feature amount calculation unit that calculates a feature amount representing a difference in the optical system between the plurality of captured images of interest based on the specification result obtained by the corresponding position specification unit;
An information processing apparatus comprising:
(2)
A random pattern image generation unit that generates the random pattern image used in the optical device based on an M-sequence signal;
The random pattern image generation unit combines a plurality of random pattern images in which the size of dots constituting the random pattern image is equal to or less than ½ of the window size that is a processing unit in the optical flow calculation process The information processing apparatus according to (1), which generates a random pattern image.
(3)
The feature amount calculation unit uses the identification result to obtain a geometrical relationship between a first image corresponding to the first imaging data and a second image corresponding to the second imaging data. Calculate a transformation matrix representing the correspondence,
Using the calculated transformation matrix, the feature amount representing a deviation of the optical system, a change in image size caused by the optical system, a tilt of the optical system, and a distortion of the optical system is calculated. (1) Or the information processing apparatus according to (2).
(4)
The feature amount calculation unit includes:
As the transformation matrix, calculate an affine transformation matrix between the first image and the second image,
Using the calculated affine transformation matrix, at least a size change amount, a position shift amount, a rotation angle, and a skew angle between the first image and the second image are calculated as the feature amount. The information processing apparatus according to 3).
(5)
The feature amount calculation unit calculates a projective transformation matrix between the first image and the second image as the transformation matrix,
Using the calculated projective transformation matrix, at least a size change amount, a position shift amount, a rotation angle, a skew angle, and a projective transformation coefficient between the first image and the second image are used as the feature amount. The information processing apparatus according to (3), which calculates.
(6)
The information processing apparatus according to (4), wherein the affine transformation matrix is a transformation matrix when an enlargement / reduction operation, a skew operation, a rotation operation, and a translation operation are sequentially performed.
(7)
The projective transformation matrix is a transformation matrix obtained when an enlargement / reduction operation, a skew operation, a rotation operation, a translation operation, and a projection operation are sequentially performed, or an enlargement / reduction operation, a skew operation, a rotation operation, a projection operation, and a parallel operation. The information processing apparatus according to (5), which is a conversion matrix when movement operations are performed in order.
(8)
The information processing apparatus according to any one of (1) to (7), further comprising: an optical system state determination unit that determines the state of the optical system based on the feature amount calculated by the feature amount calculation unit. .
(9)
The information processing apparatus according to any one of (1) to (8), wherein the optical apparatus is an optical apparatus having a zoom optical system, an optical apparatus having a stereo optical system, or an optical apparatus that generates a stereo image. apparatus.
(10)
The information processing apparatus according to (9), wherein the optical device having the zoom optical system is a camera, a camcorder, or a telescope.
(11)
The information processing apparatus according to (9), wherein the optical apparatus having the stereo optical system is a microscope or binoculars.
(12)
The information processing apparatus according to (9), wherein the optical device that generates the stereo image is a head-mounted display.
(13)
Taking a plurality of random pattern images output from an optical device having a predetermined optical system in a situation where the state of the optical system is different, and generating a plurality of imaging data corresponding to the state of the optical system,
By using the plurality of captured image data and calculating an optical flow between the plurality of captured images at an arbitrary point of at least three or more points in the captured image corresponding to the captured data, between the plurality of captured images Identifying positions corresponding to each other in
Calculating a feature amount representing a difference in the optical system between the plurality of captured images of interest based on the identification results of the positions corresponding to each other;
Including an information processing method.
(14)
An imaging unit that captures a plurality of random pattern images output from an optical device having a predetermined optical system in a situation where the state of the optical system is different, and generates a plurality of imaging data respectively corresponding to the state of the optical system Computer
By using the plurality of imaging data captured by the imaging unit and calculating the optical flow between the plurality of captured images at any point of at least three or more points in the captured image corresponding to the imaging data, A corresponding position specifying function for specifying positions corresponding to each other between the captured images;
A feature amount calculation function for calculating a feature amount representing a difference in the optical system between the plurality of captured images of interest based on the specification result obtained by the corresponding position specifying function;
A program to realize
(15)
An optical system for displaying a predetermined image on a predetermined display screen;
An imaging unit that captures a plurality of random pattern images output from an optical device having a predetermined optical system in a situation where the state of the optical system is different, and generates a plurality of imaging data respectively corresponding to the state of the optical system; , By using the plurality of imaging data captured by the imaging unit and calculating an optical flow between the plurality of captured images at any point of at least three or more points in the captured image corresponding to the imaging data, Differences in the optical system between the plurality of captured images of interest based on a corresponding position specifying unit that specifies positions corresponding to each other between the plurality of captured images and a specifying result obtained by the corresponding position specifying unit. A feature amount calculation unit that calculates a feature amount that represents the feature amount information based on the feature amount calculated by the information processing apparatus, and based on the feature amount information An adjustment unit for adjusting the optical system,
An electro-optical device comprising:

DESCRIPTION OF SYMBOLS 10 Information processing apparatus 20 Electro-optical apparatus 101,207 Imaging part 103,209 Arithmetic processing part 105,205 Storage part 111 Imaging control part 113 Random pattern image generation part 115 Data output part 117 Data acquisition part 119 Corresponding position specific | specification part 121 Feature-value Calculation unit 123 Optical system state determination unit 201 Optical system 203 Adjustment unit

Claims (15)

1. An imaging unit that captures a plurality of random pattern images output from an optical device having a predetermined optical system in a situation where the state of the optical system is different, and generates a plurality of imaging data respectively corresponding to the state of the optical system; ,
   By using the plurality of imaging data captured by the imaging unit and calculating the optical flow between the plurality of captured images at any point of at least three or more points in the captured image corresponding to the imaging data, A corresponding position specifying unit for specifying positions corresponding to each other between the captured images;
   A feature amount calculation unit that calculates a feature amount representing a difference in the optical system between the plurality of captured images of interest based on the specification result obtained by the corresponding position specification unit;
   An information processing apparatus comprising:
2. A random pattern image generation unit that generates the random pattern image used in the optical device based on an M-sequence signal;
   The random pattern image generation unit combines a plurality of random pattern images in which the size of dots constituting the random pattern image is equal to or less than ½ of the window size that is a processing unit in the optical flow calculation process The information processing apparatus according to claim 1, wherein the information processing apparatus generates a random pattern image.
3. The feature amount calculation unit uses the identification result to obtain a geometrical relationship between a first image corresponding to the first imaging data and a second image corresponding to the second imaging data. Calculate a transformation matrix representing the correspondence,
   2. The feature amount representing a deviation of the optical system, a change in image size caused by the optical system, a tilt of the optical system, and a distortion of the optical system is calculated using the calculated conversion matrix. The information processing apparatus described in 1.
4. The feature amount calculation unit includes:
   As the transformation matrix, calculate an affine transformation matrix between the first image and the second image,
   The calculated affine transformation matrix is used to calculate at least a size change amount, a position shift amount, a rotation angle, and a skew angle between the first image and the second image as the feature amount. Item 4. The information processing device according to Item 3.
5. The feature amount calculation unit calculates a projective transformation matrix between the first image and the second image as the transformation matrix,
   Using the calculated projective transformation matrix, at least a size change amount, a position shift amount, a rotation angle, a skew angle, and a projective transformation coefficient between the first image and the second image are used as the feature amount. The information processing apparatus according to claim 3, which calculates the information processing apparatus.
6. The information processing apparatus according to claim 4, wherein the affine transformation matrix is a transformation matrix when an enlargement / reduction operation, a skew operation, a rotation operation, and a translation operation are sequentially performed.
7. The projective transformation matrix is a transformation matrix obtained when an enlargement / reduction operation, a skew operation, a rotation operation, a translation operation, and a projection operation are sequentially performed, or an enlargement / reduction operation, a skew operation, a rotation operation, a projection operation, and a parallel operation. The information processing apparatus according to claim 5, wherein the information processing apparatus is a transformation matrix when the moving operations are sequentially performed.
8. The information processing apparatus according to claim 1, further comprising: an optical system state determination unit that determines a state of the optical system based on the feature amount calculated by the feature amount calculation unit.
9. The information processing apparatus according to claim 1, wherein the optical device is an optical device having a zoom optical system, an optical device having a stereo optical system, or an optical device that generates a stereo image.
10. The information processing apparatus according to claim 9, wherein the optical device having the zoom optical system is a camera, a camcorder, or a telescope.
11. The information processing apparatus according to claim 9, wherein the optical apparatus having the stereo optical system is a microscope or binoculars.
12. The information processing apparatus according to claim 9, wherein the optical device that generates the stereo image is a head-mounted display.
13. Taking a plurality of random pattern images output from an optical device having a predetermined optical system in a situation where the state of the optical system is different, and generating a plurality of imaging data corresponding to the state of the optical system,
    By using the plurality of captured image data and calculating an optical flow between the plurality of captured images at an arbitrary point of at least three or more points in the captured image corresponding to the captured data, between the plurality of captured images Identifying positions corresponding to each other in
    Calculating a feature amount representing a difference in the optical system between the plurality of captured images of interest based on the identification results of the positions corresponding to each other;
    Including an information processing method.
14. An imaging unit that captures a plurality of random pattern images output from an optical device having a predetermined optical system in a situation where the state of the optical system is different, and generates a plurality of imaging data respectively corresponding to the state of the optical system Computer
    By using the plurality of imaging data captured by the imaging unit and calculating the optical flow between the plurality of captured images at any point of at least three or more points in the captured image corresponding to the imaging data, A corresponding position specifying function for specifying positions corresponding to each other between the captured images;
    A feature amount calculation function for calculating a feature amount representing a difference in the optical system between the plurality of captured images of interest based on the specification result obtained by the corresponding position specifying function;
    A program to realize
15. An optical system for displaying a predetermined image on a predetermined display screen;
    An imaging unit that captures a plurality of random pattern images output from an optical device having a predetermined optical system in a situation where the state of the optical system is different, and generates a plurality of imaging data respectively corresponding to the state of the optical system; , By using the plurality of imaging data captured by the imaging unit and calculating an optical flow between the plurality of captured images at any point of at least three or more points in the captured image corresponding to the imaging data, Differences in the optical system between the plurality of captured images of interest based on a corresponding position specifying unit that specifies positions corresponding to each other between the plurality of captured images and a specifying result obtained by the corresponding position specifying unit. A feature amount calculation unit that calculates a feature amount that represents the feature amount information based on the feature amount calculated by the information processing apparatus, and based on the feature amount information An adjustment unit for adjusting the optical system,
    An electro-optical device comprising:

Images