JP2017130794A - Information processing apparatus, evaluation chart, evaluation system, and performance evaluation method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To efficiently find the conditions for performing information processing using captured images with high accuracy.SOLUTION: An imaging apparatus 12 captures an evaluation chart 200, generates captured image data in the same way as during operation and transmits the data to an information processing apparatus 10. The information processing apparatus 10 performs corner detection of chessboard shaped large rectangle, in the evaluation chart 200, and acquires the coordinate of a small rectangular corner of gray represented internally, as a correct answer coordinate. Furthermore, an index representing the performance, in view point of feature point extraction, is calculated by extracting a feature point by the algorithm similar to that during operation, and comparing the point with the correct answer coordinate. When the index does not reach a target value, the information processing apparatus 10 updates and adjusts the setting of adjustment object, out of the settings of various kinds of processing in the imaging apparatus 12.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information processing system that performs information processing including recognition of a real object using a captured image, an imaging device included therein, a mat, and an image generation method used.

  A game is known in which a user's body or marker is photographed with a camera, the area of the image is replaced with another image, and displayed on a display (for example, see Patent Document 1). The technology to acquire the position and movement of the subject and the camera itself and to recognize what the subject is by detecting and analyzing the image in the captured image is not limited to the camera mounted on the game device or the information terminal. It is also widely used in information processing using security cameras, in-vehicle cameras, robot cameras, and the like.

  In the technology as described above, how to accurately acquire an image of an object and its modeling features is always an important issue. For this reason, various techniques have been proposed for accurately extracting the contours and characteristic portions of an object in a captured image (see, for example, Patent Document 2).

European Patent Application No. 0999518 JP 2010-182078 A

  Characteristic locations on the captured image are the shooting environment such as brightness, the performance of the device used, such as an image sensor, various processes applied to the data from imaging to feature detection, specifically demosaic processing, correction processing, The ease of extraction varies depending on conditions and methods in compression encoding processing, communication processing, and the like. As a result of many factors related to each other in this way, information processing using the detected features is performed, so that sufficient processing accuracy can be achieved while satisfying constraints in terms of system manufacturing costs and processing costs. Finding the optimal conditions to get is not easy.

  The present invention has been made in view of these problems, and an object of the present invention is to provide a technique capable of efficiently finding processing conditions for performing information processing using a captured image with high accuracy.

  In order to solve the above problems, an aspect of the present invention relates to an information processing apparatus. The information processing apparatus includes a chart image acquisition unit that acquires captured image data of an evaluation chart representing a predetermined pattern from an imaging apparatus and develops the data as an image in a memory, and a feature point extraction that extracts a feature point from the captured image And a feature coordinate extraction unit that acquires the position coordinates of the feature points to be extracted from the captured image as correct coordinates, and the feature points and correct coordinates extracted by the feature point extraction unit using a method different from that of the feature point extraction unit. And a performance index calculation unit that calculates and outputs a performance index that quantifies the performance of obtaining an image suitable for feature point extraction.

  Another embodiment of the present invention relates to an evaluation chart. This evaluation chart is an evaluation chart for evaluating the performance of the information processing apparatus to obtain an image suitable for feature point extraction, and is a chessboard-like black and white for detecting a corner from the photographed image. A rectangle and a gray figure represented inside each rectangle, with coordinates obtained based on corner detection results and having feature points for comparison with the results of feature point extraction processing performed separately. Features.

  Yet another embodiment of the present invention also relates to an evaluation chart. This evaluation chart is an evaluation chart for evaluating the ability of the information processing apparatus to obtain an image suitable for feature point extraction, and forms one vacant space in the vertical and horizontal directions detected from the captured image. A plurality of black circles arranged in such a way that coordinates are acquired based on the detection results of the circles, and feature points for comparison with the results of feature point extraction processing performed separately, inside and in each circle And a gray figure represented.

  Yet another embodiment of the present invention relates to an evaluation system. This evaluation system includes an imaging device that captures an evaluation chart in which a predetermined pattern is represented, and an information processing device that acquires data of the captured image and evaluates the ability to obtain an image suitable for feature point extraction. The information processing apparatus includes a chart image acquisition unit that acquires captured image data from an imaging device and develops the image as an image in a memory, a feature point extraction unit that extracts a feature point from the captured image, and a feature point extraction unit. The correct coordinate acquisition unit that acquires the position coordinates of the feature points that should be extracted from the captured image as the correct coordinates, and the performance index that quantifies the performance based on the feature points and correct coordinates extracted by the feature point extraction unit A performance index calculation unit for calculating, and an adjustment unit for adjusting any of the settings used for processing in the imaging device when the performance index does not reach a preset target value. Is provided with a parameter storage unit that stores settings adjusted by the adjustment unit, and a data forming unit that generates captured image data according to the settings stored in the parameter storage unit and transmits the data to the information processing apparatus. .

  Yet another embodiment of the present invention relates to a performance evaluation method. This performance evaluation method includes a step of acquiring data of a photographed image of an evaluation chart representing a predetermined pattern from an imaging device and developing the data in a memory, a step of extracting feature points from the photographed image, and a feature point Extracting feature points based on the feature points and correct coordinates extracted in the step of acquiring the position coordinates of the feature points to be extracted from the captured image as correct coordinates and the step of extracting feature points by a method different from the extracting step And a step of calculating and outputting a performance index obtained by quantifying the performance of obtaining an image suitable for.

  It should be noted that any combination of the above-described constituent elements and a conversion of the expression of the present invention between a method, an apparatus, a system, a recording medium, a computer program, etc. are also effective as an aspect of the present invention.

  According to the present invention, it is possible to efficiently find conditions for performing information processing using captured images with high accuracy.

It is a figure which shows the structural example of the evaluation system which can apply this Embodiment. It is a figure which shows the internal circuit structure of the information processing apparatus in this Embodiment. It is a figure which shows the factor assumed as an object of adjustment in this Embodiment. It is a figure which shows the structure of the functional block of the information processing apparatus and imaging device at the time of evaluation of this Embodiment. It is a figure which illustrates the chart for evaluation in this Embodiment. It is a figure for demonstrating the method in which the correct coordinate acquisition part in this Embodiment acquires a correct coordinate from the picked-up image of the chart for evaluation of FIG. It is a figure which compares the small rectangular corner from which the correct coordinate acquisition part acquired the coordinate in this Embodiment with the feature point which the feature point extraction part extracted. In this Embodiment, it is a figure which illustrates distribution of the feature point extracted with respect to the image which image | photographed the chart for evaluation, and the calculated F value. In this Embodiment, it is a figure which illustrates distribution of the feature point extracted with respect to the image which image | photographed the chart for evaluation, and the calculated F value. In this Embodiment, it is a figure which illustrates distribution of the feature point extracted with respect to the image which image | photographed the chart for evaluation, and the calculated F value. In this Embodiment, it is a figure which illustrates distribution of the feature point extracted with respect to the image which image | photographed the chart for evaluation, and the calculated F value. It is a figure which illustrates the change of S / N ratio with respect to the change of cell size. In this Embodiment, it is a figure which shows the data structure example of the setting information of the target value stored in a target value memory | storage part. 10 is a flowchart illustrating a processing procedure for adjusting an adjustment target parameter after the information processing apparatus evaluates feature point extraction performance in the present embodiment. It is a figure which shows another example of the chart for evaluation in this Embodiment.

  The present embodiment relates to performance evaluation performed in advance and adjustment as necessary in order to perform information processing using feature points extracted from a captured image with high accuracy. This performance evaluation and adjustment may be performed by a manufacturer at the stage of developing and manufacturing the information processing system, or may be performed by a user who has purchased the product in an actual use environment. The contents of information processing such as a game performed using the feature points are not particularly limited. Hereinafter, the stage of performing some information processing using the extracted feature points is referred to as “operation time”, and the stage of performing performance evaluation and adjustment before that is referred to as “evaluation time”.

  FIG. 1 shows a configuration example of an evaluation system to which this embodiment can be applied. The evaluation system 8 includes an imaging device 12 that captures the evaluation chart 200, a feature point extracted from the captured image, an information processing device 10 that evaluates the result, and a display device 16 that outputs information such as the evaluation result as an image. including. Note that the system configuration shown in the figure is the one at the time of evaluation, and in operation, the imaging device 12 may shoot a subject space according to the application, and the information processing device 10 may perform information processing requested by the user. . The display device 16 may display the result of the information processing.

  The imaging device 12 includes a camera that captures an object and a mechanism that generates output data of a captured image by performing general processing such as demosaic processing on the output signal and sends the output data to the information processing device 10. The camera includes a visible light sensor used in general digital cameras and digital video cameras, such as a CCD (Charge Coupled Device) sensor and a CMOS (Complementary Metal Oxide Semiconductor) sensor. The imaging device 12 may include only one camera or a so-called stereo camera in which two cameras are arranged on the left and right sides at a known interval.

  The information processing apparatus 10 may be a game device or a personal computer, for example, and implements an information processing function by loading a program necessary for evaluation and operation. Alternatively, the information processing apparatus 10 may be an apparatus specialized for performing performance evaluation and adjustment. In this case, it may be replaced with a device according to the application such as a game device or a personal computer at the time of operation.

  The display device 16 may be a general display such as a liquid crystal display, a plasma display, or an organic EL display. At the time of evaluation, the display device 16 displays information such as the result of evaluation performed by the information processing device 10 and extracted feature points. When presentation of such information is not necessary, the display device 16 may be omitted at the time of evaluation.

  The connection of the imaging device 12, the information processing device 10, and the display device 16 may be wired or wireless. In the present embodiment, in particular, it is evaluated whether or not feature points are extracted with sufficient accuracy from an image captured by the imaging device 12. If necessary, conditions such as processing in the imaging device 12, data transmission from the imaging device 12 to the information processing device 10, and feature point extraction processing are adjusted. In order to determine appropriate conditions in this way and use them for operation, the imaging device 12 and the communication means follow the configuration assumed during operation.

  FIG. 2 shows an internal circuit configuration of the information processing apparatus 10. The information processing apparatus 10 includes a CPU (Central Processing Unit) 22, a GPU (Graphics Processing Unit) 24, and a main memory 26. The CPU 22 controls processing and signal transmission in the components inside the information processing apparatus 10 based on programs such as an operating system and applications. The GPU 24 performs image processing. The main memory 26 is constituted by a RAM (Random Access Memory), and stores programs and data necessary for processing.

  These units are connected to each other via a bus 30. An input / output interface 28 is further connected to the bus 30. The input / output interface 28 includes a peripheral device interface such as USB and IEEE1394, a communication unit 32 including a wired or wireless LAN network interface, a storage unit 34 such as a hard disk drive or a nonvolatile memory, an output of the display device 16 or a speaker. An output unit 36 that outputs data to the apparatus, an input unit 38 that inputs data from the imaging apparatus 12, and a recording medium driving unit 40 that drives a removable recording medium such as a magnetic disk, an optical disk, or a semiconductor memory are connected.

  The CPU 22 controls the entire information processing apparatus 10 by executing an operating system stored in the storage unit 34. The CPU 22 also executes various programs read from the removable recording medium and loaded into the main memory 26 or downloaded via the communication unit 32. The GPU 24 has a function of a geometry engine and a function of a rendering processor, performs drawing processing according to a drawing command from the CPU 22, and stores a display image in a frame buffer (not shown). The display image stored in the frame buffer is converted into a video signal and output to the output unit 36.

  At the time of evaluation, the information processing apparatus 10 extracts feature points from an image obtained by photographing an evaluation chart, and calculates a numerical value representing the performance as a performance index by a method described later. Then, the extraction result and the performance index are displayed on the display device 16, and the photographing condition and the processing condition are adjusted based on the result. However, it is not necessary to perform all of them, only the result is displayed, and the developer or user who sees it may change the condition manually, or only the processing condition is automatically adjusted without displaying the information. But you can. In any case, by repeating the adjustment until good performance is obtained, finally, a condition for obtaining an optimal image in terms of feature point extraction is obtained.

  The “condition” to be adjusted here may be any factor that may affect the extraction accuracy from imaging to feature point extraction processing, and may be one or a plurality of combinations. FIG. 3 illustrates factors assumed as adjustment targets. As shown in the figure, the type of image sensor that the imaging device 12 has, the shooting conditions, the algorithm used when generating image data from the output value of the sensor, the filter used, the method and setting value when compressing and encoding the image data, It is considered that various factors such as a wireless method for transmitting data to the information processing apparatus 10 affect the accuracy of extracting a captured image and, in turn, a feature point.

  Conventionally, elements as exemplified have been pursued in a direction that humans can see and recognize when displaying an image. However, the conditions thus determined are not necessarily advantageous for feature point extraction. For this reason, even if information processing is performed using feature points extracted from captured images obtained under such conditions, satisfactory processing accuracy cannot be obtained, or further improvement in accuracy cannot be easily realized. It is possible to do.

  Therefore, in the present embodiment, the cause of the influence on the processing accuracy is separated from the subsequent information processing by evaluating the image from the viewpoint of extraction performance until the feature point is extracted. In addition, by quantifying performance indicators and enabling easy comparison, it is possible to identify factors that have a large impact from the many factors shown in the figure, and to determine optimal conditions efficiently and automatically. To be able to. As shown in the figure, the factors to be adjusted include an individual entity such as an image sensor type and an algorithm, and numerical values such as an exposure time and a gamma value. In the following description, these are collectively referred to as “adjustment target parameters”.

  FIG. 4 shows functional block configurations of the information processing apparatus 10 and the imaging apparatus 12 at the time of evaluation. Each functional block shown in FIG. 4 can be realized in hardware by a CPU, GPU, main memory, or the like, and in software, it can be realized by a computer program or the like loaded from a hard disk or a recording medium into the main memory. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any one.

  In addition to the imaging unit 62 formed of an image sensor, the imaging device 12 includes a data forming unit 60 that forms data of a captured image to be transmitted to the information processing device 10, and parameters that store setting values of imaging conditions and parameters used for data formation A storage unit 64 is included. This parameter includes an adjustment target parameter. The imaging unit 62 includes an image sensor such as a CCD or a CMOS, and photographs an evaluation chart. At this time, imaging is performed under conditions according to setting values such as the exposure time, aperture value, and gain value stored in the parameter storage unit 64.

  The data forming unit 60 generates captured image data from the electrical signal for each pixel output from the imaging unit 62. Specifically, demosaic processing, defective pixel correction processing, gamma correction, and the like are performed. Further, the data forming unit 60 compresses and encodes the generated image data and transmits it to the information processing apparatus 10 using a predetermined communication protocol. The algorithms and filters used for these various processes also follow the settings stored in the parameter storage unit 64.

  The information processing apparatus 10 extracts a feature point from a photographed image by the same method as in operation, a chart image obtaining unit 70 that obtains image data obtained by photographing an evaluation chart, an image storage unit 71 that stores photographed image data, and the like. A feature point extraction unit 72, a correct coordinate acquisition unit 74 that acquires correct feature point coordinates by a method suitable for an evaluation chart, a performance index calculation unit 76 that calculates a performance index for feature point extraction, a preset target value and performance A comparison unit 78 that compares indexes, a target value storage unit 80 that stores the target value, an adjustment unit 82 that adjusts an adjustment target parameter based on the comparison result, and a parameter candidate storage unit 84 that stores adjustment target parameter setting candidates Including.

  The chart image acquisition unit 70 acquires data of an image obtained by photographing the chart from the imaging device 12, decodes and decompresses the data in accordance with the compression encoding format employed, and expands the data in the image storage unit 71. The feature point extraction unit 72 reads the image data from the image storage unit 71 and extracts the feature points using the same algorithm as in operation. Various algorithms such as Harris, Eigen, Fast (Features from Accelerated Segment Test), SIFT (Scale-Invariant Feature Transform), SURF (Speeded Up Robust Features) are used as techniques for extracting feature points for image recognition and other purposes. It has become.

  Any one of them may be adopted in the present embodiment, and it is appropriately selected according to the purpose of information processing. Alternatively, the algorithm itself can be set as a parameter to be adjusted. Since the individual algorithms are general, the description thereof is omitted here.

  The correct coordinate acquisition unit 74 reads the data of the photographed image of the chart from the image storage unit 71, and uses the predetermined algorithm suitable for the evaluation chart to position coordinates (hereinafter, “ Called correct coordinates). That is, the correct coordinates are acquired by a method specialized for the evaluation chart, which is different from the feature point extraction unit 72 extracting the feature points. Specific examples will be described later.

  The performance index calculation unit 76 collates the feature points extracted by the feature point extraction unit 72 and the correct coordinates acquired by the correct coordinate acquisition unit 74 from the same captured image of the evaluation chart, thereby extracting the feature points. Is expressed as a numerical value. The performance index calculation unit 76 further generates an image in which the feature point extracted by the feature point extraction unit 72 and the correct coordinate acquired by the correct coordinate acquisition unit 74 are represented by a graphic or the like on the captured image, and is displayed on the display device. 16 may be displayed. As a result, the developer or user can sensuously understand how correctly feature points are extracted.

  The comparison unit 78 reads a preset target value of the performance index from the target value storage unit 80 and compares it with the actual performance index calculated by the performance index calculation unit 76. As the target value of the performance index, a value that ensures that sufficient processing accuracy is obtained as a result of information processing using the extracted feature points is obtained by experiments or the like. The target value may have a plurality of settings depending on the contents of information processing, the subject to be detected, and the like. In this case, the optimum parameter to be adjusted is obtained for each of those elements.

  The adjustment unit 82 determines whether or not adjustment is necessary depending on whether or not the performance index has reached a target value. The determination result may be displayed on the display device 16. When determining that adjustment is necessary, the adjustment unit 82 determines a new setting of the adjustment target parameter. As described above, the parameters to be adjusted include those whose numerical values can be continuously changed such as the exposure time, aperture value, and compression rate, and those that are selected from individual entities such as algorithms and filters.

  The parameter candidate storage unit 84 has an adjustable numerical value range for the adjustment target parameter whose value can be changed and a rule relating to how to change the parameter, and a selectable candidate for the adjustment target parameter selected from the candidates and the selection thereof. Store the rules. The adjustment unit 82 determines a new setting according to the set rule, and updates the setting stored in the parameter storage unit 64 of the imaging device 12. In the adjustment, the difference between the actual performance index and the target value may be reflected. That is, the adjustment range of the adjustment target parameter is increased as the difference is increased.

  One parameter may be adjusted by one evaluation, or a plurality of combinations may be used. The target may be specified in advance by a developer or user. Further, parameters that are difficult to automatically adjust, such as the type of image sensor, can be used as a material for investigation by a developer or the like by displaying the determination result on the display device 16. When the imaging device 12 repeats a loop for performing performance evaluation and adjustment again using the captured image of the evaluation chart newly captured and transmitted with the updated setting, the optimum condition can be efficiently identified. become.

  FIG. 5 illustrates an evaluation chart. The evaluation chart 200 may be one in which a chart pattern is drawn on the surface of an object such as a board or a mat, or one created as electronic data may be displayed on the display device 16. By photographing such an evaluation chart 200 from the front at a predetermined distance, a pattern as illustrated is obtained as a photographed image. The evaluation chart 200 shown in the figure includes a plurality of chessboard-like large rectangles (for example, a large rectangle 202) in which black and white are inverted next to each other, and a gray small rectangle (for example, a small rectangle 204) represented inside each large rectangle. Consists of.

  In the evaluation chart 200, it is desirable that the gray density of the small rectangle has gradation. In this example, the feature point extraction performance is evaluated from the viewpoint of whether or not the four corners of the small rectangle can be extracted accurately and without waste. That is, the extraction performance is evaluated by representing the subject image during operation in a pseudo small rectangle. Therefore, as the gray gradation represented by the small rectangle increases, the diversity of the image in the actual photographed image is represented, and the correct evaluation can be performed.

  FIG. 6 is a diagram for explaining a method in which the correct coordinate acquisition unit 74 acquires correct coordinates from the captured image of the evaluation chart shown in FIG. 2A and 2B show a part of the image of the evaluation chart 200. FIG. First, the correct coordinate acquisition unit 74 extracts a corner of a large rectangle as a feature point as indicated by a white cross in (a) of FIG. A technique for extracting rectangular corners as feature points from a captured image of a chessboard pattern chart is widely known in camera calibration (see, for example, Japanese Patent Application Laid-Open No. 2014-92460), and an extraction algorithm is also common. It can be carried out with high accuracy using a thing.

  For example, the cornerSubPix function (http://docs.opencv.org/2.4/modules/imgproc/doc/feature_detection.html) in openCV, which is an open source widely used in the field of computer vision, can be used. The correct coordinate acquisition unit 74 acquires the position coordinates of the corner of the large rectangle in the photographed image using the method in which the accuracy is guaranteed in this way. Then, as shown by white circles in (b) of the figure, corners of small rectangles are extracted based on their relative positions to the corners of large rectangles.

  Specifically, as shown on the right side of the figure, based on the position coordinates (x0, y0) of one corner of the large rectangular image and the side lengths Δx, Δy in the horizontal and vertical directions, the corresponding small The position coordinates (x, y) of the corner of the rectangular image are obtained. In the original evaluation chart, assuming that a small rectangle having a size reduced to n% (n <100) is arranged at the center of the large rectangle, the position coordinates (x, y) of the corner of the small rectangle are It is expressed as follows.

  Here, the positive and negative signs on the right side are switched according to the positional relationship between the corresponding corners of the large rectangle and the small rectangle. In order to obtain the correct coordinates more accurately, it is desirable to perform a correction process for removing distortion caused by a lens or the like on the image obtained by photographing the evaluation chart. For such correction processing, a general method using a chessboard pattern chart, for example, the function provided in the above-described openCV can be applied to the evaluation chart of this embodiment as it is.

  As described above, by including the pattern used in the conventional camera calibration in a part of the chart for evaluation, calibration processing other than the feature point extraction performance evaluation can be achieved at the same time. Also, by making at least one of the small rectangles the same color as the gray card that adjusts exposure and white balance, these adjustments can be made simultaneously.

  On the other hand, as described above, the feature point extraction unit 72 extracts small rectangular corners from the same captured image by the same algorithm as in operation. That is, the correct coordinate acquisition unit 74 uses an algorithm specialized for a chessboard pattern, whereas the feature point extraction unit 72 uses an algorithm that can extract a feature point even for an arbitrary subject.

  Since the evaluation chart illustrated here assumes an algorithm that can extract corners, a small rectangle is represented inside a large rectangle. However, depending on the feature point extraction algorithm used during operation, the shape of the figure represented inside the large rectangle may be other than a rectangle. If the positional relationship with the corner of the large rectangle is known, the correct coordinates can be acquired regardless of the figure, so that the extraction performance can be evaluated in the same manner as the rectangle.

  FIG. 7 compares the small rectangular corner from which the correct coordinate acquisition unit 74 has acquired the coordinates with the feature point extracted by the feature point extraction unit 72. Among the captured images of the evaluation charts shown in (a) and (b), (a) shows the corner positions acquired by the correct coordinate acquisition unit 74, and (b) shows the feature points extracted by the feature point extraction unit 72. , Each is enclosed in a rectangle.

  As shown in (a), the correct coordinate acquisition unit 74 can accurately extract small rectangular corners by the above-described method. On the other hand, the feature point extraction unit 72 cannot extract a small rectangular corner or misextracts a part other than the corner. Note that in (b), the rectangle surrounding the mis-extracted feature point is represented by a darker line.

  If the gray density of the small rectangle is changed so that the contrast between the large rectangle and the small rectangle gradually increases in one direction of the evaluation chart, the relationship between the contrast and the frequency of erroneous extraction can be easily grasped. The performance index calculation unit 76 displays the feature points extracted by the feature point extraction unit 72 as shown in (b), so that the user can visually check the tendency of erroneous extraction and extraction omission and examine the countermeasure. .

  The performance index calculation unit 76 makes it easy to compare the results under a plurality of conditions and automatically adjust the adjustment target parameter by digitizing the extraction performance from the viewpoint of such erroneous extraction and extraction omission. Like that. Specifically, an F value (F-measure), which is a measure used to express information retrieval performance, is calculated as follows.

  That is, the F value is a harmonic average of Precision and Recall. Here, Precision is the ratio of the number of correct answers R out of the number N of feature points extracted by the feature point extraction unit 72, and Recall is the number of correct answers that should have been extracted, that is, correct answers. This is the ratio of the number of correct coordinates acquired by the coordinate acquisition unit 74 to the number R extracted as feature points by the feature point extraction unit 72, and is given as follows.

  By using the F value, it is possible to represent comprehensive extraction performance including the viewpoint of “accuracy” in the sense that there are few errors in the extracted feature points, and the point of few extraction omissions. 8 to 11 illustrate the distribution of feature points extracted from images obtained by photographing the evaluation chart under various conditions, and the calculated F value. In each figure, the results of photographing with cameras having four different image sensors are compared. A, B, and C are large, medium, and small RGB (red, green, and blue) sensors, and D is RGB and IR. It has an (infrared) sensor.

  However, in any sensor, feature points are extracted from a luminance image finally generated from RGB signals. FIG. 8 shows the result of shooting in a relatively bright place of 700 lux and normal gain adjustment. In photographing under such favorable conditions, there is little influence due to the cell size or the presence or absence of an infrared sensor, and a good F value can be obtained stably.

  FIG. 9 shows the result when the image is taken at the same brightness as in FIG. 8 and the gain is not adjusted. Compared with the case where the gain is adjusted, the F value is deteriorated in any sensor, but the degree of deterioration is not uniform, and the deterioration is noticeable in the sensor C having a small cell size and the sensor D also detecting infrared rays.

  FIG. 10 shows the result when shooting is performed at a location of 100 lux darker than FIGS. 8 and 9 and normal gain adjustment is performed. In this case, the F value is further deteriorated by the C sensor having a small cell size. Further, it can be seen that the sensors A and B having a large cell size have robustness against such a change in illuminance.

  FIG. 11 shows a result of shooting at a location of 10 lux that is darker than that in FIG. 10 and performing normal gain adjustment. However, an image photographed by the D sensor that also detects infrared rays is not shown because normal feature point extraction was not performed due to noise. In this case, even with a B sensor having a medium cell size, the F value is significantly deteriorated.

  That is, when such a dark place is assumed, it can be understood that a sensor having a cell size equal to or larger than a predetermined value needs to be used in order to withstand information processing using feature points. Note that the feature points extracted below the images of the B and C sensors are not shown because when the number of false detections accumulated from above exceeds a predetermined value, the subsequent feature points are excluded from the display target. This is due to the software rule of

  FIG. 12 illustrates the change in the S / N ratio with respect to the change in cell size. As shown in the figure, the S / N ratio has a characteristic of increasing linearly as the cell size increases. On the other hand, in the results of FIGS. 8 to 11, the change of the F value with respect to the cell size is not linear, and the relationship greatly changes depending on the illuminance of the photographing environment. For example, even if the cell size is the same, the F value suddenly deteriorates with a certain illuminance as a boundary, and the illuminance at the boundary varies depending on the cell size.

  Therefore, the deterioration of the performance of the feature point extraction process does not depend only on noise, and various factors are considered to be related in a complicated manner. According to the method described in the present embodiment, such performance is digitized to facilitate comparison under various conditions. This allows you to isolate a number of factors and evaluate performance, so that you can determine the optimal combination of parameters and check whether the feature point information can maintain enough accuracy to withstand use during operation. It can be done efficiently and easily.

  FIG. 13 shows an example of the data structure of the target value setting information stored in the target value storage unit 80. The target value setting table 260 includes an application field 262 and a target value field 264. The application column 262 sets application identification information as a unit that is selected by a user during information processing such as an individual game and performs information processing. In the target value column 264, a target value of the feature point extraction performance index is set in correspondence with each application.

  Depending on the application, high extraction performance may be required, but processing may not fail without pursuing feature point extraction performance so much. In addition, there are various detection targets and assumed movements, which makes it easy to extract feature points. Therefore, by setting the target value for each application, it is possible to obtain or switch the parameter to be adjusted according to the situation, and the processing cost is not increased unnecessarily.

  Next, the operation of the information processing apparatus 10 that can be realized by the above configuration will be described. FIG. 14 is a flowchart illustrating a processing procedure in which the information processing apparatus 10 adjusts the adjustment target parameter after evaluating the feature point extraction performance. This flowchart is started when the imaging device 12 captures an evaluation chart. First, the chart image acquisition unit 70 of the information processing apparatus 10 acquires captured image data of the evaluation table chart (S10).

  Various processes performed from the imaging by the imaging device 12 until the information processing device 10 acquires data are the same as in operation. That is, the adjustment target parameter is also given an initial value assumed at the time of operation. The chart image acquisition unit 70 stores the acquired data in the image storage unit 71 by appropriately decoding and expanding the acquired data. Next, the feature point extraction unit 72 reads the data of the image and extracts the feature points using an algorithm assumed at the time of operation (S12).

  At this time, if the number of extracted feature points does not exceed the predetermined threshold Th (N in S14), it is determined that the adjustment target parameter needs to be adjusted without performing the performance evaluation, and the adjustment unit 82 The parameter is adjusted (S22). Specifically, the adjustment unit 82 reads the next setting candidate from the parameter candidate storage unit 84 and updates the setting in the parameter storage unit 64 of the imaging device 12. Here, the threshold value Th is set to a low value so that it is clear that the feature point extraction is not appropriate without calculating the performance index.

  If the number of extracted feature points exceeds the predetermined threshold Th (Y in S14), the correct coordinate acquisition unit 74 acquires the correct coordinate (S16), and the performance index calculation unit 76 displays an index for performance evaluation. Calculate (S18). Specifically, the F value is calculated according to the number of correct coordinates, the number of correct answers among the feature points extracted by the feature point extraction unit 72, and the number of incorrect answers. At this time, the performance index calculation unit 76 may cause the display device 16 to display an image in which the extracted feature points are displayed on the captured image in different colors and shapes for the correct answer and the incorrect answer.

  If the comparison unit 78 compares the calculated performance index with the target value read from the target value storage unit 80 and the performance index does not reach the target value (Y in S20), the adjustment unit 82 sets the adjustment target parameter. Adjust (S22). Specifically, the next setting candidate is read from the parameter candidate storage unit 84 and the setting in the parameter storage unit 64 of the imaging device 12 is updated. Once the parameters are adjusted, the data of the new photographed image that has been photographed and transmitted reflecting the parameter is acquired again (S10), and the processes of S12 to S20 are performed.

  This process is repeated, and when the feature point extraction performance index reaches the target value in S20, the process ends (N in S20). As a result, processing conditions are set in the parameter storage unit 64 of the imaging device 12 so as to obtain sufficient feature point extraction performance.

  FIG. 15 shows another example of the evaluation chart. In the evaluation chart 300, black circles of a predetermined size (for example, a circle 302) are arranged after forming one vacant space in the upper, lower, left, and right sides, and gray triangles (for example, a triangle 304) of a predetermined size are formed inside and vacant portions thereof. It has the pattern showing. In this case, first, a black circle image is detected from the photographed image by fitting with a circular model. As shown on the right side of the figure, the center coordinates (x0, y0) and (x1, y1) of each circle and the coordinates (x1, y0), (x0, y1) is obtained, and three-related vertex coordinates of a predetermined size with these coordinates as the center of gravity are obtained as correct coordinates.

  Separately, by extracting the vertices of the triangle as feature points using an algorithm at the time of operation, the performance index can be obtained in the same manner as in the case of using the evaluation chart of FIG. In this way, the pattern of the chart for evaluation is represented as having a predetermined monotone figure and a predetermined positional relationship with the figure, and there is an algorithm for accurately obtaining the coordinates of a certain location. The shape and arrangement are not limited as long as it is a combination with a gray figure having a feature point that represents a pseudo image of the target object.

  For example, the small rectangle in the evaluation chart 200 of FIG. 5 may be a triangle, or the triangle in the evaluation chart 300 of FIG. 15 may be a rectangle. In addition, a grid-like straight line or the like may be used instead of the large rectangle or the circle in the both evaluation charts. Instead of small rectangles and triangles, straight lines, stars, polygons, etc. may be used.

  As described above, according to the present embodiment described above, it is evaluated whether an image at the stage of extracting feature points is appropriately obtained from the viewpoint of extraction processing, using an image obtained by photographing an evaluation chart. In general, the processing related to the image from imaging to data development in the information processing apparatus is optimized so that a good image can be obtained with human eyes. On the other hand, such a condition is not always preferable from the viewpoint of feature point extraction. Therefore, from the viewpoint of feature point extraction, it is possible to extract feature points with higher accuracy by giving room for adjustment to those processes and devices to be used, and as a result, it is possible to improve the accuracy of information processing using the feature points.

  The suitability of the system configuration for feature point extraction is quantified as an extraction performance index. At this time, using a method different from the feature point extraction algorithm used at the time of operation, the coordinates of the feature points to be originally extracted are obtained, and a scale that takes into account both erroneous extraction and omission of extraction is obtained. The evaluation chart has a pattern corresponding to an algorithm that can accurately obtain the coordinates of feature points that should be extracted. For example, a small gray rectangle is represented inside each rectangle of a chessboard-like pattern. First, a corner of a chessboard pattern is detected with high accuracy using a general algorithm, and corner coordinates of a small rectangle are obtained based on the relative position with the corner. By comparing this with the feature points extracted by the algorithm at the time of operation, the above scale can be obtained.

  There are many factors that can affect the feature point extraction process, and even if the conditions are diversified, it is possible to easily compare by calculating the extraction performance index as a numerical value. The device can also recognize it and determine whether adjustment is necessary or make adjustments to improve performance. As a result, it is possible to efficiently acquire processing conditions for improving the accuracy of information processing performed using feature points extracted from the captured image.

  The present invention has been described based on the embodiments. Those skilled in the art will understand that the above-described embodiment is an exemplification, and that various modifications can be made to combinations of the respective constituent elements and processing processes, and such modifications are also within the scope of the present invention. is there.

  For example, in the present embodiment, the processing performed from the imaging by the imaging device to the development of the image data by the information processing device is adjusted so that the feature points can be obtained appropriately, but the application of the present invention is not limited thereto. For example, when shooting with a camera having an RGB / IR sensor, noise may occur in the RGB image as a result of the IR signal leaking into the RGB signal. According to the present invention, since the feature points extracted from the image obtained by photographing the evaluation chart can be clearly identified, the degree of leakage of the IR signal can be evaluated based on the generation amount of the erroneous extraction.

  In this case, by calculating an index using an image at a stage where processing for the RGB signals from the image sensor is as small as possible, it is possible to distinguish from erroneous extraction caused by subsequent processing. Further, if an index is calculated using an image based on an IR signal, the amount of noise generated in the IR sensor can be evaluated.

  DESCRIPTION OF SYMBOLS 8 Evaluation system, 10 Information processing apparatus, 12 Imaging apparatus, 16 Display apparatus, 22 CPU, 24 GPU, 26 Main memory, 60 Data formation part, 62 Imaging part, 64 Parameter storage part, 70 Chart image acquisition part, 71 Image storage Unit, 72 feature point extraction unit, 74 correct coordinate acquisition unit, 76 performance index calculation unit, 78 comparison unit, 80 target value storage unit, 82 adjustment unit, 84 parameter candidate storage unit, 200 evaluation chart, 300 evaluation chart.

Claims (14)

A chart image acquisition unit that acquires data of a photographed image of an evaluation chart representing a predetermined pattern from an imaging device and develops the data as an image in a memory;
A feature point extraction unit for extracting feature points from the captured image;
A correct coordinate acquisition unit that acquires, as a correct coordinate, a position coordinate of a feature point to be extracted from the captured image by a method different from the feature point extraction unit;
Based on the feature points extracted by the feature point extraction unit and the correct coordinates, a performance index calculation unit that calculates and outputs a performance index that quantifies the performance of obtaining an image suitable for feature point extraction;
An information processing apparatus comprising: The pattern of the evaluation chart includes a monotone figure to be extracted by the extraction algorithm used by the correct coordinate acquisition unit, and a gray figure having a predetermined positional relationship therewith,
The information processing apparatus according to claim 1, wherein the correct coordinate acquisition unit acquires the coordinates of feature points of the gray graphic as the correct coordinates using the positional relationship.   3. The information processing according to claim 1, wherein the pattern of the evaluation chart includes a chessboard-like large black and white rectangle and a gray figure represented in each large rectangle. apparatus.   The pattern of the evaluation chart includes a plurality of black circles arranged so as to form one empty space in the upper, lower, left, and right directions, and a gray figure that is represented in each of the circles and in the empty space. The information processing apparatus according to claim 1 or 2.   5. The information processing apparatus according to claim 2, wherein a plurality of gray figures in the pattern of the evaluation chart have a plurality of gray densities.   The image processing apparatus further includes an adjustment unit that adjusts any setting used for processing related to the captured image that is executed from imaging to feature point extraction when the performance index does not reach a preset target value. The information processing apparatus according to any one of claims 1 to 5.   The information processing apparatus according to claim 6, wherein the target value varies depending on an application that defines information processing using the extracted feature points.   8. The performance index calculation unit according to claim 1, wherein the performance index calculation unit calculates the performance index based on the number of points erroneously extracted by the feature point extraction unit and the number of feature points that are not extracted. The information processing apparatus described in 1.   The performance index calculation unit displays on the display device an image represented on the captured image by distinguishing the feature points extracted by the feature point extraction unit from those corresponding to the correct coordinates and the others. The information processing apparatus according to any one of claims 1 to 8. An evaluation chart for evaluating the performance of an information processing device to obtain an image suitable for feature point extraction,
A chessboard-like black and white rectangle for detecting a corner from the captured image, and coordinates are obtained based on the detection result of the corner, and feature points for comparison with the result of a feature point extraction process performed separately An evaluation chart comprising: a gray figure represented inside each rectangle. An evaluation chart for evaluating the performance of an information processing device to obtain an image suitable for feature point extraction,
A plurality of black circles arranged so as to form one empty space in the up, down, left, and right detected from the captured image, and coordinates are acquired based on the detection results of the circles, and feature point extraction processing is performed separately An evaluation chart comprising: a gray graphic represented in the interior of each circle and in the vacant space having characteristic points for comparison with results. An imaging device that captures an evaluation chart representing a predetermined pattern, and an information processing device that acquires data of the captured image and evaluates the ability to obtain an image suitable for feature point extraction,
The information processing apparatus includes:
A chart image acquisition unit that acquires data of the photographed image from the imaging device and develops the data as an image in a memory;
A feature point extraction unit for extracting feature points from the captured image;
A correct coordinate acquisition unit that acquires, as a correct coordinate, a position coordinate of a feature point to be extracted from the captured image by a method different from the feature point extraction unit;
Based on the feature points extracted by the feature point extraction unit and the correct coordinates, a performance index calculation unit that calculates a performance index that quantifies the performance;
An adjustment unit that adjusts any of the settings used for processing in the imaging device when the performance index does not reach a preset target value;
The imaging device
A parameter storage unit for storing settings adjusted by the adjustment unit;
A data forming unit that generates data of a captured image according to the settings stored in the parameter storage unit and transmits the data to the information processing device;
An evaluation system comprising: Acquiring data of a photographed image of an evaluation chart representing a predetermined pattern from an imaging device and developing the data in a memory as an image;
Extracting feature points from the captured image;
Obtaining the position coordinates of the feature points to be extracted from the captured image as correct coordinates in a different method from the step of extracting the feature points;
Calculating and outputting a performance index quantifying the performance of obtaining an image suitable for feature point extraction based on the feature point extracted in the step of extracting the feature point and the correct coordinates;
The performance evaluation method by information processing apparatus characterized by including these. A function of acquiring data of a photographed image of an evaluation chart in which a predetermined pattern is represented from an imaging device and developing the data as an image in a memory;
A function of extracting feature points from the captured image;
A function for acquiring position coordinates of feature points to be extracted from the captured image as correct coordinates in a method different from the function of extracting the feature points;
A function that calculates and outputs a performance index that quantifies the ability to obtain an image suitable for feature point extraction based on the feature point extracted by the function of extracting the feature point and the correct coordinates;
A computer program for causing a computer to realize the above.