JP7387261B2 - Information processing device, information processing method and program - Google Patents

Description

The present invention relates to an information processing device, an information processing method, and a program.

When inspecting concrete walls of structures such as bridges, dams, and tunnels, survey engineers approach the concrete walls and visually check for cracks and other deformations. Since such inspection work called close visual inspection is expensive, in recent years image inspections have been carried out to check for deformations using images taken of concrete walls.
Here, in order to capture an image in which difficult-to-see cracks such as thin cracks can be confirmed, it is necessary to appropriately set shooting parameters. The user must appropriately adjust photographing parameters such as focus and exposure according to the weather and photographic equipment at the time of photographing, and photograph an image in which cracks can be confirmed. However, it is difficult to adjust the imaging parameters to take images to confirm unclear deformities such as thin cracks, and depending on subtle differences in the imaging parameters, deformations may or may not be confirmed.
An example of a conventional method for adjusting imaging parameters is the method disclosed in Patent Document 1. In Patent Document 1, first, a plurality of images are photographed using a plurality of different photographing parameters, and the plurality of photographed images are displayed on a display. The user selects the image that he or she considers most desirable from among the plurality of images. As a result, the photographing parameters for photographing the selected image are set.

Patent No. 4534816

Chopra, Sumit, Raia Hadsell, and Yann LeCun. "Learning a similarity metric discriminatively, with application to face verification." Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. Vol. 1. IEEE, 2005. Xie, Junyuan, Linli Xu, and Enhong Chen. "Image denoising and inpainting with deep neural networks." Advances in Neural Information Processing Systems. 2012. Dong, Chao, et al. "Image super-resolution using deep convolutional networks." IEEE transactions on pattern analysis and machine intelligence 38.2 (2016): 295-307. Goodfellow, Ian, et al. "Generative adversarial nets." Advances in neural information processing systems. 2014. Gatys, Leon A., Alexander S. Ecker, and Matthias Bethge. "Image style transfer using convolutional neural networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.

However, since photographing for inspecting a structure requires delicate adjustment of photographing parameters, applying the method of Patent Document 1 results in displaying a plurality of images with small changes between images. It is difficult for the user to compare images with such small changes and select the optimal image. Furthermore, since the inspection images are taken outdoors, it is difficult to judge subtle differences in the images due to the influence of external light and the available display size.

The present invention includes a first acquisition means that acquires a photographed image of a photographed object with predetermined photographing parameters, and a second acquisition means that acquires a plurality of reference images based on a search condition regarding image information of the photographed image. Estimating a first evaluation value for each of the reference images based on an acquisition means, the photographed image acquired by the first acquisition means , and the reference image acquired by the second acquisition means. and an estimating means for estimating a photographing method suitable for photographing the photographing target based on the plurality of estimated first evaluation values, the first evaluation value, and the first evaluation value. Estimation is performed using a model based on the learning result of the relationship between the imaging means and the imaging parameter corresponding to the evaluation value larger than the evaluation value of 1 .

According to the present invention, it becomes possible to easily set photographing parameters for photographing a desired image even if the details of the photographed image cannot be confirmed.

FIG. 1 is a diagram illustrating an example of a hardware configuration of an information processing device. FIG. 1 is a diagram illustrating an example of the configuration of an information processing device. FIG. 3 is a diagram illustrating information stored in an image storage unit. 3 is a flowchart illustrating an example of information processing. FIG. 3 is a diagram showing an example of a screen during image search. FIG. 3 is a diagram illustrating setting of photographing parameters. It is a figure explaining the calculation method of the evaluation value based on the partial image of a crack position. FIG. 3 is a diagram illustrating evaluation of each imaging parameter. FIG. 3 is a diagram illustrating an operation unit when adjusting shooting parameters. FIG. 7 is a diagram illustrating an example of the configuration of an information processing apparatus according to a fourth embodiment. FIG. 7 is a diagram illustrating information stored in an image storage unit according to the fifth embodiment.

Embodiments of the present invention will be described below based on the drawings.

<Embodiment 1>
Embodiment 1 will be described by taking as an example the adjustment of photographing parameters in photographing for image inspection of infrastructure structures. Examples of infrastructure structures include bridges, dams, tunnels, etc., and in image inspection, images of concrete walls of these structures are photographed to create images for inspection. Therefore, in this embodiment, these concrete wall surfaces are to be photographed. The image inspection target may be an image of another structure or a surface of a material other than concrete. For example, if the inspection target is a road, the asphalt surface may be photographed.
In this embodiment, a reference image, which is an image of ideal image quality, is prepared, and the imaging method is adjusted so that the image of the object to be photographed resembles the reference image. The reference image is an image from among concrete wall images taken in the past, in which inspection targets that are difficult to photograph, such as thin cracks, can be clearly confirmed. In other words, the reference image is an image captured with focus, brightness, hue, etc. of quality preferable as an inspection image. Further, the main photographing methods adjusted in this embodiment are photographing parameters of the photographing unit, such as exposure, focus, white balance (color temperature), shutter speed, aperture, and the like. A method of adjusting the photographing method using the reference image will be described below.

FIG. 1 is a diagram illustrating an example of the hardware configuration of the information processing apparatus 100. The information processing device 100 may be integrated with a photographing section 101 shown in FIG. The camera may be configured in a different housing (for example, a computer or a tablet) than the camera including the camera. In the example of FIG. 1, the information processing device 100 includes a CPU 10, a storage section 11, an operation section 12, and a communication section 13 as a hardware configuration. The CPU 10 controls the entire information processing device 100. When the CPU 10 executes processing based on the program stored in the storage unit 11, the configurations shown in FIG. 2, 102, 104, and 105 in FIG. 10, which will be described later, and the processing in the flowchart in FIG. . The storage unit 11 stores programs, data, images, etc. used when the CPU 10 executes processing based on the programs. The operation unit 12 displays the results of processing by the CPU 10 and inputs user operations to the CPU 10. The operation unit 12 can be configured with a display and a touch panel on the back of the camera, or a display and an interface of a notebook PC. The communication unit 13 connects the information processing device 100 to a network and controls communication with other devices and the like.

FIG. 2 is a diagram illustrating an example of the configuration of the information processing apparatus 100 according to the first embodiment. The information processing apparatus 100 includes a photographing section 101, a reference image processing section 102, an image storage section 103, an estimating section 104, and a photographing parameter setting section 105. However, as described above, the imaging unit may or may not be included in the information processing device 100. The reference image processing unit 102, the estimation unit 104, and the imaging parameter setting unit 105 are software. Furthermore, the image storage unit 103 may be provided in the storage unit 11 or may be provided in a storage server that can communicate with the information processing apparatus 100. When the image storage unit 103 is provided in a storage server, the reference image processing unit 102 acquires images stored in the image storage unit 103 and information related to the images via the network. The image storage unit 103 is a storage that stores a group of images that are candidates for reference images.

FIG. 3 is a diagram illustrating information stored in the image storage unit 103. First, the image storage unit 103 stores a plurality of images (images 201 and 202 in FIG. 3). Hereinafter, the images 201 and 202 stored in the image storage unit 103 will be referred to as stored images. The stored image is an image prepared by collecting images taken with image quality suitable for image inspection from images taken of concrete walls of various structures. The preferred image quality for image inspection is one in which deformations such as cracks can be easily confirmed by a human being when checking the image, and for example, an image with preferable focus, brightness, color tone, etc. For example, the stored image 201 is an image in which the crack 211 can be clearly seen. Furthermore, the stored image is not limited to showing cracks. The stored image 202 is an image that shows joints 212 on the concrete wall surface. Since this stored image 202 clearly shows the edges of the joints 212, it has been determined that the image quality is suitable for inspection.

Furthermore, when a captured image is inspected using a technique for automatically detecting cracks, the image quality at which the automatic detection operates preferably may be set as the image quality preferable for image inspection. In this case, the accuracy rate of the detection result of the automatic detection is calculated, and an image with a higher quality is set as a stored image.
In the image storage unit 103 of FIG. 3, image information and photographing parameters are recorded in association with stored images. The image information is information about the content of the stored image, and includes information such as the type of structure of the object, the type of concrete, the weather at the time of shooting, the target in the image, the location/area where the structure is installed, the number of years that have passed, etc. It will be done. Further, the photographing parameters are photographing parameters when each reference image is photographed.

FIG. 4 is a flowchart showing an example of information processing. Hereinafter, the operation of the information processing device 100 will be explained according to the flowchart.
S301 and S302 are processes executed by the reference image processing unit 102. The reference image processing unit 102 of the first embodiment executes a process of selecting a reference image from stored images stored in the image storage unit 103. FIG. 5 shows information displayed on the operation unit 12 when S301 and S302 are executed.
In S301, the reference image processing unit 102 searches the image storage unit 103 for reference image candidates based on the search conditions. As a search method for reference image candidates, there is a method using image information. As shown in FIG. 3, images stored in the image storage unit 103 are associated with image information. Based on this information, the reference image processing unit 102 can search for a stored image similar to the subject to be photographed. FIG. 5 shows an example of a screen for searching stored images using the operation unit 12. For example, assume that the object to be photographed is the deck slab of a bridge, and the weather at the time of photographing was cloudy. The user sets conditions related to the subject to be photographed or the photographing situation as image search conditions. By selecting the search button 410, it is possible to search the image storage unit 103 for stored images that meet the search conditions. The search results are displayed in the reference image candidate display field 420 as reference image candidates. As reference image candidates, only stored images whose search conditions and image information match may be used as reference image candidates, or a predetermined number of stored images with a high degree of item matching may be selected as reference image candidates. Further, in the example of FIG. 5, the image information for search displays only the structure type, concrete type, and weather, but the conditions for image search are not limited to these. Furthermore, although FIG. 5 shows a method of setting the search content using a pull-down menu as a search method, the method by which the user inputs image information for the search is not limited to this. For example, an operation method may be used in which stored images can be searched by inputting a free character string as a keyword.

Furthermore, as another method of searching for reference image candidates, there is also a method of using temporarily captured images. In this case, the user first photographs an object to be photographed using the photographing unit 101. This photographing is a temporary photographing, and the photographing parameters at this time use automatic settings or the like. The image photographed in this temporary photographing is set as a temporary photographed image. The user sets the temporarily captured image as a search key for selecting reference image candidates. FIG. 5 shows that a temporary photographed image 450 is set as a search condition for selecting a reference image candidate. In this state, by selecting the search button 410, an image similar to the temporarily captured image is searched from the image storage unit 103. As a result of the search, the top stored images with a high degree of similarity are selected as reference image candidates and displayed in the reference image candidate display field 420. Here, in the search using the temporarily photographed image, for example, the degree of similarity with the stored image is calculated based on the characteristics of the entire image (the color or texture of the concrete wall surface), and a reference image candidate is selected. This makes it possible to search for stored images that have a similar concrete wall surface to be photographed for inspection. Further, the reference image candidates may be searched by simultaneously using the above-described search using the image information (keyword) and the search using the temporarily captured image.
As described above, a reference image candidate is selected and displayed in the reference image candidate display field 420.

In S302 of FIG. 4, the reference image processing unit 102 selects one image as a reference image from among the reference image candidates displayed in the reference image candidate display field 420. First, as an initial value for reference image selection, the reference image candidate with the highest degree of match in the search is automatically selected as the reference image. FIG. 5 shows how the reference image 430 selected in this manner is displayed in the reference image display field. By checking the reference image displayed in this way, the user can check the standard for adjusting the photographing method. If the user determines that the selected reference image 430 is inappropriate as a reference for adjustment, the user can select another image from the reference image candidates as the reference image. When another image is selected from the reference image candidates, the reference image processing unit 102 sets the selected image as the reference image.
Note that the image in FIG. 5 shows cracks (for example, 440 and 441). As will be described later, when calculating an evaluation value using an image of a cracked portion, the crack must be included in the reference image. Further, by setting an image including the crack 440 in the temporary photographed image 450, it may be possible to search for a stored image including a crack similar to the crack to be photographed when searching for a reference image candidate.

In S303, the imaging parameter setting unit 105 determines initial values of imaging parameters (hereinafter referred to as initial imaging parameters). The initial photographing parameters may be set by setting the photographing parameters determined by the normal photographing parameter adjustment method (auto parameter adjustment) of the photographing device as the initial parameters. Alternatively, the imaging parameters associated with the reference image may be used as the initial parameters. As shown in FIG. 3, the image storage unit 103 records the shooting parameters for each stored image when the image was shot. Therefore, when setting the imaging parameters associated with the reference image as initial parameters, the reference image processing unit 102 reads the imaging parameters associated with the image selected as the reference image from the image storage unit 103, and sets the initial parameters. Set as .

In S304, the imaging parameter setting unit 105 sets a plurality of imaging parameters based on the initial imaging parameters. FIG. 6 shows how a plurality of imaging parameters are set based on the initial imaging parameters. First, FIG. 6A is a diagram illustrating an embodiment in which exposure (EV) is adjusted as an example of photographing parameters to be adjusted by the method of this embodiment. In FIG. 6A, a white triangle 501 indicates that EV0 is set as an initial parameter. The photographing parameter setting unit 105 sets a plurality of photographing parameters centering on these initial parameters. In FIG. 6(A), the shooting parameter setting unit 105 changes the exposure by one step centering on EV0, and sets a plurality of EV-1 (black triangle 502 in FIG. 6) and EV+1 (black triangle 503 in FIG. 6). It is set as a parameter. Although this example shows how three imaging parameters are set together with the initial imaging parameters, the number of imaging parameters to be set is not limited to this. For example, the shooting parameter setting unit 105 may further set an exposure that is two steps different to set a total of five shooting parameters. Further, in this example, a plurality of photographing parameters are set according to the rule of changing the exposure by one step, but the increments of changing the photographing parameters may be set using other setting methods. For example, the shooting parameter setting unit 105 may set the exposure in 1/2-stop increments, or may set the exposure randomly around the initial shooting parameters.

In the above, an embodiment has been described in which exposure is used as a photographing parameter, but the photographing parameter set in this embodiment is not limited to exposure. Any shooting parameters may be used as long as they control the shooting unit 101, such as focus, white balance (color temperature), shutter speed, aperture, ISO sensitivity, image saturation, hue, etc. may be used as a shooting parameter.
Further, in FIG. 6A, an embodiment has been described in which only exposure is the shooting parameter to be adjusted in this embodiment, but a plurality of shooting parameters may be adjusted simultaneously. For example, FIG. 6(B) is a diagram illustrating an embodiment in which a combination of exposure and focus is used as a shooting parameter to be adjusted. In FIG. 6B, a certain combination of exposure and focus parameters is set as an initial parameter, and is shown as a white circle 511. The photographing parameter setting unit 105 may set a combination of photographing parameters, such as the black circle 512, as a plurality of photographing parameters, centering on these initial parameters.

Note that the combination of shooting parameters to be adjusted is not limited to the combination of exposure and focus shown in FIG. 6(B), but may be a combination of other shooting parameters. Further, in the above description, an embodiment in which a combination of two parameters is adjusted has been described, but the number of combinations of imaging parameters is not limited to this, and combinations of three or more imaging parameters may be adjusted simultaneously. .
In S304, a plurality of imaging parameters are set in the imaging parameter setting unit 105 as described above. The following processing will be described in the case where the photographing parameter to be adjusted is exposure as shown in FIG. 6(A).

In S305 of FIG. 4, the photographing unit 101 photographs the subject using the plurality of photographing parameters set in S304. More specifically, when three exposures are set as multiple shooting parameters as shown in FIG. 6A, the shooting unit 101 shoots three images while changing the exposure according to the user's shutter operation. Capture images automatically. Hereinafter, the image photographed in this step will be referred to as a photographed image.

The processes after S306 in FIG. 4 are mainly executed by the estimation unit 104, and are processes for selecting optimal imaging parameters or processing for further searching for optimal imaging parameters.
In S306, the estimation unit 104 calculates evaluation values for each of the plurality of imaging parameters. The evaluation value indicates a higher value as the photographing parameters are more appropriate for photographing the inspection image. The estimation unit 104 calculates this evaluation value by comparing a photographed image photographed with each photographing parameter and a reference image. More specifically, if the captured image is similar to the reference image, it can be determined that the imaging parameters used to capture the captured image are preferable parameters. Therefore, in such a case, the estimation unit 104 calculates a high evaluation value. In order to calculate this evaluation value, it is sufficient to calculate the degree of similarity between the photographed image and the reference image. A specific example of the evaluation value calculation method will be described below.

As an example of a method for calculating an evaluation value between a photographed image and a reference image, a method of calculating the similarity of the entire image and using it as an evaluation value will be described first. For example, if you want to compare the similarity of the entire image by the brightness of the entire image, after converting the captured image and the reference image to grayscale, create a luminance histogram of the entire image, and then compare the luminance histogram of the captured image and the reference image. What is necessary is to calculate the degree of similarity with the brightness histogram of . The degree of similarity between histograms can be calculated by simply calculating Euclidean distance, or by a method such as Histogram Intersection. In addition, when calculating the similarity in color tone of the entire image, create a color histogram for each image based on a color space such as RGB or YCrCb without performing gray scale conversion, and calculate the similarity of the color histogram. All you have to do is calculate. The feature amounts for determining the similarity of the entire image are not limited to these histogram feature amounts, and other feature amounts may be used.

Further, as another example of an evaluation value calculation method, partial similarity of images may be calculated. For example, when you want to take an inspection image of a concrete wall, the part of the image that is of interest is the part of the image that shows the concrete wall. Therefore, if a portion other than the concrete wall surface is captured in the photographed image and the reference image, the degree of similarity may be calculated based on the image of the portion of the concrete wall surface excluding that portion. More specifically, for example, when a bridge deck is photographed from below the bridge, a sky area (background portion) may be included in the photographed image. In such a photographed image, the estimation unit 104 calculates an evaluation value by removing the sky area and calculating the degree of similarity between the image portion of the concrete wall surface of the floor slab and the reference image. In calculating this degree of similarity, the above-described histogram feature amounts may be created for each of the partial image of the captured image and the entire reference image, and the degree of similarity between the histogram feature amounts may be calculated. In this example, the degree of similarity between a partial image on the photographed image side and the reference image is calculated on the premise that the entire reference image is a concrete wall image. However, if a portion of the reference image includes a portion that can be considered as a background, the degree of similarity between the partial image of the reference image and the photographed image may be calculated.

Furthermore, another method of calculating the partial similarity of images will be explained. When taking images of concrete walls for image inspection, it is important to take the images at a quality that allows thin cracks to be seen in the images. Assuming that the reference image is an ideal inspection image in which thin cracks can be sufficiently confirmed, it is preferable that the thin crack portion in the photographed image is photographed in the same manner as the thin crack portion in the reference image. In order to determine whether the cracked portion is similarly photographed, the estimation unit 104 calculates an evaluation value using a partial image of the cracked portion in the image. Any method may be used to calculate the evaluation value of the image of the cracked portion, but in the following example, the more similar the edge strengths of the cracked portion, the higher the evaluation value is calculated. For this purpose, first, the estimating unit 104 identifies cracks between the photographed image and the reference image. This method of identifying the cracked portion may be performed automatically or may be performed manually by the user. When automatically detecting cracks, the estimation unit 104 uses automatic crack detection processing. When manually specifying the crack position, the estimating unit 104 receives an input of the crack position in the image from the user via the operation unit 12. Although it is necessary to specify the crack position in a photographed image after photographing through these processes, the crack position in the reference image is specified in advance and recorded in the image storage unit 103 in association with the stored image. You can leave it there. If the crack positions between the photographed image and the reference image have been obtained as described above, the estimation unit 104 calculates the edge strength of the image at each crack position. The edge strength may simply be the brightness value at the crack position, or the gradient at the crack position may be calculated using a Sable filter or the like, and the gradient strength may be used as the edge strength. These edge strengths are determined on a pixel-by-pixel basis, so in order to calculate the degree of similarity in edge strength between the photographed image and the reference image, it is necessary to create an edge strength feature amount for the entire image. For this purpose, for example, the estimating unit 104 may create a histogram feature amount by forming a histogram of the edge strength of the crack position in each image. The estimation unit 104 calculates the similarity of the edge strength histogram feature amount between the captured image and the reference image, and the higher the similarity, the higher the evaluation value between the captured image and the reference image.

Note that, as described above, in order to calculate the evaluation value based on the edge strength of the cracked portion, it is a prerequisite that the crack is included in both the photographed image and the reference image. Regarding the photographed image, the user can obtain a photographed image including the crack by photographing the part where the crack exists from the concrete wall surface to be photographed. On the other hand, regarding the reference image, in the reference image selection process of S301 and S302, the user searches and selects an image containing cracks from among the stored images stored in the image storage unit 103 as the reference image. Select and set.
In the above, evaluation values between the photographed image and the reference image were calculated based on the edge strength of the crack position, but cracks do not always exist on the concrete wall surface to be photographed. Therefore, the estimating unit 104 may calculate the evaluation value based on the edge strength of an image edge portion that definitely appears due to the structure of concrete, such as concrete joints or formwork traces, for example. In this case, the evaluation value is calculated using the same method as the evaluation value calculation method using the edges of the cracked area described above, except that the edge strength of the concrete joint and formwork traces included in the photographed image and the reference image is used. It can be calculated.

Note that, as described above, in order to calculate the evaluation value based on the edge strength of the concrete joint, it is a prerequisite that the concrete joint is included in both the photographed image and the reference image. Regarding the photographed image, the user can obtain a photographed image including cracks by photographing the part where concrete joints are present from the concrete wall surface to be photographed. On the other hand, regarding the reference image, in the reference image selection process of S301 and S302, the user searches for an image containing concrete joints from among the stored images stored in the image storage unit 103 so that the image containing concrete joints becomes the reference image. and select and set.

Further, as a modification of the evaluation value calculation using the edge strength of the crack position, the estimation unit 104 may calculate the evaluation value of the edge strength between the photographed image and the reference image using information on the crack width. . In this method, the estimation unit 104 calculates a higher evaluation value as the edge strengths of cracks having the same thickness and width in the captured image and the reference image are similar. FIG. 7 is a diagram illustrating a method of calculating an evaluation value based on a partial image of a crack position using information on the crack width. First, FIG. 7A is an example of a captured image 620 captured using certain imaging parameters, and is assumed to be an image showing a crack 600 in a concrete wall surface. The crack 600 is a crack that has various crack widths depending on the part within one crack. In FIG. 7A, it is assumed that the local crack width has been measured for this crack 600. For example, FIG. 7A shows locations where crack widths of 0.15 mm, 0.50 mm, etc. are obvious. These crack widths are assumed to be input by the user by measuring the actual crack widths on the concrete wall surface and inputting them via the operation unit 12 while taking an image. Alternatively, the user may check the photographed image, estimate the crack width, and input the image via the operation unit 12 while the image is being photographed. The CPU 10 stores the input crack width in the image storage unit 103 in association with the photographed image. On the other hand, FIG. 7B is an example of a reference image 621, which is an image of a concrete wall surface in which cracks 610 are shown. Like the crack 600, the crack 610 is also a crack that has various crack widths depending on the part within one crack. Regarding the crack 600, local crack widths are also recorded; for example, in FIG. 7(B), crack widths of 0.10 mm, 0.50 mm, etc. are recorded. It is assumed that the crack width information of these reference images is stored in the image storage unit 103 and is called from the image storage unit 103 together with the reference image 621.

ここで、ｄ_iは、あるひび割れ幅（例えば、０．１０ｍｍ幅のひび割れ）の画像部分の類似度である。α_iは、あるひび割れ幅の評価値への重みである。α_iは、例えば、細いひび割れ幅に大きな重みを付けるようにする。このようにすることで、撮影画像の細いひび割れ部分が、基準画像の画質に一致するほど高い評価値が算出される。したがって、細いひび割れ部分が基準画像の画質に近づくことを重視して、撮影条件を調整することができるようになる。
なお、ひび割れ部分の画像を用いて評価値を算出する場合には、撮影画像と基準画像とのコンクリート壁面の撮影解像度は、同解像度とすることが好ましい。より具体的には、撮影画像と基準画像とに写るコンクリート壁面が、例えば１．０ｍｍ／画素となる解像度になるように調整する処理を予め実施する。これは、解像度によって、同じひび割れでもエッジ強度等の見た目が変化するためである。また、画像中のコンクリート壁面が正対するように、あおり補正を予め実施することも好適な実施形態である。 In calculating the evaluation value between the photographed image 620 and the reference image 621 in FIG. 7, the estimating unit 104 compares the edge strengths of cracked portions with the same crack width. For example, for a portion with a crack width of 0.50 mm, the estimation unit 104 calculates the degree of similarity based on the edge strength between the partial image 601 of the photographed image 620 and the partial image 611 of the reference image 621. Furthermore, for a portion with a crack width of 0.10 mm, the estimation unit 104 calculates the degree of similarity based on the edge strength between the partial image 602 of the photographed image 620 and the partial image 612 of the reference image 621. In this way, the evaluation value s between the photographed image 620 and the reference image 621 is determined by the following formula based on the similarity between image portions having the same crack width.

Here, d _i is the similarity of image parts with a certain crack width (for example, a crack with a width of 0.10 mm). α _i is the weight given to the evaluation value of a certain crack width. α _i is set such that, for example, a large weight is given to narrow crack widths. By doing so, a higher evaluation value is calculated as the narrow crack portion of the photographed image matches the image quality of the reference image. Therefore, it becomes possible to adjust the photographing conditions with emphasis on bringing the image quality of the thin crack portion closer to that of the reference image.
In addition, when calculating an evaluation value using the image of a crack part, it is preferable that the imaging|photography resolution of the concrete wall surface of a photographed image and a reference|standard image shall be the same resolution. More specifically, processing is performed in advance to adjust the resolution of the concrete wall surface appearing in the captured image and the reference image to, for example, 1.0 mm/pixel. This is because the appearance of the same crack, such as edge strength, changes depending on the resolution. It is also a preferred embodiment to perform tilt correction in advance so that the concrete wall surface in the image faces directly.

ここで、ｘ_nは任意の基準画像である。ｙ_nは任意の撮影画像である。ｔ_nは、ｘ_nとｙ_nとが類似画像と見なせる場合には１をとり、類似画像と見なせない場合には０をとる教師データである。このデータセットを用いて学習を行う学習方法はどのようなものを用いてもよいが、例えば、ＣＮＮ（Ｃｏｎｖｏｌｕｔｉｏｎａｌ Ｎｅｕｒａｌ Ｎｅｔｗｏｒｋ）を用いた学習方法の例として、非特許文献１のような方法がある。非特許文献１に記載の方法でデータセットＤを学習したモデルは、評価値を算出したい撮影画像と基準画像とをモデルに入力すると、評価値を算出することができる。
以上では、撮影画像と基準画像との評価値の算出方法について様々な手法を説明した。説明した方法以外にも、画像と画像との類似度の算出方法については、従来、様々な方法が提案されている。推定部１０４は、これら公知の手法を用いて本実施形態の評価値を算出してもよい。 In the above, an embodiment has been described in which image feature amounts are created for each of the photographed image and the reference image, and the degree of similarity between the images is calculated based on the distance between the feature amounts, etc., and this is used as the evaluation value. The method of calculating the similarity of images is not limited to this, and the evaluation value may be calculated using a learning model learned in advance. In this method, a model is trained in advance that outputs a higher evaluation value as the input image and the reference image become more similar. This learning can be performed using, for example, the following data set D.

Here, x _n is an arbitrary reference image. y _n is an arbitrary photographed image. t _n is teacher data that takes 1 when x _n and y _n can be considered similar images, and takes 0 when they cannot be considered similar images. Any learning method may be used to perform learning using this data set, but for example, as an example of a learning method using CNN (Convolutional Neural Network), there is a method as described in Non-Patent Document 1. . A model that has learned the dataset D using the method described in Non-Patent Document 1 can calculate an evaluation value by inputting a photographed image for which an evaluation value is to be calculated and a reference image to the model.
In the above, various methods have been described for calculating evaluation values between a photographed image and a reference image. In addition to the method described above, various methods have been conventionally proposed for calculating the similarity between images. The estimation unit 104 may calculate the evaluation value of this embodiment using these known methods.

ここで、ｓ_jは、ある方法により求めた評価値である。ｗ_jはその方法の重みである。
Ｓ３０６では、以上のような方法で、推定部１０４は、撮影画像と基準画像との評価値を算出する。また、Ｓ３０６では、推定部１０４は、複数の撮影パラメータで撮影した撮影画像それぞれについて評価値を算出する。 Moreover, although a plurality of methods have been described above regarding the evaluation value calculation method, each of these evaluation value calculation methods may be used alone, or a plurality of methods may be used in combination. When combining multiple methods, the estimation unit 104 calculates the final evaluation value s using the following formula, for example.

Here, s _j is an evaluation value obtained by a certain method. w _j is the weight of the method.
In S306, the estimation unit 104 calculates the evaluation value of the captured image and the reference image using the method described above. Further, in S306, the estimating unit 104 calculates an evaluation value for each photographed image photographed using a plurality of photographing parameters.

In S307, the estimation unit 104 evaluates the imaging parameters based on the evaluation value calculated in S306.
In S308, the estimation unit 104 determines whether to readjust the imaging parameters based on the evaluation result.
When readjusting the imaging parameters, in S309, the estimating unit 104 estimates a method for improving the shadow parameters. Then, the estimating unit 104 returns to the process of capturing multiple images in S305.
If the photographing parameters are not readjusted, the photographing parameter setting section 105 sets the photographing parameters in the photographing section 101 in S310. Then, the process of the flowchart shown in FIG. 4 ends.
These processes will be explained below.

First, in the imaging parameter evaluation in S307, the estimation unit 104 selects the maximum evaluation value from the evaluation values of the plurality of imaging parameters, and compares it with a predetermined threshold value. FIG. 8(A) is a diagram illustrating evaluation of each imaging parameter. In this embodiment, three exposures (EVs) are set as a plurality of shooting parameters. In FIG. 8(A), exposures -1, 0, and +1 are set as a plurality of shooting parameters, which is represented by triangles 501, 502, and 503, as in FIG. 6(A). Moreover, the lower part of FIG. 8(A) shows evaluation values s _-1 , s ₀ , and s ₊₁ obtained from the photographed image photographed with each photographing parameter and the reference image. In FIG. 8(A), the evaluation value s ₊₁ of the exposure 503 of +1 is the highest evaluation value, and indicates a value exceeding the predetermined threshold value s _th . If there is a photographing parameter having an evaluation value exceeding the predetermined threshold s _th , the estimation unit 104 determines that the photographing parameter is suitable as a photographing parameter for an inspection image. In the case of FIG. 8A, the estimation unit 104 selects +1 exposure 503 as the optimal parameter. Then, in S308, the estimating unit 104 determines that readjustment of the imaging parameters is unnecessary, and proceeds to S310 for setting imaging parameters. In S310, the photographing parameter setting unit 105 sets an exposure of +1 to the photographing unit 101, and ends the process shown in FIG. 4.

Similar to FIG. 8(A), FIG. 8(B) is an example in which exposures -1, 0, and +1 are set as multiple shooting parameters and evaluation values are calculated, but this is different from FIG. 8(A). Indicates the situation in which evaluation values are obtained. In FIG. 8(B), the evaluation value s ₊₁ shows the maximum evaluation value, but even s ₊₁ does not exceed the predetermined threshold value s _th . Photographed images photographed using these photographic parameters have a low degree of similarity to the reference image, and these photographic parameters are not suitable as photographic parameters for inspection images. In this case, in S308, the estimating unit 104 determines that it is necessary to readjust the imaging parameters, and proceeds to S309. In S309, the estimation unit 104 estimates a method for improving the imaging parameters.

Estimation of a method for improving imaging parameters will be explained using FIG. 8(B). In FIG. 8B, although the evaluation value s ₊₁ for exposure of +1 is less than the threshold value s _th , it is the largest evaluation value among the evaluation values s _-1 to s ₊₁ . Therefore, in readjusting the shooting parameters, the estimation unit 104 sets a plurality of shooting parameters from shooting parameters around this shooting parameter (+1 exposure). For example, when setting three shooting parameters in the next shooting parameter adjustment, the estimation unit 104 sets exposures 721, 722, and 723 around +1 exposure 503 as multiple parameters, as shown in FIG. 8(B). do. Then, the process returns to S305, and these photographing parameters are set in the photographing section 101 via the photographing parameter setting section 105, and a plurality of images are photographed again. Then, the estimating unit 104 again executes the processes after S306 in FIG. 4 (evaluation value calculation process) to search for optimal imaging parameters. Even in the evaluation of this imaging parameter set, if an evaluation value equal to or greater than the threshold value s _th is not obtained, the estimation unit 104 again determines a plurality of new imaging parameters around the imaging parameter indicating the maximum evaluation value. Then, the photographing process is executed again. This loop is repeatedly executed until an imaging parameter that provides an evaluation value exceeding the threshold value s _th is determined. Note that the maximum number of repetitions may be determined in advance, and if the optimal imaging parameter (imaging parameter that yields an evaluation value greater than or equal to the threshold value s _th ) is not obtained by then, the process may be aborted. When the imaging parameter adjustment process is aborted, the estimation unit 104 displays a warning on the operation unit 12 to notify the user that the imaging parameters have not been adjusted sufficiently. Further, the photographing parameters for photographing the image for which the maximum evaluation value obtained before the process is terminated may be set in the photographing section 101 via the photographing parameter setting section 105.

In the above description, an embodiment has been described in which estimation of improved imaging parameters and repeated adjustment are performed only when an evaluation value equal to or greater than the threshold value s _th is not obtained in S307. However, even if a photographing parameter having an evaluation value greater than or equal to the threshold value s _th is found, a photographing parameter having an even higher evaluation value may be searched. In this case, the information processing device 100 sets the shooting parameters around the shooting parameter that showed the maximum evaluation value as improved shooting parameters, shoots a plurality of images again, and repeatedly executes the evaluation value calculation process. The conditions for terminating this repetitive process are when a predetermined number of repetitions is reached, or when the evaluation value no longer changes even if the imaging parameters are changed around the maximum evaluation value.

On the other hand, the user may check the imaging parameters and determine whether to end the repetitive process for adjusting the imaging parameters. In this case, in S308 of FIG. 4, the estimating unit 104 does not perform the optimal imaging parameter determination using the evaluation value threshold s _th , and instead makes the determination of whether or not to readjust the imaging parameters based on the user's operation. Judgment based on. For this purpose, the estimation unit 104 presents necessary information to the user on the operation unit 12 and further receives input from the user via the operation unit 12. FIG. 9 is a diagram illustrating the operation unit 12 when photographing parameters are adjusted based on the user's judgment. The information presented to the user and the user's operations will be described below using FIG. 9.
First, the operation unit 12 in FIG. 9 is a display for displaying information. An image 801 on the screen displayed on the operation unit 12 is a photographed image taken with a photographing parameter of exposure +1, and photographed images 802 and 803 are photographed images taken with other photographic parameters. A reference image 804 is also displayed on the display, and the user can compare and confirm the captured image and the reference image.
Further, at the bottom of the image 801, a plurality of shooting parameters set for adjusting the shooting parameters are shown. In FIG. 9, three levels of exposure (EV) are indicated by black triangles as an example of a plurality of photographing parameters. Among these, the black triangle 811 indicating exposure +1, which has the maximum evaluation value, is highlighted (largely displayed). Furthermore, an open triangle 812 and the like indicate a plurality of photographing parameter candidates that are set based on the photographing parameter 811 of exposure +1 and are used when the photographing parameter is further adjusted.

In this embodiment, the user checks the information displayed on the operation unit 12 and determines whether to adopt the current imaging parameters or to further execute the imaging parameter adjustment process. More specifically, the user compares the photographed image with the reference image using the image 801 for which the maximum evaluation value was obtained, and if the degree of agreement is satisfactory, the user selects the photographing parameter that showed the maximum evaluation value. You can decide to adopt it. The user selects the icon 821 displayed as "set" when selecting the imaging parameter showing the maximum evaluation value. Through this operation, the photographing parameter setting section 105 sets the photographing parameter indicating the maximum evaluation value in the photographing section 101 (S310 in FIG. 4), and the photographing parameter adjustment process ends. On the other hand, if the user is not satisfied with the current optimal imaging parameters after checking the image 801, etc., the user selects an icon 822 labeled "Readjust". In response to this instruction, the processes after S306 in FIG. 4 (evaluation value calculation process) are executed again using the next plurality of shooting parameters (for example, exposure 812, etc.). After that, the information processing device 100 again presents various types of information to the user as shown in FIG. 9. Based on the presented information, the user decides whether to adopt the imaging parameters again or to further adjust the imaging parameters.

If you wish to quit adjusting the photographing parameters midway through, select the icon 823 labeled "End". With this operation, the information processing apparatus 100 can end the process of adjusting the imaging parameters (the loop in the flowchart of FIG. 4). At this time, the information processing apparatus 100 may set, in the imaging unit 101, the imaging parameter with the highest evaluation value among the imaging parameters that have been photographed and evaluated up to that point.
Note that even when determining whether to continue adjusting shooting parameters through user operations, a threshold value s _th for the evaluation value is set in advance, and the presence of shooting parameters with evaluation values exceeding the threshold s _th is displayed. You may also do so. For example, in FIG. 9, when the evaluation value s ₈₁₁ of the image shot using the shooting parameters 811 exceeds the threshold value s _th , the information processing device 100 may blink a black triangle 811 indicating the shooting parameters. . Although the user can select a shooting parameter regardless of the evaluation value, the information processing device 100 can assist in deciding whether to adopt a shooting parameter by displaying the existence of a shooting parameter that exceeds the evaluation value. You will be able to do this.

As mentioned above, in Embodiment 1, an embodiment for estimating a method for improving imaging parameters has been described. However, the imaging method estimated by the method of this embodiment is not limited to imaging parameters, but can also be estimated by estimating other imaging methods. good. In an embodiment in which a photographing method other than photographing parameters is estimated, if an evaluation value equal to or higher than a predetermined threshold is not obtained even after executing the loop of the processing flow in FIG. 4 multiple times, the estimation unit 104 further estimates the image or Analyze the shooting situation and suggest an appropriate shooting method. For example, if it is determined that the brightness of the image is insufficient, or if it is determined that the white balance cannot be adjusted using the shooting parameters, the estimating unit 104 may ask the user to use lighting. A notification may be issued that recommends photographing at a brighter time by changing the conditions or changing the photographing time. As another example, if the position and orientation of the imaging unit 101 can be acquired, the estimating unit 104 analyzes the positional relationship with the inspection target structure and proposes a position and orientation for improving imaging. More specifically, for example, if the image is being photographed in a position and posture with a large tilt angle relative to the wall surface of the structure to be inspected, the estimation unit 104 recommends to the user that the image be photographed in a position and posture that reduces the tilt angle. .

<Embodiment 2>
In the first embodiment, one image is selected from the image storage unit 103 and used as the reference image. An embodiment has been described in which the photographing method is adjusted based on this selected one reference image. In Embodiment 2, an embodiment will be described in which a photographing method is adjusted using a plurality of reference images. Note that in the following embodiments, mainly the differences from the first embodiment will be explained.
First, in the second embodiment, the reference image processing unit 102 selects a plurality of reference images. In the following, it is assumed that the reference image processing unit 102 selects M reference images. The M reference images may be selected using any method, but for example, the M reference images may be the top M stored images in the search results.

以降の処理は、ＣＰＵ１０が、評価値ｓを基に、撮影パラメータの調整を実施する（実施形態１の図４Ｓ３０７以降の処理を実施する）。評価値ｓ^mの平均を用いることにより、複数の基準画像と全体的に類似する画像が撮影されるように、撮影パラメータが調整される。 Next, the estimating unit 104 calculates evaluation values for the photographed image and the M reference images. In this process, first, evaluation values of the photographed image and each reference image are calculated. For example, the estimation unit 104 calculates the evaluation value of the photographed image and the m-th reference image, and sets this evaluation value as s ^m . The evaluation value calculation method for the photographed image and the reference image is the same as the method in the first embodiment. When M evaluation values are obtained by calculating the evaluation values of the photographed image and M reference images, the estimation unit 104 calculates the final evaluation value s by averaging these values.

In subsequent processing, the CPU 10 adjusts the imaging parameters based on the evaluation value s (implements the processing from S307 onward in FIG. 4 of the first embodiment). By using the average of the evaluation values s ^m , the photographing parameters are adjusted so that an image that is overall similar to the plurality of reference images is photographed.

この方法では、Ｍ枚の基準画像のうち、いずれかに類似するように、撮影パラメータが調整される。Ｍ枚の基準画像は、それぞれ好ましい画質の画像であるから、撮影画像は、そのいずれかに類似していればよい。 Further, as another form using a plurality of reference images, there is a method of adjusting photographing parameters based on an evaluation value with one reference image that is most similar among the plurality of reference images. In this case, the final evaluation value s of the photographed image and the M reference images is determined by the following formula.

In this method, imaging parameters are adjusted to resemble one of M reference images. Since each of the M reference images has a preferable image quality, the photographed image only needs to be similar to any of them.

<Embodiment 3>
In the above embodiments, the reference image is assumed to be an image photographed with a structure different from the structure to be photographed, but a past image of the structure to be photographed may be used as the reference image. In infrastructure inspections, past inspection results and the latest inspection results are compared, and for this comparison, it is preferable that the past images and the latest images are captured with the same image quality. If a past image of the structure to be photographed exists, by using this as a reference image, the photographing parameters can be adjusted so that an image similar to the past image can be photographed.

In order to use past images as reference images, the image storage unit 103 of the third embodiment stores past images of structures to be photographed. The reference image processing unit 102 performs processing to acquire past images from the image storage unit 103 and set them as reference images based on information on the structure to be photographed. To this end, the reference image processing unit 102 of the third embodiment may be configured to search the images stored in the image storage unit 103 using unique information such as the name of a structure. After setting the past image of the structure to be photographed as the reference image, the photographing method can be adjusted by performing the same process as in the first embodiment. With the above configuration, it becomes possible to adjust the shooting parameters so that a shooting result similar to a past image can be obtained.

When setting a past image as a reference image, it is preferable that the shooting ranges of the reference image and the photographed image match. In this case, with respect to the structure to be photographed, the photographing position and photographing range are adjusted so that the same range photographed in the past image set as the reference image is photographed in the current photographing. In order to support this adjustment of the photographing position and range, information on the photographing position and photographing range may be stored in association with the past images stored in the image storage unit 103.
When the photographing ranges of the reference image (past image) and the photographed image match, the evaluation value may be calculated using the reciprocal of the sum of squared errors between the pixels of the past image and the photographed image. In reality, it is extremely difficult to match past and present images at the pixel level, so it is preferable to use a similarity calculation method that allows for slight positional deviation.

Further, if the past image includes deformation of the concrete wall surface, the evaluation value may be calculated based on the deformed portion in the image. In this case, the estimation unit 104 photographs the same portion as the deformation of the past image, and calculates a higher evaluation value as the deformation of the past image and the deformation of the photographed image become more similar. By doing this, it becomes possible to adjust the photographing parameters so that deformities that appeared in past images can also be confirmed in the photographed images. Further, the estimating unit 104 may calculate evaluation values for past images and photographed images, taking into account changes in deformation over time. For example, a case where the deformation included in the past image is a crack will be explained. Cracks recorded during past inspections will not disappear on their own unless repair work is carried out. On the other hand, cracks may extend due to aging. Therefore, when comparing a crack in a photographed image with a crack in a past image, the estimation unit 104 does not use an image of an extended portion of the crack in the photographed image to calculate the similarity of the crack portion.

<Embodiment 4>
In the above embodiment, the reference image processing unit 102 searches the images stored in the image storage unit 103 and sets the reference image. In the fourth embodiment, an embodiment in which an image is generated by the reference image processing unit 102 and this generated image is used as the reference image will be described.
In recent years, learning-based methods have been developed for image denoising and super-resolution. For example, Non-Patent Document 2 describes an image noise removal technique using an autoencoder. This technique learns a denoising model by training an autoencoder on images with noise and images without noise. When a noisy image from which noise is to be removed is input to this noise removal model, an image from which noise has been removed is obtained as an output. Furthermore, Non-Patent Document 3 is an image super-resolution technique using Fully CNN. In this technique, a super-resolution model is learned by learning a Fully CNN using low-resolution images and high-resolution images. When a low-resolution image to be enhanced is input to this super-resolution model, a high-resolution image is obtained as output. These techniques are techniques for acquiring an image transformation model through learning. In the fourth embodiment, these techniques are used to generate a reference image from a temporarily captured image. Note that although the technology of noise removal and super resolution is taken as an example, the technology used in Embodiment 4 may be any method as long as it can perform image conversion. The present invention is not limited to the technique of Document 3.

ここで、ｘ_nは撮影パラメータを調整が不十分な状態で撮影した画像である。ｘ_nに対応するｙ_nは、ｘ_nと同一の撮影対象を好ましい撮影パラメータで撮影した画像である。この学習データセットＤを用いると、基準画像生成モデルＦの学習は、以下の式で表される。

ここで、Ｆ（ｘ_n）－ｙ_nの部分は、基準画像生成モデルＦで画像ｘ_nを変換した画像と、画像ｙ_nとの誤差を表す。したがって、データセットＤのＮ個のデータについて、この誤差が最少となる基準が造成性モデルＦを学習することになる。 FIG. 10 is a diagram illustrating an example of the configuration of an information processing device 100 according to the fourth embodiment. The information processing apparatus 100 of the fourth embodiment differs from FIG. 2 (first embodiment) in that it includes a model storage section 106 instead of the image storage section 103. The model storage unit 106 stores a model for generating a reference image. Hereinafter, this model will be referred to as a reference image generation model. The reference image generation model is obtained by learning an image conversion method using a technique such as that described in Non-Patent Document 2 or Non-Patent Document 3. The reference image generation model is learned using, for example, the following learning data set D.

Here, x _n is an image shot with the shooting parameters insufficiently adjusted. y _n corresponding to x _n is an image obtained by photographing the same photographic subject as x _n using preferable photographing parameters. Using this learning data set D, learning of the reference image generation model F is expressed by the following equation.

Here, the part F(x _n )−y _n represents the error between the image obtained by converting the image x _n using the reference image generation model F and the image y _n . Therefore, for the N pieces of data in the data set D, the creation model F is learned based on the criterion that minimizes this error.

When an image whose photographing parameters have not been adjusted is input to the learned reference image generation model F, an image photographed with preferable photographing parameters (generated image) is output. However, since the generated image is a fake image generated by the reference image generation model F, there is a risk in using it as it is as an inspection image or the like. Although it depends on the performance of the reference image generation model, for example, there is a possibility that the generated image includes fine artifacts accompanying image generation processing. Therefore, in this embodiment, the generated image itself is not used as a photographing result, but is used as a reference for adjusting photographing parameters.
The model storage unit 106 stores the reference image generation model F learned as described above. In addition, as a learning method for an image generation model, a method called GAN (Generative adversarial nets), as described in Non-Patent Document 4, has been developed in recent years. This method may be used to learn the reference image generation model F.

Next, the process of creating a reference image using this image generation model F will be explained. First, a user temporarily photographs an object to be photographed using the photographing unit 101. It is assumed that the photographing parameters for this temporary photographing use automatic settings, etc., and that the temporary photographic image is an image in which the photographing parameters are insufficiently adjusted when photographing the photographic subject. The reference image processing unit 102 creates a generated image by inputting the temporary photographed image into the image generation model F read out from the model storage unit 106, and sets this generated image as a reference image for adjusting photographing parameters.
In the above, the generated image is created using the temporary photographed image, but the generated image may also be created using information about the photographic target. For example, as one method, first, the reference image generation model F is learned for each condition, such as for each type of structure to be photographed or for each type of concrete. The model storage unit 106 stores a plurality of reference image generation models together with learning condition information. In the step of creating a reference image, the user specifies the conditions of the object to be imaged (for example, the type of structure to be imaged), calls the reference image generation model matching the conditions from the model storage unit 106, and generates the image. used for By doing the above, it becomes possible to create a generated image using a reference image generation model suitable for the object to be photographed.

Next, another embodiment will be described in which a generated image is created using information about a photographic target. In this embodiment, in addition to the model storage section 106, an image storage section 103 is provided as in the first embodiment. The image storage unit 103 stores ideal photographic result images of the photographic subject, as in the first embodiment. The user selects an image similar to the conditions of the photographing target from the image storage unit 103. This operation of selecting an image from the image storage section 103 can be performed by searching the image storage section 103 based on the information of the photographing target, similarly to the reference image selection in the first embodiment. In this embodiment, the image selected from the image storage unit 103 is called a style image. Then, for example, using a technique such as that disclosed in Non-Patent Document 5, the appearance of the temporarily photographed image is converted into an image similar to the style image. Non-Patent Document 5 is a technique that, when an original image and a style image are input, converts the appearance of the original image into an image similar to the style of the style image, and is a technique that can, for example, convert the style of the image. In this embodiment, by setting a temporary photographed image to the original image of Non-Patent Document 5, it is possible to make the appearance of the temporary photographed image similar to the style image and create an ideal photographed result image. The image created in this way is used as a reference image. According to this embodiment, by selecting a style image from the image storage unit 103 using information about the subject to be photographed, it becomes easier to generate an image similar to the subject to be photographed.
In the fourth embodiment, the generated image generated by the reference image processing unit 102 as described above is used as the reference image. By performing the subsequent processing in the same manner as in Embodiment 1, it is possible to adjust the photographing parameters for photographing an image similar to the reference image.

ここで、式（８）のｐ_nは撮影パラメータである。ｓ_nはｐ_nで撮影した画像から得られた評価値である。ｓ_nは閾値以下の評価値とする。式（９）のｐ_dst__nは、（ｓ_n、ｐ_n）の状態から撮影パラメータを調整し、最終的に評価値が閾値以上となったときの撮影パラメータである。これらのデータの組をｎ個収集し、学習データ（Ｘ、Ｙ）を作成する。この学習データを用いて、ある閾値未満の評価値ｓと撮影パラメータｐとが入力されたときに、改善パラメータｐ_dstを出力するモデルＥを学習する。

このモデルの学習には、どのようなアルゴリズムを用いてもよく、例えば、撮影パラメータを連続値であるとすると、線形回帰等の回帰モデルを適用することができる。 Embodiment 4 will also describe an embodiment in which multiple shooting parameter settings and multiple shootings (S304 and S305 in FIG. 4) are abolished, and shooting parameters are adjusted from a reference image and one shot image. .
First, in the fourth embodiment, one image of the object is photographed using certain initial parameters. A process (S306) of calculating an evaluation value by comparing this one image with a reference image is executed. If the calculated evaluation value is equal to or greater than the threshold, a process for terminating the parameter setting (S307, S308, S310 in FIG. 4) is performed.
Processing that is different from the configuration using multiple imaging parameters in the first embodiment is S309 in which a method for improving the imaging parameters is estimated when the evaluation value is less than or equal to the threshold value. In the fourth embodiment, improved imaging parameters are estimated using a statistical method from a certain evaluation value and the imaging parameters at that time. For this purpose, in the fourth embodiment, the relationship between the evaluation value and the improved imaging parameter is learned in advance. This relationship can be learned using, for example, the following data.

Here, p _n in equation (8) is a photographing parameter. s _n is the evaluation value obtained from the image taken at p _n . s _n is an evaluation value that is less than or equal to the threshold value. p _{dst_n} in equation (9) is the imaging parameter when the imaging parameter is adjusted from the state of (s _n , p _n ) and the evaluation _value finally becomes equal to or greater than the threshold value. N sets of these data are collected to create learning data (X, Y). This learning data is used to learn a model E that outputs an improvement parameter p _dst when an evaluation value s less than a certain threshold value and a shooting parameter p are input.

Any algorithm may be used to learn this model. For example, if the imaging parameters are continuous values, a regression model such as linear regression may be applied.

In the above, an embodiment has been described in which a model for calculating the improvement parameter p _dst is learned using the evaluation value s and the photographing parameter p as input, but information on the photographed image may be input to this model. The information on the photographed image is, for example, the feature amount of the entire image, and more specifically, the brightness histogram of the entire image. The information on the photographed image is not limited to this, and may be a partial feature amount of the image, or may be in the form of inputting the image itself to the model. By inputting image information into the model in this way, it becomes possible to estimate the improvement parameter p _dst based not only on the evaluation value and the photographing parameters but also on the photographed image.
By using the model E prepared in advance as described above in the improved parameter estimation in S309, in the fourth embodiment, it becomes possible to estimate the improved imaging parameters from only one image.

なお、目的変数（又は教師データ）Ｙは、式（９）と同様である。 Note that the configuration in which a learned model is used as a method for determining improved imaging parameters may be used in the method of the first embodiment in which images are captured using a plurality of imaging parameters. That is, the model E is not limited to a configuration in which photographing parameters are estimated from one image, but may be used in a method of estimating photographing parameters from a plurality of photographed images. In this case, as in Embodiment 1, a model E is created in which an evaluation value is calculated from a photographed image taken with a plurality of photographing parameters and a reference image, and an improvement parameter is calculated by inputting a plurality of photographing parameters and a plurality of evaluation values. learn. The learning data X for learning this model M can be rewritten from equation (8) as follows, assuming that the number of multiple images to be photographed in the photographing parameter adjustment is M.

Note that the objective variable (or teacher data) Y is the same as in equation (9).

<Embodiment 5>
In the fourth embodiment, an embodiment has been described in which a reference image is generated from a temporarily captured image using a reference image generation model. The reference image generation model is not limited to the embodiment of the fourth embodiment, but may be an embodiment that is used to generate a reference image from a stored image. In the fifth embodiment, an embodiment will be described in which a reference image is created by converting a stored image stored in the image storage unit 103 in advance. In the first embodiment, a stored image stored in the image storage unit 103 is selected and the selected stored image is set as the reference image, but in the fifth embodiment, the selected stored image is converted according to the shooting conditions and then set as the reference image. Set as the reference image. This embodiment will be described below.

Although the image storage unit 103 stores a large number of stored images under various photographing conditions, it is difficult to prepare images that match all photographing conditions. Therefore, a new image is generated by converting and adjusting the stored image according to the shooting conditions, and this generated image is used as a reference image. For example, a case where different camera models are used as photographing conditions will be explained. First, it is assumed that the stored images are composed only of images taken with a camera of a certain specific model (hereinafter referred to as camera A). On the other hand, it is assumed that the camera used to photograph the object is a camera of a different model from camera A (hereinafter referred to as camera B). Here, since camera A and camera B are different models, the image quality of the captured images is different. For example, since the quality of photographed images, such as color tone, differs depending on the camera model, even if camera A and camera B photograph the same object under the same circumstances, images with different image quality, such as color tone, are obtained.
In the fifth embodiment, in such a case, a stored image of camera A quality is converted into an image of camera B quality using a reference image generation model, and then set as the reference image. The reference image generation model in this case is, for example, a conversion parameter that converts the color tone of camera A to the color tone of camera B. Regarding the processing after the reference image is set, by performing the same processing as in the first embodiment, it is possible to adjust the shooting parameters to match the image quality of the reference image.

Next, processing for implementing such processing will be explained. The information processing apparatus of this embodiment further includes a model storage section in addition to the information processing apparatus 100 of FIG. 2, and the model storage section stores a reference image generation model according to imaging conditions. First, in the first process, similarly to the first embodiment, the reference image processing unit 102 searches the image storage unit for a stored image similar to the object to be photographed and acquires it. In the fifth embodiment, the stored image of this search result is used as a temporary reference image. In the next process, the reference image processing unit 102 prepares a reference image generation model for converting the temporary reference image into a reference image based on the imaging conditions. In the above example, the reference image processing unit 102 creates a reference image generation model that includes a conversion parameter for converting the color tone of camera A to the color tone of camera B by setting the camera model (camera B) as the shooting condition. Read from storage. For this purpose, the user inputs information on photographing conditions via the operation unit 12. Alternatively, information on photographing conditions that can be automatically acquired, such as camera model, may be automatically acquired and used for searching for a reference image generation model. In the next process, the reference image processing unit 102 converts the temporary reference image using this reference image generation model to create a reference image.

In the above example, an embodiment has been described in which a reference image generation model is selected based on the shooting conditions, and a temporarily shot image is converted to generate a reference image, using the camera model as the shooting condition. The photographing conditions for generating the reference image are not limited to the camera model, and may be other conditions. For example, the weather may be set as the shooting condition. In this case, if the stored image (temporary reference image) selected as similar to the photographic subject was an image photographed in sunny weather, and the weather when photographing the photographic subject was cloudy, the image quality of the sunny image would be A reference image generation model that converts (hue or brightness) into the image quality of a cloudy image is selected from the model store. Other variations in the photographing conditions may include photographing time, photographing season, and the like. Furthermore, conditions such as hand-held photography, tripod photography, or photography with a camera mounted on a moving object such as a drone may be one of the photography conditions. Further, such photographing conditions may be a combination of a plurality of conditions. For example, under the shooting condition "Camera B, cloudy", an image generation model may be selected that converts the temporary reference image of "Camera A, sunny" to the image quality of "Camera B, cloudy".

Further, in the above example, the image generation model is a parameter for converting an image, but the image generation model in the fifth embodiment is not limited to this, and may be a model based on learning as described in the fourth embodiment. In this case, for example, an image generation model is learned for each shooting condition for which an image is to be converted, and the image generation model is selected and used in accordance with the shooting condition. Learning of this image generation model can be performed in the same manner as the learning in the fourth embodiment by using a data set consisting of a group of images before conversion and a group of images after preferable conversion.
In the fourth and fifth embodiments, the reference image processing unit 102 displays the generated reference image on the operation unit 12 so that the user can confirm the reference image. If the user determines that the reference image is not suitable as a reference for adjusting the imaging parameters, the user may be able to generate another reference image again. In this case, you can display candidates for the reference image generation model so that you can reselect the reference image generation model for generating another reference image, or you can search for the reference image generation model again. Good too. Furthermore, when the methods of Embodiment 4 and Embodiment 5 are used simultaneously, a reference image obtained by converting the temporary photographed image (a reference image created by the method of Embodiment 4) is used, or a temporary reference image is converted. The user may be able to select whether to use the reference image (the reference image created by the method of the fifth embodiment). In this case, the reference image processing unit 102 displays, on the operation unit 12, a reference image obtained by converting the temporary photographed image and a reference image obtained by converting the temporary reference image in a state that allows the user to compare the reference image and the reference image obtained by converting the temporary reference image. To enable selection of a reference image suitable as a reference image.

<Embodiment 6>
In the above embodiment, an embodiment in which the information processing apparatus 100 of this embodiment is applied to photographing an inspection image of an infrastructure structure has been described. The information processing apparatus 100 of this embodiment can be applied not only to inspection image photography but also to adjustment of photography parameters for other photography targets. In a sixth embodiment, an embodiment will be described in which the above-described processing and the like are applied to photographing parameter adjustment in general photography.
In this embodiment, the stored image stored in the image storage unit 103 of Embodiment 1 is changed in order to apply the above-described processing and the like to general photography. FIG. 11 is a diagram illustrating information stored in the image storage unit 103 in the fifth embodiment. The image storage unit 103 in FIG. 11 stores stored images, image information, and photographing parameters as in FIG. 3. The stored image 1010 in FIG. 11 is a seascape photograph, and information such as scene: landscape, weather: sunny, detail 1: sea, detail 2: summer, etc. is recorded as image information of this stored image. Further, the stored image 1011 is a baseball image, and image information indicating the image content is also recorded in association with this stored image.

In this embodiment as well, as in the first embodiment, a reference image is selected from the image storage unit 103 based on information about the object that the user intends to photograph. The user selects the type of scene to be photographed, the weather, and other information, or inputs a keyword. The reference image processing unit 102 searches the image information stored in the image storage unit 103 based on this user input information, and selects a stored image suitable as the reference image. The selection of the reference image can be done by presenting the reference image candidates to the user and selecting the image that the user considers to be the most suitable, as in Embodiment 1, or by selecting the topmost image in the search results. It may also be automatically used as a reference image. Further, as in the first embodiment, a temporary photographed image may be photographed, and the reference image may be searched from the image storage unit 103 based on the temporary photographed image.

Through the above processing, even in the case of general photography, it is possible to select a reference image that is a reference for adjusting the photographing parameters of a photographed image. In the process after setting the reference image, similarly to the above-described embodiment, evaluation values of the captured image and the reference image are calculated, and imaging parameter adjustment is executed.
In the above description of the embodiment of general photography, an embodiment has been described in which the above-described configuration and the like are applied to general photography by changing the stored image in the image storage unit 103. As an embodiment in which the above-described configuration and the like are applied to general photography, a configuration may be adopted in which a reference image generation model is used to generate a reference image, as in the fourth embodiment.

<Other embodiments>
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium. It can also be realized by a process in which one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Although an example of the embodiment of the present invention has been described in detail above, the present invention is not limited to such specific embodiment.

As described above, according to the information processing apparatus 100 of each of the embodiments described above, it is possible to easily set shooting parameters for shooting a desired image even if details of the shot image cannot be confirmed.

10 CPUs
11 Storage unit 12 Operation unit 13 Communication unit 100 Information processing device

Claims (14)

a first acquisition means for acquiring a photographed image of a photographed subject using predetermined photographing parameters;
a second acquisition means for acquiring a plurality of reference images based on search conditions regarding image information of the photographed image;
Estimating means for estimating a first evaluation value for each of the reference images based on the captured image obtained by the first obtaining means and the reference image obtained by the second obtaining means; ,
has
The estimating means determines, based on the plurality of estimated first evaluation values, a photographing method suitable for photographing the photographing target , and a first evaluation value and an evaluation value larger than the first evaluation value. An information processing device that performs estimation using a model based on learning results of a relationship between a photographing method and a photographing parameter corresponding to the photographing method . The second acquisition means selects a plurality of reference image candidates from stored images stored in an image storage means that stores stored images serving as candidates for the reference image and image information of the stored images, based on the search condition. The information processing apparatus according to claim 1, wherein the reference image candidate is selected, the selected reference image candidate is displayed on an operation unit, and the reference image is selected based on a user operation via the operation unit. The information processing apparatus according to claim 1 or 2, wherein the first evaluation value is a degree of similarity between the photographed image and the reference image. The model is a learning model created by a set of the first evaluation value, a shooting parameter from which the first evaluation value is obtained, and a shooting parameter corresponding to an evaluation value larger than the first evaluation value. The information processing device according to any one of claims 1 to 3, wherein the information processing device is a model learned using data. The information processing apparatus according to any one of claims 1 to 4, further comprising a setting means for setting the photographing parameters in a photographing unit that photographs the photographed image. If the first evaluation value does not exceed a threshold even after adjusting the photographing parameters, the estimating means estimates any one of the photographing position and orientation, photographing time, and illumination conditions as the photographing method. 4. The information processing device according to 4. The estimating means displays the estimated photographing method on an operation section, and selects the estimated photographing method or re-executes the estimation of the photographing method based on a user operation via the operation section. The information processing apparatus according to any one of claims 1 to 6, wherein the information processing apparatus makes a determination. The second acquisition means acquires a plurality of reference images,
The information processing apparatus according to any one of claims 1 to 7, wherein the estimating means calculates a first evaluation value from the plurality of reference images and the photographed image. The photographic subject is a concrete wall of an infrastructure structure,
4. The image information includes at least one of the type of structure to be photographed, the type of concrete, the weather at the time of photographing, the target in the image, the installation location and region of the structure, and the number of years that have passed. Information processing device. The photographic subject is a concrete wall of an infrastructure structure,
2. The estimation means determines the first evaluation value by determining the degree of similarity between partial images of any one of cracks, joints, and formwork of the concrete wall surface included in the photographed image and the reference image. 10. The information processing device according to any one of 9. The estimating means estimates a second evaluation value, which is a statistical value, based on the plurality of estimated first evaluation values, and based on the estimated second evaluation value, a The information processing device according to claim 1, which estimates a photographing method suitable for photographing. The information processing apparatus according to claim 1 or 2, wherein the image information includes at least one of a photographing target and a photographing situation of the photographed image. An information processing method executed by an information processing device, the method comprising:
a first acquisition step of acquiring a photographed image of the photographic subject with predetermined photographing parameters;
a second acquisition step of acquiring a plurality of reference images based on search conditions regarding image information of the captured image;
an estimating step of estimating a first evaluation value for each of the reference images based on the photographed image obtained in the first obtaining step and the reference image obtained in the second obtaining step; ,
including;
In the estimation step, based on the plurality of estimated first evaluation values, a photographing method suitable for photographing the photographing target is determined based on a first evaluation value and an evaluation value larger than the first evaluation value. An information processing method that estimates using a model based on the learning results of the relationship between the corresponding imaging method and the imaging parameters . A program for causing a computer to function as each means of the information processing apparatus according to claim 1.

Images