JP2020088647A - Information processing device, information processing method, and program - Google Patents

Abstract

To easily set image-taking parameters for taking a desired image even if the detail of a taken image cannot be checked.SOLUTION: An evaluation value is obtained from a reference image and plural taken images obtained by taking images of an image-taking target with plural image-taking parameters and receiving from an image-taking unit for taking plural images, and an image-taking method suitable for taking images of the image-taking target is estimated on the basis of the evaluation value.SELECTED DRAWING: Figure 4

Description

The present invention relates to an information processing device, an information processing method, and a program.

When inspecting the concrete wall surface of structures such as bridges, dams, and tunnels, survey engineers are approaching the concrete wall surface and visually checking for deformations such as cracks. Since such an inspection work called close-up visual inspection has a high work cost, an image inspection for confirming the deformation by the image of the concrete wall surface has recently been performed.
Here, in order to shoot an image in which cracks such as thin cracks that are difficult to see are confirmed, it is necessary to set shooting parameters appropriately. The user must appropriately adjust shooting parameters such as focus and exposure according to the weather at the time of shooting and the shooting equipment, and shoot an image in which cracks can be confirmed. However, it is difficult to adjust the shooting parameters for shooting an image for confirming an unclear deformation such as a thin crack, and the deformation may or may not be confirmed due to a slight difference in the shooting parameters.
As a conventional example of the method of adjusting the shooting parameter, there is, for example, the method of Patent Document 1. In Patent Document 1, first, a plurality of images are photographed with a plurality of different photographing parameters, and the plurality of photographed images are displayed on a display. The user selects the image that is judged most preferable from the plurality of images. As a result, shooting parameters for shooting the selected image are set.

Japanese Patent No. 4534816

Chopra, Sumit, Raia Hadsell, and Yann LeCun. "Learning a similarity metric discriminatively, with application to face verification." Computer Vision and Pattern Recognition, 2005. CVPR 2005. IEEE Computer Society Conference on. Vol. 1. IEEE, 2005. Xie, Junyuan, Linli Xu, and Enhong Chen. "Image denoising and inpainting with deep neural networks." Advances in Neural Information Processing Systems. 2012. Dong, Chao, et al. "Image super-resolution using deep convolutional networks." IEEE transactions on pattern analysis and machine intelligence 38.2 (2016): 295-307. Goodfellow, Ian, et al. "Generative adversarial nets." Advances in neural information processing systems. 2014. Gatys, Leon A., Alexander S. Ecker, and Matthias Bethge. "Image style transfer using convolutional neural networks." Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition. 2016.

However, since imaging for structure inspection requires fine adjustment of imaging parameters, application of the method of Patent Document 1 causes display of a plurality of images with small changes between images. It is difficult for the user to compare images with such small changes and select the optimum image. Further, since the inspection image is taken outdoors, it is difficult to determine a subtle difference in the image due to the influence of external light and the available display size.

The present invention evaluates from an acquisition unit that acquires a reference image, the reference image, and the plurality of captured images that are captured from a capturing unit that captures a plurality of captured images by capturing a plurality of captured images of an object to be captured. Estimating means for obtaining a value and estimating a photographing method suitable for photographing the photographing target based on the evaluation value.

According to the present invention, it becomes possible to easily set shooting parameters for shooting a desired image without confirming the details of the shot image.

It is a figure which shows an example of the hardware constitutions of an information processing apparatus. It is a figure which shows an example of a structure of an information processing apparatus. It is a figure explaining the information stored in an image storage part. It is a flow chart which shows an example of information processing. It is a figure which shows an example of the screen at the time of image search. It is a figure explaining the setting of a photography parameter. It is a figure explaining the calculation method of the evaluation value based on the partial image of a crack position. It is a figure explaining evaluation of each photography parameter. It is a figure explaining the operation part at the time of performing photography parameter adjustment. It is a figure which shows an example of a structure of the information processing apparatus of Embodiment 4. It is a figure explaining the information stored in the image storage part of Embodiment 5.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<Embodiment 1>
In the first embodiment, description will be given by taking as an example the adjustment of imaging parameters in imaging for inspecting an image of an infrastructure structure. The infrastructure structure is, for example, a bridge, a dam, a tunnel, or the like, and in the image inspection, the concrete wall surface of these structures is photographed to create an image for the inspection. Therefore, in this embodiment, these concrete wall surfaces are objects to be photographed. The object of image inspection may be an image of the surface of a material other than concrete or other structures. For example, if the inspection target is a road, the asphalt surface may be the imaging target.
In the present embodiment, a reference image having an ideal image quality is prepared, and the shooting method is adjusted so that the image of the shooting target is similar to the reference image. The reference image is an image that can be clearly confirmed as an inspection target, such as a thin crack, which is difficult to photograph, among the concrete wall surface images photographed in the past. That is, the reference image is an image in which the focus, the brightness, the color tone, and the like are taken with the quality that is preferable as the inspection image. The main shooting method adjusted in the present embodiment is shooting parameters of the shooting unit, such as exposure, focus, white balance (color temperature), shutter speed, and aperture. Hereinafter, a method of adjusting the shooting method using the reference image will be described.

FIG. 1 is a diagram illustrating an example of a hardware configuration of the information processing device 100. The information processing apparatus 100 may be integrated with a photographing unit 101 of FIG. 2 to be described later and included in the housing of the camera. Alternatively, the image photographed by the photographing unit 101 may be wirelessly or wired transmitted to the photographing unit 101. The camera may include a housing (for example, a computer or a tablet) different from the camera. In the example of FIG. 1, the information processing device 100 includes a CPU 10, a storage unit 11, an operation unit 12, and a communication unit 13 as a hardware configuration. The CPU 10 controls the entire information processing device 100. When the CPU 10 executes the process based on the program stored in the storage unit 11, the configuration shown by 102, 104, 105 in FIG. 2, which will be described later, and the process of the flowchart of FIG. 4, which will be described later, are realized. .. The storage unit 11 stores a program, data used when the CPU 10 executes processing based on the program, an image, and the like. The operation unit 12 displays the result of the processing of the CPU 10 and inputs a user operation to the CPU 10. The operation unit 12 can be configured by a display and a touch panel on the back of the camera, or a display and an interface of a notebook PC. The communication unit 13 connects the information processing device 100 to a network and controls communication with other devices and the like.

FIG. 2 is a diagram illustrating an example of the configuration of the information processing device 100 according to the first embodiment. The information processing apparatus 100 includes a photographing unit 101, a reference image processing unit 102, an image storage unit 103, an estimation unit 104, and a photographing parameter setting unit 105 as a configuration. However, as described above, the image capturing unit may or may not be included in the information processing device 100. The reference image processing unit 102, the estimation unit 104, and the shooting parameter setting unit 105 are software. The image storage unit 103 may be provided in the storage unit 11 or a storage server that can communicate with the information processing apparatus 100. When the image storage unit 103 is provided in the storage server, the reference image processing unit 102 acquires the image stored in the image storage unit 103 and the information related to the image via the network. The image storage unit 103 is a storage that stores a group of images that are candidates for reference images.

FIG. 3 is a diagram illustrating information stored in the image storage unit 103. First, the image storage unit 103 stores a plurality of images (images 201 and 202 in FIG. 3). Hereinafter, the images 201 and 202 stored in the image storage unit 103 will be referred to as stored images. The stored image is an image prepared by collecting images taken with image quality suitable for image inspection from images taken of concrete wall surfaces of various structures. The image quality preferable for the image inspection is an image quality in which a deformation such as a crack can be easily confirmed when a person confirms the image, and, for example, an image in which the focus, the brightness, and the hue are preferable. For example, the stored image 201 is an image in which the crack 211 can be clearly confirmed. Further, the stored image is not limited to the one in which cracks are shown. The stored image 202 is an image showing the joint 212 of the concrete wall surface. Since the edge of the joint 212 is clearly reflected in the stored image 202, it is determined that the image has a preferable image quality for inspection.

Further, when inspecting a captured image using a technique for automatically detecting cracks in the captured image, the image quality at which automatic detection preferably operates may be the image quality preferable for image inspection. In this case, the correct answer rate and the like of the detection result of the automatic detection is calculated, and an image having a high image quality is used as the stored image.
In the image storage unit 103 of FIG. 3, the stored image is recorded in association with the image information and the shooting parameter. The image information is information related to the shooting content of the stored image, and includes, for example, the type of structure of the object, the type of concrete, the weather at the time of shooting, the target in the image, the installation location/region of the structure, the number of years elapsed, etc. Be done. The shooting parameters are shooting parameters when the respective reference images are shot.

FIG. 4 is a flowchart showing an example of information processing. Hereinafter, the operation of the information processing apparatus 100 will be described according to the flowchart.
S301 and S302 are processes executed by the reference image processing unit 102. The reference image processing unit 102 of the first embodiment executes a process of selecting a reference image from the stored images stored in the image storage unit 103. FIG. 5 shows information displayed on the operation unit 12 when executing S301 and S302.
In S301, the reference image processing unit 102 searches the image storage unit 103 for a reference image candidate based on the search condition. As a reference image candidate search method, there is a method using image information. As shown in FIG. 3, image information is associated with the image stored in the image storage unit 103. The reference image processing unit 102 can search for a stored image similar to the shooting target based on this information. FIG. 5 shows an example of a screen for searching the stored image on the operation unit 12. For example, suppose that the shooting target is a bridge floor slab and the weather at the time of shooting was cloudy. The user sets the conditions related to such a shooting target or shooting condition as the image search condition. Then, by selecting the search button 410, the stored image corresponding to the search condition can be searched from the image storage unit 103. The search result is displayed in the reference image candidate display field 420 as a reference image candidate. As the reference image candidates, only the stored images in which the search conditions and the image information match may be used as the reference image candidates, or a predetermined number of stored images having a high degree of item matching may be selected as the reference image candidates. Further, in the example of FIG. 5, the image information for the search displays only the structure type, the concrete type, and the weather, but the conditions for the image search are not limited to these. Furthermore, although FIG. 5 shows a method of setting the search content by a pull-down menu as the search method, the method of the user inputting image information for the search is not limited to this. For example, the operation method may be such that the stored image can be searched by inputting a free character string as a keyword.

Further, as another search method of reference image candidates, there is a method of using a tentative captured image. In this case, first, the user shoots a shooting target with the shooting unit 101. This shooting is tentative shooting, and automatic setting or the like is used as the shooting parameter at this time. The image photographed by this temporary photographing is referred to as a temporary photographed image. The user sets the tentative captured image as a search key for selecting a reference image candidate. FIG. 5 shows that the tentative captured image 450 is set as the search condition for selecting the reference image candidate. In this state, by selecting the search button 410, the image storage unit 103 is searched for an image similar to the tentative captured image. As a result of the search, the upper stored image having a high degree of similarity is selected as the reference image candidate and displayed in the reference image candidate display field 420. Here, in the search using the tentatively photographed image, for example, the degree of similarity with the stored image is calculated based on the characteristics of the entire image (color tone or texture of the concrete wall surface), and the reference image candidate is selected. As a result, it becomes possible to search for stored images in which the concrete wall surface to be photographed for inspection is similar. Further, the reference image candidates may be searched by simultaneously using the above-described search by the image information (keyword) and the search by the tentative captured image.
As described above, the reference image candidate is selected and displayed in the reference image candidate display field 420.

In S302 of FIG. 4, the reference image processing unit 102 selects one image as a reference image from the reference image candidates displayed in the reference image candidate display field 420. First, as an initial value of reference image selection, a reference image candidate with the highest degree of matching in search is automatically selected as a reference image. FIG. 5 shows a state in which the reference image 430 thus selected is displayed in the reference image display field. The user can check the reference for adjusting the shooting method by checking the reference image displayed in this manner. If the user determines that the selected reference image 430 is not suitable as the reference for adjustment, the user can select another image from the reference image candidates as the reference image. When another image is selected from the reference image candidates, the reference image processing unit 102 sets the selected image as the reference image.
Note that cracks (for example, 440 and 441) are reflected in the image of FIG. As will be described later, when the evaluation value is calculated using the image of the crack portion, the reference image needs to include the crack. In addition, by setting an image including the crack 440 in the temporary captured image 450, it is possible to search for a stored image including a crack similar to the crack to be captured in the reference image candidate search.

In S303, the shooting parameter setting unit 105 determines initial values of shooting parameters (hereinafter, initial shooting parameters). To set the initial shooting parameters, the shooting parameters determined by the normal shooting parameter adjustment method (automatic parameter adjustment) of the shooting apparatus may be set as the initial parameters. Further, as another method, the shooting parameter associated with the reference image may be used as the initial parameter. As shown in FIG. 3, the image storage unit 103 records, for each stored image, shooting parameters at the time of shooting the image. Therefore, when the shooting parameter associated with the reference image is used as the initial parameter, the reference image processing unit 102 calls the shooting parameter associated with the image selected as the reference image from the image storage unit 103 to set the initial parameter. Set as.

In S304, the shooting parameter setting unit 105 sets a plurality of shooting parameters based on the initial shooting parameters. FIG. 6 shows how a plurality of shooting parameters are set based on the initial shooting parameters. First, FIG. 6A is a diagram illustrating an embodiment in which the exposure (EV) is adjusted as an example of a shooting parameter adjusted by the method of the present embodiment. In FIG. 6A, a white triangle 501 indicates that EV0 is set as the initial parameter. The shooting parameter setting unit 105 sets a plurality of shooting parameters centered on the initial parameters. In FIG. 6A, the shooting parameter setting unit 105 changes the exposure by one step centering on EV0, and sets a plurality of EV-1 (black triangle 502 in FIG. 6) and EV+1 (black triangle 503 in FIG. 6). It is set as a parameter. In this example, three shooting parameters are set together with the initial shooting parameters, but the number of shooting parameters to be set is not limited to this. For example, the shooting parameter setting unit 105 may set two different exposures and set a total of five shooting parameters. Further, in this example, a plurality of shooting parameters are set according to the rule that the exposure is changed by one step, but the steps of changing the shooting parameters may be set by other methods. For example, the shooting parameter setting unit 105 may set the exposure in half steps, or may set the exposure randomly around the initial shooting parameters.

In the above, the embodiment has been described in which the shooting parameter is the exposure, but the shooting parameter set in this embodiment is not limited to the exposure. Any shooting parameter may be used as long as it is a parameter for controlling the shooting unit 101. For example, focus, white balance (color temperature), shutter speed, aperture, ISO sensitivity, image saturation, tint, etc. May be used as the shooting parameter.
Further, in FIG. 6A, the embodiment in which only the exposure is used as the shooting parameter to be adjusted in the present embodiment has been described, but a plurality of shooting parameters may be simultaneously adjusted. For example, FIG. 6B is a diagram illustrating an embodiment in which a combination of exposure and focus is used as a shooting parameter for adjusting. In FIG. 6B, a combination of a certain parameter of exposure and focus is set as an initial parameter and is shown as a white circle 511. The shooting parameter setting unit 105 may set a combination of shooting parameters, such as a black circle 512, as a plurality of shooting parameters centering on the initial parameters.

Note that the combination of the shooting parameters to be adjusted is not limited to the combination of the exposure and focus shown in FIG. 6B, and may be a combination of other shooting parameters. Further, in the above description, the embodiment in which the combination of two parameters is adjusted has been described, but the number of combinations of shooting parameters is not limited to this, and combinations of three or more shooting parameters may be adjusted at the same time. ..
In S304, as described above, the shooting parameter setting unit 105 sets a plurality of shooting parameters. Regarding the subsequent processing, a case will be described in which the shooting parameter to be adjusted is exposure as shown in FIG.

In step S305 of FIG. 4, the image capturing unit 101 captures an image of the image capturing target using the plurality of image capturing parameters set in step S304. More specifically, when three exposures are set as a plurality of shooting parameters as shown in FIG. 6A, the shooting unit 101 changes the exposure according to the user's shutter operation and changes the exposure to three. Take an image automatically. Hereinafter, the image captured in this step is referred to as a captured image.

The process from S306 onward in FIG. 4 is a process mainly executed by the estimation unit 104, and is a process for selecting the optimum shooting parameter or a process for further searching for the optimum shooting parameter.
In S306, the estimation unit 104 calculates an evaluation value for each of the plurality of shooting parameters. The evaluation value indicates a higher value as the shooting parameter is more appropriate for shooting the inspection image. The estimation unit 104 calculates this evaluation value by comparing the captured image captured with each capturing parameter with the reference image. More specifically, when the captured image is similar to the reference image, the capturing parameter for capturing the captured image can be determined to be a preferable parameter. Therefore, in such a case, the estimation unit 104 is configured to calculate a high evaluation value. In order to calculate this evaluation value, the degree of similarity between the photographed image and the reference image may be calculated. Hereinafter, a specific example of the method of calculating the evaluation value will be described.

As one example of the method of calculating the evaluation value between the captured image and the reference image, first, a method of calculating the similarity of the entire image and using it as the evaluation value will be described. For example, when comparing the similarity of the entire image with the brightness of the entire image, after the grayscale conversion between the captured image and the reference image, a luminance histogram of the entire image is created, and the luminance histogram of the captured image and the reference image are compared. It is only necessary to calculate the degree of similarity with the luminance histogram of. The similarity of the histogram can be calculated by a method of simply calculating the Euclidean distance or a method such as Histogram Intersectio. Further, when calculating the similarity of the hue of the entire image, the color histogram of each image is created based on the color space such as RGB or YCrCb without performing the grayscale conversion, and the similarity of the color histogram is calculated. Should be calculated. The feature amount for determining the similarity of the entire image is not limited to these histogram feature amounts, and other feature amounts may be used.

As another example of the evaluation value calculation method, the partial similarity of images may be calculated. For example, when it is desired to take an inspection image of a concrete wall surface, the portion of interest is the portion in which the concrete wall surface is shown. Therefore, when a portion other than the concrete wall surface is shown in the captured image and the reference image, the similarity may be calculated based on the image of the portion of the concrete wall surface excluding the portion. More specifically, for example, when the floor slab of the bridge is photographed from below the bridge, the photographed image may include a sky region (background portion). In such a captured image, the estimation unit 104 removes the sky region and calculates the evaluation value by calculating the similarity between the image portion of the concrete wall surface of the floor slab and the reference image. In the calculation of the similarity, the above-described histogram feature amount may be created for each of the partial image of the captured image and the entire reference image, and the similarity between the histogram feature amounts may be calculated. This example is an example of calculating the degree of similarity between the partial image on the captured image side and the reference image on the assumption that the entire reference image is a concrete wall surface image. However, when a part of the reference image includes a part that can be regarded as the background, the similarity between the partial image of the reference image and the captured image may be calculated.

Further, another method of calculating the partial similarity of images will be described. When taking images of concrete wall surfaces for image inspection, it is important to take images with the image quality that allows fine cracks to be confirmed in the taken images. Assuming that the reference image is an ideal inspection image in which thin cracks can be sufficiently confirmed, it is preferable that the thin crack portion of the photographed image be photographed in the same manner as the thin crack portion of the reference image. In order to determine whether the cracked portion is similarly photographed, the estimation unit 104 calculates the evaluation value using the partial image of the cracked portion in the image. Any method may be used to calculate the evaluation value of the image of the cracked portion, but in the following example, a higher evaluation value is calculated as the edge strength of the cracked portion is similar. For this purpose, first, the estimation unit 104 identifies the cracked portion between the captured image and the reference image. The method of identifying the cracked portion may be performed automatically or manually by the user. When automatically detecting a crack, the estimation unit 104 uses the automatic crack detection process. When manually specifying the crack position, the estimation unit 104 receives an input of the crack position in the image from the user via the operation unit 12. With respect to the captured image, it is necessary to specify the crack position after shooting by these processes, but the crack position of the reference image is specified in advance and recorded in the image storage unit 103 in association with the stored image. You can leave it. As described above, if the crack positions of the captured image and the reference image are obtained, the estimation unit 104 calculates the edge strength of the image at each crack position. The edge strength may be simply a luminance value at the crack position, or the gradient at the crack position may be calculated by a Sobel filter or the like, and the gradient strength may be used as the edge strength. Since these edge intensities are obtained on a pixel-by-pixel basis, in order to calculate the similarity of the edge intensities of the captured image and the reference image, it is necessary to create the edge intensity feature amount of the entire image. For this purpose, for example, the estimation unit 104 may create a histogram feature amount by histogramming the edge strength at the crack position in each image. The estimation unit 104 calculates the similarity of the edge strength histogram feature amount between the captured image and the reference image, and the higher the similarity, the higher the evaluation value between the captured image and the reference image.

As described above, in order to calculate the evaluation value based on the edge strength of the cracked portion, it is premised that both the photographed image and the reference image include cracks. With regard to the photographed image, the user can obtain a photographed image including a crack by photographing the portion where the crack exists from the concrete wall surface to be photographed. On the other hand, regarding the reference image, in the step of selecting the reference image in S301 to S302, the user searches and stores the stored image stored in the image storage unit 103 so that the image including the crack becomes the reference image. Select and set.
In the above, the evaluation values of the photographed image and the reference image are calculated based on the edge strength at the crack position, but the concrete wall surface of the photographed object does not always have a crack. Therefore, the estimation unit 104 may calculate the evaluation value based on the edge strength of the image edge portion that surely appears in the structure of concrete such as the joint of the concrete or the trace of the mold. In this case, except for using the concrete joints included in the captured image and the reference image, the edge strength of the portion of the mold trace, the evaluation value is calculated by the same method as the evaluation value calculation method using the edge of the cracked portion described above. It can be calculated.

As described above, in order to calculate the evaluation value based on the edge strength of the concrete joint, it is premised that both the photographed image and the reference image include the concrete joint. With regard to the photographed image, the user can obtain a photographed image including cracks by photographing the portion where concrete joints are present from the concrete wall surface of the photographing target. On the other hand, regarding the reference image, in the step of selecting the reference image in S301 to S302, the user searches the stored images stored in the image storage unit 103 so that the image including the concrete joint becomes the reference image. Also, set by selecting.

Further, as a modification of the evaluation value calculation using the edge strength at the crack position, the estimation unit 104 may use the information about the crack width to calculate the evaluation value of the edge strength between the captured image and the reference image. .. In this method, the estimation unit 104 calculates a higher evaluation value as the edge strengths of the cracks having the same width in the captured image and the reference image are similar to each other. FIG. 7 is a diagram illustrating a method of calculating an evaluation value based on a partial image of a crack position by using the information on the crack width. First, FIG. 7A is an example of a photographed image 620 photographed with a certain photographing parameter, and is an image in which cracks 600 on a concrete wall surface are photographed. The crack 600 is a crack having various crack widths depending on the part in one crack. In FIG. 7A, it is assumed that a local crack width can be measured for this crack 600. For example, FIG. 7(A) shows places where the crack width is obvious, such as 0.15 mm and 0.50 mm. For these crack widths, the user measures the actual crack width of the concrete wall surface and inputs it through the operation unit 12 during image capturing. Alternatively, the user may check the captured image, estimate the crack width, and input the image via the operation unit 12 during capturing. The CPU 10 stores the input crack width in the image storage unit 103 in association with the captured image. On the other hand, FIG. 7B is an example of the reference image 621, which is an image of the concrete wall surface in which the crack 610 is reflected. Similarly to the crack 600, the crack 610 is also a crack having various crack widths depending on the part in one crack. A local crack width is also recorded for the crack 600, and for example, crack widths of 0.10 mm, 0.50 mm, etc. are recorded in FIG. 7B. The crack width information of these reference images is stored in the image storage unit 103 and is called together with the reference image 621 from the image storage unit 103.

ここで、ｄ_iは、あるひび割れ幅（例えば、０．１０ｍｍ幅のひび割れ）の画像部分の類似度である。α_iは、あるひび割れ幅の評価値への重みである。α_iは、例えば、細いひび割れ幅に大きな重みを付けるようにする。このようにすることで、撮影画像の細いひび割れ部分が、基準画像の画質に一致するほど高い評価値が算出される。したがって、細いひび割れ部分が基準画像の画質に近づくことを重視して、撮影条件を調整することができるようになる。
なお、ひび割れ部分の画像を用いて評価値を算出する場合には、撮影画像と基準画像とのコンクリート壁面の撮影解像度は、同解像度とすることが好ましい。より具体的には、撮影画像と基準画像とに写るコンクリート壁面が、例えば１．０ｍｍ／画素となる解像度になるように調整する処理を予め実施する。これは、解像度によって、同じひび割れでもエッジ強度等の見た目が変化するためである。また、画像中のコンクリート壁面が正対するように、あおり補正を予め実施することも好適な実施形態である。 In the evaluation value calculation of the captured image 620 and the reference image 621 of FIG. 7, the estimation unit 104 compares the edge strengths of cracked portions having the same crack width. For example, the estimation unit 104 calculates the degree of similarity based on the edge strength between the partial image 601 of the captured image 620 and the partial image 611 of the reference image 621 as the portion having the crack width of 0.50 mm. Further, the estimation unit 104 calculates the degree of similarity based on the edge strength between the partial image 602 of the captured image 620 and the partial image 612 of the reference image 621 as the portion having the crack width of 0.10 mm. In this way, the evaluation value s between the captured image 620 and the reference image 621 is calculated by the following formula based on the similarity between the image portions having the same crack width.

Here, d _i is the degree of similarity of an image portion having a certain crack width (for example, a crack having a width of 0.10 mm). α _i is a weight for the evaluation value of a certain crack width. For α _i , for example, a large weight is given to a narrow crack width. By doing so, a higher evaluation value is calculated as the fine cracked portion of the captured image matches the image quality of the reference image. Therefore, it becomes possible to adjust the shooting conditions, placing importance on the fact that the thin cracked portion approaches the image quality of the reference image.
In addition, when the evaluation value is calculated using the image of the cracked portion, it is preferable that the photographing resolutions of the concrete wall surface of the photographed image and the reference image are the same. More specifically, a process of adjusting the concrete wall surfaces reflected in the photographed image and the reference image to have a resolution of, for example, 1.0 mm/pixel is performed in advance. This is because the appearance such as edge strength changes depending on the resolution even with the same crack. Further, it is also a preferable embodiment to carry out the tilt correction in advance so that the concrete wall surface in the image faces directly.

ここで、ｘ_nは任意の基準画像である。ｙ_nは任意の撮影画像である。ｔ_nは、ｘ_nとｙ_nとが類似画像と見なせる場合には１をとり、類似画像と見なせない場合には０をとる教師データである。このデータセットを用いて学習を行う学習方法はどのようなものを用いてもよいが、例えば、ＣＮＮ（Ｃｏｎｖｏｌｕｔｉｏｎａｌ Ｎｅｕｒａｌ Ｎｅｔｗｏｒｋ）を用いた学習方法の例として、非特許文献１のような方法がある。非特許文献１に記載の方法でデータセットＤを学習したモデルは、評価値を算出したい撮影画像と基準画像とをモデルに入力すると、評価値を算出することができる。
以上では、撮影画像と基準画像との評価値の算出方法について様々な手法を説明した。説明した方法以外にも、画像と画像との類似度の算出方法については、従来、様々な方法が提案されている。推定部１０４は、これら公知の手法を用いて本実施形態の評価値を算出してもよい。 The embodiment has been described above in which the image feature amount is created for each of the captured image and the reference image, the similarity between the images is calculated based on the distance of the feature amount, and the calculated similarity is used as the evaluation value. The method of calculating the image similarity is not limited to this, and the evaluation value may be calculated using a learning model learned in advance. In this method, a model that outputs a higher evaluation value as the input image and the reference image are similar to each other is learned in advance. This learning can be learned using the following data set D, for example.

Here, x _n is an arbitrary reference image. y _n is an arbitrary captured image. t _n is teacher data that takes 1 when x _n and y _n can be regarded as similar images, and takes 0 when they cannot be regarded as similar images. Although any learning method for performing learning using this data set may be used, as an example of a learning method using CNN (Convolutional Neural Network), there is a method as described in Non-Patent Document 1. .. The model in which the data set D is learned by the method described in Non-Patent Document 1 can calculate the evaluation value by inputting the captured image and the reference image for which the evaluation value is to be calculated into the model.
In the above, various methods have been described as the method of calculating the evaluation value of the captured image and the reference image. In addition to the method described above, various methods have been conventionally proposed as a method of calculating the similarity between images. The estimation unit 104 may calculate the evaluation value of this embodiment using these known methods.

ここで、ｓ_jは、ある方法により求めた評価値である。ｗ_jはその方法の重みである。
Ｓ３０６では、以上のような方法で、推定部１０４は、撮影画像と基準画像との評価値を算出する。また、Ｓ３０６では、推定部１０４は、複数の撮影パラメータで撮影した撮影画像それぞれについて評価値を算出する。 Further, although a plurality of methods for calculating the evaluation value have been described above, each of these evaluation value calculating methods may be used alone, or a plurality of methods may be used in combination. When combining a plurality of methods, the estimation unit 104 calculates the final evaluation value s by the following formula, for example.

Here, s _j is an evaluation value obtained by a certain method. w _j is the weight of the method.
In S306, the estimation unit 104 calculates the evaluation values of the captured image and the reference image by the above method. Further, in S306, the estimation unit 104 calculates an evaluation value for each of the photographed images photographed with the plurality of photographing parameters.

In S307, the estimation unit 104 evaluates the shooting parameter based on the evaluation value calculated in S306.
In S308, the estimation unit 104 determines whether or not to readjust the imaging parameter based on the evaluation result.
When readjusting the imaging parameter, the estimation unit 104 estimates a method for improving the shadow parameter in S309. Then, the estimation unit 104 returns to the process of capturing a plurality of images in S305.
When the read parameters are not readjusted, the shooting parameter setting unit 105 sets the shooting parameters in the shooting unit 101 in step S310. Then, the processing of the flowchart shown in FIG. 4 ends.
Hereinafter, these processes will be described.

First, in the imaging parameter evaluation of S307, the estimation unit 104 selects the maximum evaluation value from the evaluation values of the plurality of imaging parameters and compares it with a predetermined threshold value. FIG. 8A is a diagram illustrating the evaluation of each shooting parameter. In this embodiment, three exposures (EV) are set as a plurality of shooting parameters. In FIG. 8(A), a state in which exposures -1, 0, +1 are set as a plurality of shooting parameters is represented by triangles 501, 502, 503, as in FIG. 6(A). The lower part of FIG. 8A shows evaluation values s _-1 , s ₀ , and s ₊₁ obtained from the photographed image photographed with each photographing parameter and the reference image. In FIG. 8A, the evaluation value s ₊₁ of the exposure 503 of ₊₁ is the highest evaluation value, and indicates a value exceeding the predetermined threshold value s _th . When there is a shooting parameter indicating an evaluation value that exceeds the predetermined threshold value s _th , the estimation unit 104 determines that the shooting parameter is a shooting parameter suitable as the shooting parameter of the inspection image. In the case of FIG. 8A, the estimation unit 104 selects the exposure 503 of +1 as the optimum parameter. Then, in S308, the estimation unit 104 determines that the readjustment of the shooting parameter is not necessary, and proceeds to S310 of setting the shooting parameter. In S310, the shooting parameter setting unit 105 sets the exposure of the shooting unit 101 to +1 and ends the processing illustrated in FIG.

Similar to FIG. 8A, FIG. 8B is an example in which exposures -1, 0, and +1 are set as a plurality of shooting parameters and evaluation values are calculated, but different from FIG. 8A. The situation where the evaluation value is obtained is shown. In FIG. 8B, the maximum evaluation value is shown as the evaluation value s ₊₁ but even s ₊₁ does not exceed the predetermined threshold value s _th . The photographed image photographed with these photographing parameters has a low degree of similarity to the reference image, and these photographing parameters are not suitable as the photographing parameters of the inspection image. In this case, in S308, the estimation unit 104 determines that readjustment of the imaging parameter is necessary, and proceeds to S309. In S309, the estimation unit 104 estimates a method for improving the shooting parameter.

The estimation of the imaging parameter improvement method will be described with reference to FIG. In FIG. 8B, the evaluation value s ₊₁ of the exposure of ₊₁ is less than the threshold value s _th , but the maximum evaluation value is shown among the evaluation values s _{−1 to} s ₊₁ . Therefore, in the readjustment of the shooting parameters, the estimation unit 104 sets a plurality of shooting parameters from the shooting parameters around this shooting parameter (+1 exposure). For example, in the case of setting three shooting parameters even in the next shooting parameter adjustment, the estimation unit 104 sets the exposures 721, 722, and 723 around the +1 exposure 503 as a plurality of parameters as illustrated in FIG. 8B. To do. Then, the process returns to S305, sets these shooting parameters in the shooting unit 101 via the shooting parameter setting unit 105, and shoots a plurality of images again. Then, the estimation unit 104 again executes the processing (evaluation value calculation processing) after S306 in FIG. 4 to search for the optimum imaging parameter. If the evaluation value equal to or _larger than the threshold value s _th is not obtained even in the evaluation of this imaging parameter set, the estimation unit 104 again determines a plurality of new imaging parameters around the imaging parameter having the maximum evaluation value. Then, the photographing process is executed again. This loop is repeatedly executed until the imaging parameter for which the evaluation value exceeding the threshold value s _th is obtained is determined. The maximum number of repetitions may be determined in advance, and if the optimum shooting parameter (shooting parameter for which an evaluation value equal to or greater than the threshold value s _th is obtained) cannot be obtained by then, the processing may be terminated. When the shooting parameter adjustment processing is terminated, the estimation unit 104 displays a warning on the operation unit 12 to notify the user that the shooting parameters have not been sufficiently adjusted. Further, the shooting parameter for shooting the image for which the maximum evaluation value obtained until the processing is terminated may be set in the shooting unit 101 via the shooting parameter setting unit 105.

In the above description, the embodiment in which the improved imaging parameter is estimated and the repeated adjustment is performed only when the evaluation value equal to or _larger than the threshold value s _th is not obtained in S307 has been described. However, even if a shooting parameter having an evaluation value equal to or higher than the threshold value s _th is found, a shooting parameter having a higher evaluation value may be searched for. In this case, the information processing apparatus 100 sets a shooting parameter around the shooting parameter that shows the maximum evaluation value as an improved shooting parameter, shoots a plurality of images again, and repeatedly executes the evaluation value calculation process. The termination condition of this iterative process is when the predetermined number of iterations is reached, or when the evaluation value does not change even if the imaging parameter is changed near the maximum evaluation value.

On the other hand, the user may check the shooting parameters and determine the end of the repeated process for adjusting the shooting parameters. In this case, in step S308 of FIG. 4, the estimation unit 104 does not perform the optimum shooting parameter determination using the threshold value s _{th of the} evaluation value, but determines whether to perform readjustment of the shooting parameter as a user operation. Judge based on Therefore, the estimation unit 104 presents necessary information to the user on the operation unit 12, and further receives an input from the user via the operation unit 12. FIG. 9 is a diagram illustrating the operation unit 12 in the case of adjusting the shooting parameters according to the user's judgment. The information presented to the user and the user's operation will be described below with reference to FIG. 9.
First, the operation unit 12 in FIG. 9 is a display for displaying information. An image 801 on the screen displayed on the operation unit 12 is a photographed image photographed with a photographing parameter of exposure+1, and photographed images 802 and 803 are photographed images photographed with other photographing parameters. The reference image 804 is also displayed on the display, and the user can check the captured image and the reference image by comparing them.
Further, in the lower part of the image 801, a plurality of shooting parameters set for adjusting the shooting parameters are shown. In FIG. 9, as an example of a plurality of shooting parameters, three-step exposure (EV) is shown by a black triangle. Among these, the black triangle 811 indicating the exposure +1 showing the maximum evaluation value is highlighted (largely displayed). Further, a white triangle 812 and the like indicate a plurality of shooting parameter candidates for further adjustment of the shooting parameters, which are set based on the shooting parameter 811 of exposure+1.

In this embodiment, the user confirms these pieces of information displayed on the operation unit 12 and determines whether to adopt the current shooting parameter or to execute the shooting parameter adjustment process. More specifically, the user compares the photographed image with the reference image in the image 801 for which the maximum evaluation value is obtained, and if the degree of matching is satisfactory, the user sets the photographing parameter showing the maximum evaluation value. It can be judged to be adopted. When adopting the shooting parameter indicating the maximum evaluation value, the user selects the icon 821 displayed as “set”. By this operation, the shooting parameter setting unit 105 sets the shooting parameter indicating the maximum evaluation value in the shooting unit 101 (S310 in FIG. 4), and the shooting parameter adjustment process ends. On the other hand, if the user is not satisfied with the current optimum shooting parameters by checking the image 801 or the like, the user selects the icon 822 displayed as “readjustment”. By this instruction, the processing (evaluation value calculation processing) after S306 in FIG. 4 is executed again using the next plurality of shooting parameters (for example, exposure 812 and the like). After that, the information processing apparatus 100 again presents various kinds of information to the user as shown in FIG. 9. Based on the presented information, the user decides whether to adopt the shooting parameter again or further adjust the shooting parameter.

If the adjustment of the shooting parameters is stopped on the way, the icon 823 displayed as “End” is selected. By this operation, the information processing apparatus 100 can end the process of adjusting the shooting parameters (loop of the flowchart of FIG. 4). At this time, the information processing apparatus 100 may set, in the image capturing unit 101, the image capturing parameter having the maximum evaluation value among the image capturing parameters that have been captured and evaluated so far.
Even when it is determined by the user operation that the shooting parameter adjustment is to be continued, the threshold value s _{th of the} evaluation value is set in advance, and it is displayed that the shooting parameter having the evaluation value exceeding the threshold value s _th exists. You can For example, in FIG. 9, when the evaluation value s _{811 of the} image shot with the shooting parameter 811 exceeds the threshold s _th , the information processing apparatus 100 may blink the black triangle 811 indicating the shooting parameter. .. Although the user can adopt the shooting parameter regardless of the evaluation value, the information processing apparatus 100 displays the existence of the shooting parameter exceeding the evaluation value in this way to assist the determination of the shooting parameter selection. Will be able to.

As described above, in the first embodiment, the embodiment in which the method for improving the shooting parameter is estimated has been described. However, the shooting method estimated by the method of the present embodiment is not limited to the shooting parameter, and other shooting methods may be estimated. Good. In the embodiment in which the imaging method other than the imaging parameters is estimated, when the evaluation value equal to or higher than the predetermined threshold is not obtained even if the loop of the processing flow of FIG. We analyze shooting conditions and propose appropriate shooting methods. For example, when it is determined that the brightness of the image is insufficient, or when it is determined that the adjustment of the white balance cannot be adjusted by the shooting parameter, the estimation unit 104 uses the illumination for the user to illuminate. The condition may be changed, or the shooting time may be changed to give a notification recommending shooting at a brighter time. As another example, when the position and orientation of the imaging unit 101 can be acquired, the estimation unit 104 analyzes the positional relationship with the structure to be inspected and proposes a position and orientation that improves imaging. More specifically, for example, when an image is taken at a position and orientation with a large tilt angle with respect to the wall surface of the structure to be inspected, the estimation unit 104 recommends to the user to take an image at a position and attitude that reduces tilt. ..

<Embodiment 2>
In the first embodiment, one image is selected from the image storage unit 103 and used as the reference image. Then, the embodiment in which the shooting method is adjusted based on the selected one reference image has been described. In the second embodiment, an embodiment in which the shooting method is adjusted using a plurality of reference images will be described. It should be noted that in the following embodiments, the parts that are different from the first embodiment will be mainly described.
First, in the second embodiment, the reference image processing unit 102 selects a plurality of reference images. In the following, it is assumed that M reference images are selected by the reference image processing unit 102. The M reference images may be selected by any method. For example, the upper M stored images of the search result may be used as the M reference images.

以降の処理は、ＣＰＵ１０が、評価値ｓを基に、撮影パラメータの調整を実施する（実施形態１の図４Ｓ３０７以降の処理を実施する）。評価値ｓ^mの平均を用いることにより、複数の基準画像と全体的に類似する画像が撮影されるように、撮影パラメータが調整される。 Next, the estimation unit 104 calculates evaluation values for the captured image and the M reference images. In this process, first, the evaluation values of the captured image and each reference image are calculated. For example, the estimation unit 104 calculates the evaluation value of the captured image and the m-th reference image and the evaluation value s ^m. The method of calculating the evaluation value of the captured image and the reference image is the same as the method of the first embodiment. When M evaluation values are obtained by calculating the evaluation values of the captured image and the M reference images, the estimation unit 104 calculates the final evaluation value s by averaging the evaluation values.

In the subsequent processing, the CPU 10 adjusts the imaging parameter based on the evaluation value s (executes the processing from S307 in FIG. 4 of the first embodiment). By using the average of the evaluation value s ^m, as images similar plurality of the overall reference image is captured, the imaging parameter is adjusted.

この方法では、Ｍ枚の基準画像のうち、いずれかに類似するように、撮影パラメータが調整される。Ｍ枚の基準画像は、それぞれ好ましい画質の画像であるから、撮影画像は、そのいずれかに類似していればよい。 Further, as another form using a plurality of reference images, there is a method of adjusting a shooting parameter based on an evaluation value of one reference image that is most similar among a plurality of reference images. In this case, the final evaluation value s of the captured image and the M reference images is obtained by the following formula.

In this method, the shooting parameters are adjusted so as to resemble any of the M reference images. Since each of the M reference images is an image having a preferable image quality, the captured image may be similar to any of them.

<Embodiment 3>
In the above embodiments, the reference image is assumed to be an image captured by a structure different from the target structure, but a past image of the target structure may be used as the reference image. In the infrastructure inspection, the past inspection result and the latest inspection result are compared with each other. For this comparison, it is preferable that the past image and the latest image are captured with the same image quality. When there is a past image of the structure to be photographed, by using this as a reference image, the photographing parameters can be adjusted so that an image similar to the past image can be photographed.

In order to use the past image as the reference image, the image storage unit 103 of the third embodiment stores the past image of the structure to be photographed. The reference image processing unit 102 performs a process of acquiring a past image from the image storage unit 103 and setting it as a reference image based on the information of the structure of the imaging target. Therefore, the reference image processing unit 102 according to the third embodiment may be able to search the stored image in the image storage unit 103 by using unique information such as the name of the structure. The processing after the setting of the past image of the structure to be captured as the reference image can be performed by performing the same processing as that of the first embodiment to adjust the imaging method. With the above configuration, the shooting parameters can be adjusted so that a shooting result similar to the past image can be obtained.

When the past image is set as the reference image, it is preferable that the shooting ranges of the reference image and the shot image match. In this case, the photographing position and the photographing range are adjusted so that the same range as the range photographed in the past image set as the reference image is photographed in the present photographing with respect to the structure to be photographed. In order to support the adjustment of the photographing position and the range, information of the photographing position and the photographing range may be stored in association with the past image stored in the image storage unit 103.
When the shooting ranges of the reference image (past image) and the shot image match, the evaluation value may be calculated by using the reciprocal of the sum of squared errors between the pixels of the past image and the shot image. In reality, it is extremely difficult to match the past and present shootings at the pixel level, so it is preferable to use a similarity calculation method that allows a slight positional deviation.

In addition, when the past image includes a deformation of the concrete wall surface, the evaluation value may be calculated based on the deformation portion in the image. In this case, the estimation unit 104 captures the same portion as the deformation of the past image, and calculates a higher evaluation value as the deformation of the past image and the deformation of the captured image are similar to each other. By doing so, it becomes possible to adjust the shooting parameters so that the deformations shown in the past images can be confirmed in the shot images. Furthermore, the estimation unit 104 may calculate the evaluation values of the past image and the captured image in consideration of the secular change of the deformation. For example, a case where the deformation included in the past image is a crack will be described. Cracks recorded in past inspections will not disappear spontaneously unless repair work is performed. On the other hand, cracks may spread due to aging. Therefore, when comparing the cracks of the captured image with the cracks of the past image, the estimation unit 104 does not use the image of the extended portion of the crack of the captured image for calculating the similarity of the cracked portion.

<Embodiment 4>
The reference image processing unit 102 of the above-described embodiment has described the embodiment in which the image stored in the image storage unit 103 is searched and the reference image is set. In the fourth embodiment, an embodiment will be described in which an image is generated by the reference image processing unit 102 and the generated image is used as the reference image.
In recent years, learning-based methods have been developed for image denoising and super-resolution. For example, Non-Patent Document 2 is an image noise removal technique using an auto encoder. In this technique, a noise removal model is learned by learning an auto encoder with an image including noise and an image without noise. When a noise image to be noise-removed is input to this noise-reduction model, an image from which noise is removed is obtained as an output. Further, Non-Patent Document 3 is a super-resolution technique of an image by Fully CNN. In this technique, a super-resolution model is learned by learning Full CNN with a low resolution image and a high resolution image. When a low-resolution image to be high-resolution is input to this super-resolution model, a high-resolution image is obtained as an output. These techniques are techniques for acquiring a conversion model of an image by learning. In the fourth embodiment, these techniques are used to generate a reference image from a temporary captured image. Note that the technology of noise removal and super-resolution has been described as an example, but the technology used in the fourth embodiment may use any method as long as image conversion can be performed. It is not limited to the technique of Document 3.

ここで、ｘ_nは撮影パラメータを調整が不十分な状態で撮影した画像である。ｘ_nに対応するｙ_nは、ｘ_nと同一の撮影対象を好ましい撮影パラメータで撮影した画像である。この学習データセットＤを用いると、基準画像生成モデルＦの学習は、以下の式で表される。

ここで、Ｆ（ｘ_n）−ｙ_nの部分は、基準画像生成モデルＦで画像ｘ_nを変換した画像と、画像ｙ_nとの誤差を表す。したがって、データセットＤのＮ個のデータについて、この誤差が最少となる基準が造成性モデルＦを学習することになる。 FIG. 10 is a diagram illustrating an example of the configuration of the information processing device 100 according to the fourth embodiment. The information processing apparatus 100 of the fourth embodiment is different from FIG. 2 (first embodiment) in that the image storage unit 103 is replaced by a model storage unit 106. The model storage unit 106 stores a model for generating a reference image. Hereinafter, this model is referred to as a reference image generation model. The reference image generation model acquires a conversion method of an image by learning using a technique such as Non-Patent Document 2 or Non-Patent Document 3. The reference image generation model is learned using the following learning data set D, for example.

Here, x _n is an image photographed in a state where the photographing parameters are not sufficiently adjusted. y _n corresponding to x _n is the image captured by the preferred imaging parameters of the same imaging target and x _n. Using this learning data set D, learning of the reference image generation model F is expressed by the following equation.

Here, the part of F(x _n )−y _n represents the error between the image obtained by converting the image x _n by the reference image generation model F and the image y _n . Therefore, for N pieces of data in the data set D, the criterion that minimizes this error is to learn the plasticity model F.

When an image for which the shooting parameters are not adjusted is input to the learned reference image generation model F, an image (generated image) shot with the preferred shooting parameters is output. However, since the generated image is a fake image generated by the reference image generation model F, there is a risk in using it as it is for an inspection image or the like. Although it depends on the performance of the reference image generation model, for example, the generated image may include fine artefacts associated with the image generation processing. Therefore, in the present embodiment, the generated image itself is not used as a shooting result, but is used as a reference for shooting parameter adjustment.
The model storage unit 106 stores the reference image generation model F learned as described above. In addition, as a learning method of an image generation model, a method called GAN (Generative advertisers nets) such as Non-Patent Document 4 has been developed in recent years. This method may be used for learning the reference image generation model F.

Next, a process of creating a reference image using the image generation model F will be described. First, the user tentatively shoots a shooting target with the shooting unit 101. The shooting parameters for this temporary shooting use automatic settings and the like, and it is assumed that the temporary shooting image is an image for which shooting parameter adjustment is insufficient for shooting the shooting target. The reference image processing unit 102 creates a generated image by inputting the tentative captured image to the image generation model F read from the model storage unit 106, and sets this generated image as a reference image for capturing parameter adjustment.
In the above, the generated image is created using the tentative captured image, but the generated image may also be created using the information of the shooting target. For example, as one method, first, the reference image generation model F is learned for each condition such as each type of structure to be imaged or each type of concrete. The model storage unit 106 stores a plurality of reference image generation models together with information on learning conditions. In the step of creating the reference image, the user specifies a condition of the imaging target (for example, the type of structure of the imaging target) to call a reference image generation model that matches the condition from the model storage unit 106 to generate the image. Used for. As described above, the generated image can be created by using the reference image generation model suitable for the shooting target.

Next, another embodiment will be described in which a generated image is created using the information of the shooting target. In this embodiment, in addition to the model storage unit 106, the image storage unit 103 is provided as in the first embodiment. Similar to the first embodiment, the image storage unit 103 stores an ideal shooting result image of a shooting target. The user selects, from the image storage unit 103, an image similar to the condition of the shooting target. The operation of selecting an image from the image storage unit 103 can be performed by searching the image storage unit 103 based on the information of the photographing target, similarly to the reference image selection of the first embodiment. In the present embodiment, the image selected from the image storage unit 103 is called a style image. Then, for example, using the technique of Non-Patent Document 5, the appearance of the tentatively photographed image is converted into an image similar to the style image. Non-Patent Document 5 is a technique for converting the appearance of the original image into an image similar to the style of the style image when the original image and the style image are input, and is, for example, a technique capable of converting the style of the image. In the present embodiment, by setting a temporary captured image as the original image of Non-Patent Document 5, the appearance of the temporary captured image can be made to resemble a style image, and an ideal captured result image can be created. The image created in this way is used as a reference image. According to this embodiment, by selecting the style image from the image storage unit 103 using the information of the shooting target, it becomes easy to generate an image similar to the shooting target.
In the fourth embodiment, the generated image generated by the reference image processing unit 102 as described above is used as the reference image. By performing the subsequent processing in the same manner as in the first embodiment, it is possible to adjust shooting parameters for shooting an image similar to the reference image.

ここで、式（８）のｐ_nは撮影パラメータである。ｓ_nはｐ_nで撮影した画像から得られた評価値である。ｓ_nは閾値以下の評価値とする。式（９）のｐ_dst__nは、（ｓ_n、ｐ_n）の状態から撮影パラメータを調整し、最終的に評価値が閾値以上となったときの撮影パラメータである。これらのデータの組をｎ個収集し、学習データ（Ｘ、Ｙ）を作成する。この学習データを用いて、ある閾値未満の評価値ｓと撮影パラメータｐとが入力されたときに、改善パラメータｐ_dstを出力するモデルＥを学習する。

このモデルの学習には、どのようなアルゴリズムを用いてもよく、例えば、撮影パラメータを連続値であるとすると、線形回帰等の回帰モデルを適用することができる。 In the fourth embodiment, further, an embodiment will be described in which a plurality of shooting parameter settings and a plurality of shootings (S304 and S305 in FIG. 4) are abolished and the shooting parameters are adjusted from the reference image and one shot image. .
First, in the fourth embodiment, one image to be photographed is photographed with certain initial parameters. A process (S306) of calculating an evaluation value by comparison with the reference image is executed for this one image. When the calculated evaluation value is equal to or more than the threshold value, the process of ending the parameter setting (S307, S308, S310 in FIG. 4) is performed.
The process different from the configuration using the multiple shooting parameters of the first embodiment is S309 of estimating the shooting parameter improvement method when the evaluation value is equal to or less than the threshold value. In the fourth embodiment, the improved shooting parameter is estimated by a statistical method from a certain evaluation value and the shooting parameter at that time. Therefore, in the fourth embodiment, the relationship between the evaluation value and the improved shooting parameter is learned in advance. This relationship can be learned using the following data, for example.

Here, p _n in Expression (8) is a shooting parameter. s _n is an evaluation value obtained from an image taken with _pn . s _n is an evaluation value equal to or less than the threshold value. P _{dst — n} in the equation (9) is a shooting parameter when the shooting parameter is adjusted from the state of (s _n , p _n ) and the evaluation value finally becomes equal to or more than the threshold value. Learning data (X, Y) is created by collecting n sets of these data. The learning data is used to learn the model E that outputs the improvement parameter p _dst when the evaluation value s less than a certain threshold and the shooting parameter p are input.

Any algorithm may be used for learning this model. For example, if the imaging parameter is a continuous value, a regression model such as linear regression can be applied.

In the above, the embodiment in which the model for calculating the improvement parameter p _dst is learned by inputting the evaluation value s and the shooting parameter p has been described, but the information of the shot image may be input to this model. The information of the captured image is, for example, a feature amount of the entire image, and more specifically, a luminance histogram of the entire image. The information of the captured image is not limited to this, and may be a partial feature amount of the image, or the image itself may be input to the model. By thus inputting image information into the model as well, it becomes possible to estimate not only the evaluation value and the shooting parameter but also the improvement parameter p _dst based on the shot image.
By using the model E prepared in advance as described above in the estimation of the improvement parameter in S309, in the fourth embodiment, the improved shooting parameter can be estimated from only one image.

なお、目的変数（又は教師データ）Ｙは、式（９）と同様である。 The configuration using the learned model as the method of obtaining the improved shooting parameter may be used in the method of shooting an image with a plurality of shooting parameters according to the first embodiment. That is, the model E is not limited to the configuration in which the shooting parameter is estimated from one image, and may be used in the method of estimating the shooting parameter from a plurality of shot images. In this case, as in the first embodiment, a model E that calculates an evaluation value from a captured image captured with a plurality of imaging parameters and a reference image and inputs a plurality of imaging parameters and a plurality of evaluation values to obtain an improvement parameter is obtained. learn. The learning data X for learning the model M can be rewritten from the equation (8) as follows, where M is the number of images to be captured by adjusting the imaging parameters.

The objective variable (or teacher data) Y is the same as in equation (9).

<Embodiment 5>
In the fourth embodiment, the reference image generation model is used to generate the reference image from the tentative captured image. The reference image generation model is not limited to the embodiment of Embodiment 4, and may be an embodiment used for generating a reference image from a stored image. In the fifth embodiment, an embodiment will be described in which a stored image previously stored in the image storage unit 103 is converted to create a reference image. In the first embodiment, the stored image stored in the image storage unit 103 is selected and the selected stored image is set as the reference image. However, in the fifth embodiment, the selected stored image is converted according to the shooting condition, and then the stored image is converted. Set as a reference image. Hereinafter, this embodiment will be described.

Although a large number of stored images under various shooting conditions are stored in the image storage unit 103, it is difficult to prepare images that match all shooting conditions. Therefore, the stored image is converted and adjusted according to shooting conditions to generate a new image, and the generated image is used as a reference image. For example, a case where the camera models are different as the shooting conditions will be described. First, it is assumed that the stored image is composed only of images captured by a camera of a specific model (hereinafter, camera A). On the other hand, it is assumed that the camera at the time of shooting the shooting target is a camera of a model different from the camera A (hereinafter, camera B). Here, since the models of the camera A and the camera B are different, the image quality of the captured image is different. For example, since the color tone of the image quality and the like differ depending on the model of the camera, even if the camera A and the camera B photograph the same object in the same situation, images having different image quality such as color tone can be obtained.
In the fifth embodiment, in such a case, the stored image of the camera A image quality is converted into an image of the camera B image quality by using the reference image generation model and then set as the reference image. The reference image generation model in this case is a conversion parameter for converting the hue of the camera A into the hue of the camera B, for example. By performing the same processing as that of the first embodiment for the processing after the reference image is set, it is possible to adjust the shooting parameters to match the image quality of the reference image.

Next, a process for carrying out such a process will be described. The information processing apparatus according to the present embodiment further includes a model storage unit in the information processing apparatus 100 in FIG. 2, and the model storage unit stores a reference image generation model according to a shooting condition. First, in the first process, as in the first embodiment, the reference image processing unit 102 retrieves a stored image similar to the shooting target from the image storage unit and acquires it. In the fifth embodiment, the stored image of the search result is used as the temporary reference image. In the next process, the reference image processing unit 102 prepares a reference image generation model for converting the temporary reference image into the reference image based on the shooting conditions. In the above-described example, the reference image processing unit 102 sets the model of the camera (camera B) as the photographing condition to model the reference image generation model including the conversion parameter for converting the hue of the camera A into the hue of the camera B. Read from storage. For this purpose, the user inputs the information on the photographing conditions via the operation unit 12. Alternatively, the information on the photographing conditions that can be automatically acquired, such as the camera model, may be automatically acquired and used for searching the reference image generation model. In the next process, the reference image processing unit 102 converts the temporary reference image using this reference image generation model to create a reference image.

In the above example, the embodiment has been described in which the photographing condition is set as the camera model, the reference image generation model is selected based on the photographing condition, and the temporary photographed image is converted to generate the reference image. The shooting condition for generating the reference image is not limited to the camera model and may be other conditions. For example, the weather may be set as the shooting condition. In this case, if the stored image (temporary reference image) selected as similar to the shooting target is an image shot in sunny weather and the weather when shooting the shooting target is cloudy, the image quality of the sunny image A reference image generation model for converting (hue or brightness) into the image quality of a cloudy image is selected from the model storage unit. Other variations of the shooting conditions may be the shooting time, the shooting season, and the like. Furthermore, a condition such as handheld shooting, tripod shooting, or shooting with a camera mounted on a moving body such as a drone may be one of the shooting conditions. Moreover, such an imaging condition may be a combination of a plurality of conditions. For example, under the shooting condition of “camera B, cloudy”, an image generation model for converting the temporary reference image of “camera A, sunny” into the image quality of “camera B, cloudy” may be selected.

Further, in the above, the example of the image generation model is the parameter for converting the image, but the image generation model in the fifth embodiment is not limited to this, and may be a model based on learning as described in the fourth embodiment. In this case, for example, the image generation model is learned for each shooting condition for which an image is desired to be converted, and the image generation model is selected and used according to the shooting condition. The learning of the image generation model can be performed in the same manner as the learning of the fourth embodiment by using a data set including an image group before conversion and a preferable image group after conversion.
In the fourth and fifth embodiments, the reference image processing unit 102 displays the generated reference image on the operation unit 12 so that the user can check the reference image. If the user determines that the reference image is not suitable as a reference for adjusting the shooting parameters, another reference image may be generated again. In this case, the reference image generation model candidates are displayed so that the reference image generation model for generating another reference image can be reselected, and the reference image generation model can be searched again. Good. Further, when the methods of the fourth and fifth embodiments are used at the same time, the reference image obtained by converting the temporarily captured image (the reference image created by the method of the fourth embodiment) is used, or the temporary reference image is converted. The reference image (the reference image created by the method of the fifth embodiment) may be used, or the user may select it. In this case, the reference image processing unit 102 displays, on the operation unit 12, the reference image obtained by converting the temporary captured image and the reference image obtained by converting the temporary reference image in a comparable state, and the user displays the reference for adjusting the capturing parameters. So that a reference image suitable as can be selected.

<Sixth Embodiment>
In the above embodiment, the embodiment in which the information processing apparatus 100 of the present embodiment is applied to the inspection image capturing of the infrastructure structure has been described. The information processing apparatus 100 of the present embodiment can be applied not only to inspection image shooting but also to shooting parameter adjustment for other shooting targets. In the sixth embodiment, an embodiment will be described in which the above-described processing and the like are applied to shooting parameter adjustment in general photography.
In the present embodiment, the stored image stored in the image storage unit 103 of the first embodiment is changed in order to apply the above-described processing to general photography. FIG. 11 is a diagram illustrating information stored in the image storage unit 103 in the fifth embodiment. Similar to FIG. 3, the image storage unit 103 in FIG. 11 stores the stored image, image information, and shooting parameters. The stored image 1010 in FIG. 11 is a landscape photograph of the sea, and as the image information of the stored image, information such as scene: landscape, weather: sunny, detail 1: ocean, detail 2: summer is recorded. Further, the stored image 1011 is a baseball image, and image information indicating the image content is also recorded in association with this stored image.

In the present embodiment as well, similar to the first embodiment, the reference image is selected from the image storage unit 103 based on the information of the shooting target that the user is going to shoot. The user selects the type of scene to be photographed, weather, or other information, or inputs a keyword. The reference image processing unit 102 searches the image information stored in the image storage unit 103 based on the input information of the user, and selects the stored image suitable as the reference image. As with the first embodiment, the reference image may be selected by presenting the reference image candidates to the user, selecting the image that the user determines to be the most suitable, and setting the reference image. The reference image may be automatically used. Further, similarly to the first embodiment, the temporary captured image may be captured, and the reference image may be retrieved from the image storage unit 103 based on the temporary captured image.

Through the above processing, even in the case of general photography, it is possible to select the reference image that is the reference for adjusting the shooting parameters of the shot image. In the process after the setting of the reference image, the evaluation values of the captured image and the reference image are calculated and the shooting parameter adjustment is executed, as in the above-described embodiment.
In the above description of the general photography embodiment, the embodiment in which the configuration described above is applied to the general photography by changing the image stored in the image storage unit 103 has been described. As an embodiment in which the above-described configuration or the like is applied to general photography, a configuration in which a reference image is generated using a reference image generation model may be used as in the fourth embodiment.

<Other embodiments>
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium. Then, it can be realized by a process in which one or more processors in the computer of the system or apparatus read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

Although an example of the embodiment of the present invention has been described above in detail, the present invention is not limited to the specific embodiment.

As described above, according to the information processing apparatus 100 of each of the above-described embodiments, it is possible to easily set the shooting parameter for shooting a desired image without confirming the details of the shot image.

10 CPU
11 storage unit 12 operation unit 13 communication unit 100 information processing device

Claims (20)

Acquisition means for acquiring a reference image,
An evaluation value is obtained from the reference image and the plurality of captured images received from the capturing unit that captures the plurality of captured images with the plurality of capturing parameters, and the target is captured based on the evaluation value. Estimating means for estimating a shooting method suitable for shooting
Information processing device having a. An image storage unit that stores a stored image that is a candidate for the reference image and image information of the stored image;
The information processing apparatus according to claim 1, wherein the acquisition unit acquires the reference image by selecting the reference image from a stored image stored in the image storage unit based on a search condition. The acquisition unit selects a plurality of reference image candidates from the stored images stored in the image storage unit based on the search condition, displays the selected reference image candidates on the operation unit, and the user via the operation unit. The information processing apparatus according to claim 2, wherein the reference image is selected based on an operation. The information processing apparatus according to claim 1, wherein the evaluation value is a degree of similarity between the plurality of captured images and the reference image. The information processing apparatus according to claim 1, wherein the shooting method is a shooting parameter related to the shooting unit. The information processing apparatus according to claim 1, further comprising a setting unit that sets the shooting parameter in the shooting unit. The said estimation means estimates any one of the position and attitude|position of photography, photography time, and illumination conditions as said photography method, when the said evaluation value does not exceed a threshold value even if the said photography parameter is adjusted. Information processing equipment. The estimating means displays the estimated photographing method on the operation unit and selects whether the estimated photographing method is selected or the estimation of the photographing method is executed again based on a user operation via the operation unit. The information processing apparatus according to any one of claims 1 to 7, which makes a determination. The acquisition means acquires a plurality of reference images,
The information processing apparatus according to claim 1, wherein the estimation unit obtains an evaluation value from the plurality of reference images and the plurality of captured images. The object to be photographed is a concrete wall surface of an infrastructure structure,
The image information includes at least one of the type of structure to be photographed, the type of concrete, the weather at the time of photography, the target in the image, the installation location and area of the structure, and the number of elapsed years. Information processing device. The object to be photographed is a concrete wall surface of an infrastructure structure,
The said estimation means calculates|requires the said evaluation value by calculating|requiring the similarity of the partial image of the crack of the said concrete wall surface contained in the said image|photographed image and the said reference image, a joint, and a formwork. The information processing apparatus according to any one of items. The information processing apparatus according to claim 1, wherein the acquisition unit acquires the reference image by generating the reference image using a model learned in advance. Generating means for generating a reference image from a tentatively photographed image of an object to be photographed using a model learned in advance;
An estimation unit that obtains an evaluation value from the reference image and the provisional captured image, and estimates a capturing method suitable for capturing the capturing target based on the evaluation value.
Information processing device having a. Generating means for generating a reference image from the stored image and the shooting conditions of the shooting target using the model learned in advance;
An estimation unit that obtains an evaluation value from the reference image and the provisional captured image, and estimates a capturing method suitable for capturing the capturing target based on the evaluation value,
Information processing device having a. Selection means for selecting a reference image,
An estimation unit that obtains an evaluation value from the reference image and a captured image received from a capturing unit that captures the capturing target, and estimates a capturing method suitable for capturing the capturing target based on the evaluation value.
Have
The estimating unit again obtains an evaluation value from the reference image and a captured image captured by the estimated capturing method, and re-estimates a capturing method suitable for capturing the capturing target based on the evaluation value. Information processing device. An information processing method executed by an information processing device, comprising:
An acquisition process for acquiring a reference image,
An evaluation value is obtained from the reference image and the plurality of captured images received from the capturing unit that captures the plurality of captured images with the plurality of capturing parameters, and the target is captured based on the evaluation value. An estimation step of estimating a shooting method suitable for shooting
Information processing method including. An information processing method executed by an information processing device, comprising:
A generation step of generating a reference image from a tentatively photographed image of an object to be photographed using a model learned in advance;
An estimation step of obtaining an evaluation value from the reference image and the provisional captured image, and estimating a capturing method suitable for capturing the capturing target based on the evaluation value,
Information processing method including. An information processing method executed by an information processing device, comprising:
A generation step of generating a reference image from the stored image and the shooting conditions of the shooting target using a model learned in advance;
An estimation step of obtaining an evaluation value from the reference image and the tentatively photographed image, and estimating a photographing method suitable for photographing the photographing target based on the evaluation value;
Information processing method including. An information processing method executed by an information processing device, comprising:
An acquisition process for acquiring a reference image,
An estimation step of obtaining an evaluation value from the reference image and a captured image received from a capturing unit that captures the capturing target, and estimating a capturing method suitable for capturing the capturing target based on the evaluation value.
Including,
In the estimating step, an evaluation value is obtained again from the reference image and a captured image captured by the estimated capturing method, and a capturing method suitable for capturing the capturing target is estimated again based on the evaluation value. Information processing method. A program for causing a computer to function as each unit of the information processing apparatus according to claim 1.

Images