JP7451392B2 - Image processing device, image processing method, image processing program - Google Patents

Description

The present invention relates to image processing technology.

Patent Document 1 discloses an image processing device that corrects distortion of an image taken with a camera installed on a construction machine and detects an object with high accuracy.

JP2020-13220A

When a camera is used outdoors, such as at a construction site, as in Patent Document 1, the photographing state of the camera affects image processing. Here, the photographing state of the camera is a concept that includes abnormalities such as scratches, dirt, and malfunctions that may occur in the camera itself, as well as the environment at the time of photographing, such as weather, brightness, time of day, and electromagnetic waves. When the shooting conditions are normal, objects can be detected with high precision by the image processing device of Patent Document 1, but when the shooting conditions are abnormal, such as when there is a scratch on the camera lens or at night when it is raining. When photographing, object detection accuracy may decrease. The property of preventing deterioration of image processing accuracy due to such deterioration of photographing conditions is called robustness. Conventionally, in order to evaluate the robustness of an image processing device, it has been necessary to individually evaluate the results of processing images actually taken by the image processing device under poor shooting conditions. That is, in the above example, it is necessary to actually scratch the camera lens to take a picture or to actually take a picture at night when it is raining, which is inefficient.

The present invention has been made in view of these circumstances, and its purpose is to provide an image processing device that can efficiently evaluate robustness.

In order to solve the above problems, an image processing device according to an aspect of the present invention includes an image input unit that inputs an image, an image processing unit that performs predetermined image processing on the image, and an image processing unit that captures the image and inputs the image. a simulated image generation unit that performs simulation processing on an image input in the image input unit to simulate a photographing state assumed by a photographing device that can be input to the image processing unit , and provides the image processing unit to the image processing unit; Regarding the evaluation images used for the evaluation of the parts, the results of the image processing performed by the image processing part on the evaluation images that have been subjected to the simulation processing, and the results of the image processing performed by the image processing part on the evaluation images that have not been subjected to the simulation processing. and an evaluation section that evaluates the image processing section based on a comparison of the results of the image processing performed .

In this aspect, since the input image is subjected to simulation processing that simulates the imaging state assumed by the imaging device, there is no need to actually capture the image in that imaging state. Therefore, robustness can be evaluated efficiently. Note that in the example of Patent Document 1, a camera (photographing device) used outdoors has been described, but since the photographing state can change even with a photographing device used indoors, the image processing device of this aspect can be similarly applied. can.

Another aspect of the invention is an image processing method. This method consists of an image input step for inputting an image, an image processing step for performing predetermined image processing on the image, and a simulation of a shooting condition assumed by a shooting device that can shoot an image and input it to the image input step. A simulated image generation step performs simulation processing on the image input in the image input step and provides the image to the image processing step , and a simulation processing is performed on the evaluation image input by the image input step and used for evaluation of the image processing step. Evaluate the image processing step based on a comparison between the results of image processing performed by the image processing step on the evaluation image that has been processed and the results of image processing performed by the image processing step on the evaluation image that has not been subjected to simulation processing. and an evaluation step .

Note that arbitrary combinations of the above components and expressions of the present invention converted between methods, devices, systems, recording media, computer programs, etc. are also effective as aspects of the present invention.

According to the present invention, it is possible to provide an image processing device that can efficiently evaluate robustness.

FIG. 1 is a functional block diagram showing an image processing device according to an embodiment. 3 is a flowchart showing processing of the image processing device.

DESCRIPTION OF EMBODIMENTS Hereinafter, embodiments for carrying out the present invention will be described in detail with reference to the drawings. In the description and drawings, the same or equivalent components, members, and processes are denoted by the same reference numerals, and overlapping explanations will be omitted as appropriate. The scales and shapes of the parts shown in the figures are set for convenience to facilitate explanation, and should not be interpreted in a limited manner unless otherwise stated. The embodiments are illustrative and do not limit the scope of the present invention. All features and combinations thereof described in the embodiments are not necessarily essential to the invention.

FIG. 1 is a functional block diagram showing an image processing apparatus 10 according to an embodiment. The image processing device 10 is a device that performs predetermined image processing on images supplied from the camera 20 or the image memory 21.

A camera 20 serving as a photographing device photographs an image and supplies it to the image processing device 10 or the image memory 21 . The installation location and installation mode of the camera 20 are arbitrary; it may be attached to other equipment with moving parts such as construction machinery, it may be installed at a specific point, or it may be carried by a person. . In this embodiment, an example will be mainly described in which the camera 20 is attached to a construction machine. In this example, the image processing device 10 detects objects such as people, building materials, buildings, and other construction machines that may come into contact with the construction machine, based on images of the surroundings of the construction machine taken by the camera 20. . The images taken by the camera 20 may be still images or moving images, but in the case of still images, it is preferable to take images continuously at a high frequency of, for example, several seconds or less so that objects around the construction machine can be detected in a timely manner.

The camera 20 directly supplies the image to the image processing device 10 without going through the image memory 21 if the captured image is required for real-time image processing such as detecting contact with an object. The image processing device 10 in this case may be installed in a computer within the same construction machine as the camera 20, or may be installed in a computer outside the construction machine at a construction site or a remote location. In either case, the camera 20 and the image processing device 10 are configured to be able to communicate in real time by wire or wirelessly, and images captured by the camera 20 are supplied to the image processing device 10 in real time.

On the other hand, if the image taken by the camera 20 is not required for real-time image processing, the image is stored in the image memory 21 so that it can be used for later image processing. The image memory 21 may be the built-in memory of the camera 20, or may be a general-purpose removable medium such as a memory card. Alternatively, it may be a storage inside or outside the construction machine that can communicate with the camera 20 and the image processing device 10 by wire or wirelessly.

An image registration unit 22 installed in a computer inside or outside the construction machine can register any image in the image memory 21 and the simulated image memory 23. Although details will be described later, the image registration unit 22 stores an arbitrary evaluation image used for evaluating the robustness of the image processing device 10 in the image memory 21, and stores an arbitrary evaluation image used for evaluating the robustness of the image processing device 10, and stores an arbitrary evaluation image used for evaluating the robustness of the image processing device 10, The simulated image is stored in the simulated image memory 23. The simulated image memory 23 may be any type of memory as long as it can be accessed by the image registration unit 22 and the image processing device 10 regardless of whether they are accessed at the same time or not.

The image processing device 10 includes an image input section 11 , an image processing section 12 , a simulated image generation section 13 , an evaluation section 14 , and an adjustment section 15 .

The image input unit 11 inputs images from the camera 20 or the image memory 21. The image processing unit 12 performs predetermined image processing on the image input by the image input unit 11. The content of the image processing can be arbitrarily set depending on the installation location or manner of installation of the camera 20 and the purpose of the image processing device 10. In the example of this embodiment in which the camera 20 is attached to a construction machine and the purpose of the image processing device 10 is to detect objects around the construction machine, the image processing unit 12 performs object detection processing based on the input image. The object detection result as an image processing result of the image processing unit 12 includes the presence or absence of a detected object, and if an object is detected, the type of object (person, building material, building, other construction machine, etc.), It includes the possibility (risk level) of the object coming into contact with the construction machine, and is output to other devices outside the image processing device 10.

The following are other examples of image processing to which the present invention is applicable.
・Detection of abnormalities in the inspection object such as damage, dirt, defects, distortion, etc. based on images taken with a camera installed in an image inspection device ・Detection of abnormalities of the inspection object such as damage, dirt, defects, distortion, etc. based on images taken with a camera installed on a vehicle such as a car Detection of surrounding objects ・Measurement of distance to photographed object ・Recognition of information and characters in images (Optical Character Recognition (OCR), etc.)
・Recognition of gestures of photographed people and animals ・Image interpretation and diagnosis using artificial intelligence ・Image processing (image quality improvement processing, filter processing, viewpoint conversion, 3D model generation, etc.)

The simulated image generation unit 13 includes a simulated image superimposition unit 131 and a simulated image switching unit 132, and performs simulation processing on the image input by the image input unit 11 to simulate the shooting state assumed by the camera 20, and converts the image into an image processing unit. Provided on 12th. As an example of simulation processing, the simulated image superimposition unit 131 superimposes a simulated image that simulates a photographing state assumed by the camera 20 on the input image. The simulated image is registered in advance in the simulated image memory 23 by the image registration section 22. A simulated image switching unit 132 serving as a simulated processing switching unit that switches between a plurality of simulated processes corresponding to different shooting conditions reads out the simulated image stored in the simulated image memory 23 and provides it to the simulated image superimposing unit 131. The simulated image switching unit 132 simultaneously reads an arbitrary number of simulated images, and the simulated image superimposing unit 131 simultaneously superimposes these simulated images on the input image. For example, by simultaneously superimposing a simulated image representing "night" and a simulated image representing "rain" on the input image, it is possible to simulate or reproduce a shooting condition of "rainy night." Further, the simulated image switching unit 132 can easily simulate different shooting conditions by switching the simulated images superimposed by the simulated image superimposing unit 131. For example, following the above-mentioned shooting condition "rainy night", a different shooting condition such as "there is a scratch on the camera lens" can be quickly simulated by switching to a simulated image showing a scratch on the lens of the camera 20. can. Note that in the above example of simulation processing, a simulation image is superimposed on the input image, but various photographing states of the camera 20 may be simulated by applying appropriate image processing processing such as filter processing to the input image. . For example, in order to simulate a shooting condition called "night", filter processing is performed to uniformly lower the brightness of the input image. Furthermore, the image processing for simulating such a shooting state may be applied to a part of the input image, or may be different for each part of the input image. For example, it is possible to simulate darkness by simply forcibly turning pixels at regular intervals black, or to simulate noise based on dark current or the like by randomly changing the brightness of each pixel. By preparing such various simulation processing patterns in advance, it is possible to generate simulation images from various viewpoints for one evaluation image. Therefore, there is no need to actually photograph evaluation images under various photographing conditions as in the past, and robustness can be evaluated efficiently.

The simulated image switching unit 132 is configured with an FPGA (Field-Programmable Gate Array), which is a dynamically reconfigurable circuit, and switches between a plurality of simulated processes or simulated images by the dynamic reconfiguration. The simulated image switching unit 132 may be configured with a general-purpose arithmetic processing device, and a plurality of simulated processes or simulated images are switched by software executed by the arithmetic processing device.

The photographing conditions simulated by the simulated image superimposing unit 131 are broadly divided into abnormalities that may occur in the camera 20 and environments in which the camera 20 is placed. The types of abnormalities that may occur in the former camera 20 include, for example, lens damage, scratches, and dirt, and circuit operation delays, malfunctions, and failures. The type of environment in which the latter camera 20 is placed includes, for example, the weather, brightness, time of day, incident light, electromagnetic waves, and environmental noise (white spot noise that occurs during night shooting due to the dark current of the camera's image sensor, strong winds, etc.). (Noise caused by dust from time to time, noise caused by carbon chips flying around inside the coke oven, etc.) In order to simulate each of these types individually, the simulated image memory 23 stores simulated images corresponding to each type. Furthermore, if a plurality of photographing states are assumed for the same type, separate simulated images are prepared for each photographing state. For example, for the type of shooting condition "rain," simulated images are prepared according to various possible modes such as "storm," "strong rain," "light rain," and "drizzle." Similarly, with respect to the type of photographic state "there is a scratch on the lens", simulated images are prepared according to various possible sizes, shapes, and positions of scratches. By preparing individual simulated images for each possible aspect of the photographing state in this manner, the simulated image superimposing unit 131 can finely simulate any photographing state by combining these simulated images. For example, complex shooting situations such as ``There is a large streak-like scratch in the center of the camera lens, there is a moderate operation delay in the camera circuit, the weather is cloudy, around 5 p.m., and there is strong backlight'' can be easily handled. can be simulated.

Among the above-mentioned photographing conditions, it is preferable to use a moving image as a simulated image that simulates a photographing condition with movement or temporal changes such as "circuit operation delay" or "rain". In addition, for shooting conditions such as "incident light" that have different effects on the image depending on the posture and properties of the photographed subject, the simulated image superimposition unit The simulated image superimposed by 131 may be adjusted. For example, if the reflectance of the subject is low, the intensity of the incident light to be superimposed as a simulated image is weakened.

The image processing unit 12 also performs the above-described predetermined image processing on the input image on which the simulated image has been superimposed by the simulated image superimposition unit 131. That is, the image processing section 12 processes an input image without a simulated image directly provided from the image input section 11 and a simulated image provided indirectly from the image input section 11 via the simulated image superimposition section 131. Perform image processing on input images with images in parallel. The former image processing result for the input image without a simulated image is output to another device outside the image processing device 10, and the latter image processing result for the input image with the simulated image is output to the evaluation unit 14 described below. The data is provided and used for evaluating the robustness of the image processing device 10. In this way, the image processing unit 12 processes the input image with a simulated image and the input image without a simulated image in parallel, thereby performing the normal operation of the image processing device 10 of outputting an image processing result. The robustness of the image processing device 10 can be evaluated in real time. As a modified example, by switching the input images provided to the image processing unit 12, a normal operation mode in which normal operation is performed based on an input image without a simulated image, and a robustness evaluation based on an input image with a simulated image are performed. The robustness evaluation mode may be switched.

The evaluation unit 14 evaluates the image processing unit 12 based on a comparison between the result of image processing performed by the image processing unit 12 on the evaluation image on which the simulated image is superimposed and the result of the intended image processing on the evaluation image. do. Here, the "evaluation image" means an image input by the image input section 11 when the evaluation section 14 performs evaluation. The evaluation image may be an image taken by the camera 20 in real time, an image taken in the past by the camera 20 stored in the image memory 21, or an image that is stored in the image memory 21 by the image registration unit 22 without using the camera 20. It can be any image that has been created. When an image photographed by the camera 20 in real time is used as an evaluation image, there is an advantage that the evaluation by the image processing unit 12 can be performed in real time. On the other hand, when an image stored in the image memory 21 is used as an evaluation image, there is an advantage that an image suitable for evaluation by the image processing section 12 can be arbitrarily selected.

The evaluation image in an example in which the camera 20 is attached to a construction machine is an image taken of the surroundings of the construction machine. The image processing unit 12 performs object detection processing as image processing on the evaluation image, and outputs image processing results including the presence or absence of a detected object, the type of object, the possibility of the object coming into contact with a construction machine, and the like. For example, if a person is shown at a close distance in the evaluation image, the expected image processing result is "Presence or absence of detected object: Yes/Type of object: Human/Possibility of the object coming into contact with construction machinery: High" becomes. If the image processing result of the image processing unit 12 when the simulated image superimposing unit 131 superimposes the simulated image on this evaluation image differs from the expected image processing result, the image processing unit 12 will change the imaging state that the simulated image simulates. (for example, "rain") is evaluated as having low robustness. On the other hand, if the expected image processing result is obtained even when the simulated image is superimposed, the image processing unit 12 is evaluated to have high robustness to the imaging state simulated by the simulated image. In this way, the evaluation unit 14 compares the image processing result of the evaluation image on which the simulated image is superimposed with the intended image processing result, thereby evaluating the robustness of the image processing unit 12 with respect to the imaging state simulated by the simulated image. It can be evaluated.

Note that when using an image taken in real time by the camera 20 as an evaluation image, the desired image processing result cannot be known in advance; The image processing result of the evaluation image can be substituted for the desired image processing result. At this time, in order to ensure thorough evaluation, the evaluation unit 14 may present a provisional evaluation result to the user and ask for the user's final evaluation. Alternatively, the user may be responsible for all evaluations without providing the evaluation section 14.

On the other hand, when using an image stored in the image memory 21 as an evaluation image, the user can select any image suitable for evaluation by the image processing unit 12 and know the desired image processing result in advance. The evaluation unit 14 can effectively perform the evaluation without asking for the final evaluation. For example, when detecting people at a construction site, it is required to detect individual construction workers with high accuracy. Therefore, using an image showing multiple workers wearing helmets as an evaluation image, the evaluation unit 14 evaluates whether the image processing unit 12 can detect individual workers even under various shooting conditions (simulated images). will be evaluated. Conversely, it is also possible to evaluate whether the image processing unit 12 mistakenly detects a helmet as a person by using an image in which a helmet not worn by a worker is reflected as an evaluation image. Furthermore, in the same example of detecting people at construction sites, a blue tarpaulin sheet (so-called blue sheet), which is often used at construction sites, may be mistakenly detected as human clothing. In order to prevent such false detections, the image processing unit 12 uses images in which blue sheets are reflected in various places as evaluation images to falsely detect blue sheets as people under various shooting conditions (simulated images). The evaluation unit 14 evaluates whether or not to do so. In this way, an appropriate evaluation image can be arbitrarily selected depending on the installation location or installation mode of the camera 20 and the purpose of the image processing device 10, so that the robustness of the image processing section 12 can be evaluated effectively.

The adjustment unit 15 adjusts the image processing unit 12 in order to improve the imaging state that the evaluation unit 14 evaluates as having low robustness.

FIG. 2 is a flowchart showing the processing of the image processing device 10. "S" in this figure means "step". In S1, the image input unit 11 inputs a real-time image from the camera 20 or a non-real-time image from the image memory 21. In S2, the image processing unit 12 performs predetermined image processing on the image input in S1. No simulated image is superimposed on this image, and the image processing unit 12 outputs the image processing result to other devices outside the image processing device 10. If the image is a non-real-time image and the image processing result is not needed outside the image processing apparatus 10, the image processing result may not be output.

In S3, the simulated image superimposition unit 131 superimposes the simulated image on the image input in S1. In S4, the image processing unit 12 performs predetermined image processing on the image on which the simulated image was superimposed in S3. In S5, the evaluation unit 14 determines whether the evaluation image input in S1 is a real-time image of the camera 20. In the case of a real-time image, the process proceeds to S6, where the evaluation unit 14 obtains the image processing result without the simulated image in S2 as the intended image processing result. In the case of a non-real-time image, the process proceeds to S7, where the evaluation unit 14 acquires the desired image processing result stored in the image memory 21 together with the non-real-time image. If the desired image processing result is not stored in the image memory 21, the image processing result in S2 without the simulated image may be acquired as the desired image processing result, similar to S6.

In S8, the evaluation unit 14 evaluates the robustness of the image processing unit 12 based on the comparison between the image processing result with the simulated image in S4 and the intended image processing result obtained in S6 or S7. In S9, the adjustment unit 15 adjusts the image processing unit 12 to improve the robustness when it is evaluated as low in S8. In S10, the simulated image switching unit 132 determines whether there is another simulated image whose robustness should be evaluated. If there is another simulated image, the process advances to S11, and the simulated image switching unit 132 switches to the other simulated image. Subsequently, the process returns to S3, and the simulated image superimposing unit 131 superimposes the other simulated image on the evaluation image. Thereafter, the robustness evaluation and the like are similarly performed for the other simulated images (photographing conditions).

The present invention has been described above based on the embodiments. Those skilled in the art will understand that the embodiments are merely illustrative, and that various modifications can be made to the combinations of the constituent elements and processing processes, and that such modifications are also within the scope of the present invention.

Note that the functional configuration of each device described in the embodiments can be realized by hardware resources, software resources, or cooperation between hardware resources and software resources. A processor, ROM, RAM, and other LSIs can be used as hardware resources. Programs such as operating systems and applications can be used as software resources.

10 image processing device, 11 image input unit, 12 image processing unit, 13 simulated image generation unit, 14 evaluation unit, 15 adjustment unit, 20 camera, 21 image memory, 22 image registration unit, 23 simulated image memory, 131 simulated image superimposition Section, 132 Simulated image switching section.

Claims (9)

an image input section for inputting an image;
an image processing unit that performs predetermined image processing on the image;
a simulated image generation unit that performs simulation processing on the image input by the image input unit to simulate a photographing state assumed by a photographing device capable of photographing an image and inputting the image to the image input unit, and provides the resultant image to the image processing unit; and ,
Regarding the evaluation image input by the image input unit and used for the evaluation of the image processing unit, the results of the image processing performed by the image processing unit on the evaluation image subjected to the simulation processing, and the results of the simulation processing an evaluation unit that evaluates the image processing unit based on a comparison of results of image processing performed by the image processing unit on the evaluation image that has not been subjected to the evaluation image;
An image processing device comprising: The image processing device according to claim 1 , further comprising a simulation processing switching unit that switches between a plurality of the simulation processing corresponding to the different shooting states. The image processing device according to claim 2 , wherein the simulation processing switching unit is configured with a dynamically reconfigurable circuit, and switches between the plurality of simulation processes by the dynamic reconfiguration. The image processing device according to claim 3 , wherein the dynamically reconfigurable circuit is an FPGA. The image processing device according to any one of claims 1 to 4 , wherein the photographing state includes at least one of an abnormality that may occur in the photographing device and an environment in which the photographing device is placed. The image processing device according to any one of claims 1 to 5 , wherein the simulation process involves superimposing a moving image that simulates a temporal change in the photographing state on the image input by the image input unit. The image processing device according to any one of claims 1 to 6 , wherein the image input to the image input unit includes an image not generated by the photographing device. an image input step of inputting an image;
an image processing step of performing predetermined image processing on the image;
a simulated image generation step of performing simulation processing on the image input in the image input step to simulate a photographing state assumed by a photographing device capable of photographing an image and inputting the image to the image input step, and providing the resultant image to the image processing step; and ,
Regarding the evaluation image inputted by the image input step and used for the evaluation of the image processing step, the results of the image processing performed by the image processing step on the evaluation image subjected to the simulation processing and the results of the simulation processing an evaluation step of evaluating the image processing step based on a comparison of the results of image processing performed by the image processing step on the evaluation image that has not been subjected to the evaluation image;
An image processing method comprising: an image input step of inputting an image;
an image processing step of performing predetermined image processing on the image;
a simulated image generation step of performing simulation processing on the image input in the image input step to simulate a photographing state assumed by a photographing device capable of photographing an image and inputting the image to the image input step, and providing the resultant image to the image processing step; and ,
Regarding the evaluation image inputted by the image input step and used for the evaluation of the image processing step, the results of the image processing performed by the image processing step on the evaluation image subjected to the simulation processing and the results of the simulation processing an evaluation step of evaluating the image processing step based on a comparison of the results of image processing performed by the image processing step on the evaluation image that has not been subjected to the evaluation image;
An image processing program that causes a computer to execute.

Images