JP2022138333A - Image processing device, image processing method, and image processing program - Google Patents

Abstract

To provide an image processing device, an image processing method, and an image processing program capable of obtaining a highly visible image regardless of the imaging status.SOLUTION: The image processing device includes a dividing unit for dividing an image into a plurality of pixels, a substituting unit for substituting pixels of a saturation threshold or higher out of the plurality of pixels into substituted pixels, a first median calculating unit for calculating a first median value of the pixels of the plurality of pixels that are less than the saturation threshold value, and a conversion unit for converting the pixels of the plurality of pixels that are less than the saturation threshold value into pixels based on a second median value different from the first median value.SELECTED DRAWING: Figure 4

Description

The present invention relates to an image processing device, an image processing method, and an image processing program.

Japanese Unexamined Patent Application Publication No. 2002-100002 discloses a tone conversion apparatus that aims to generate a gain map suitable for tone adjustment of a RAW image with a small processing load. This gradation conversion device has an image input section, a representative value processing section, a gain map generation section, and a gradation conversion section. The image input unit takes in a RAW image. The representative value processing unit divides the image area of the RAW image into a plurality of partial areas, obtains representative values based on the color components included in the partial areas, and generates a reduced image having the representative values as pixel values. A gain map generator generates a gain map based on the reduced image. The gradation conversion section performs gradation conversion of the RAW image based on the gain map.

Further, for example, in image processing for vehicles, algorithms such as dynamic range correction at night or when passing over an overpass are being studied.

JP 2007-180851 A

However, conventional image processing techniques have room for improvement in terms of obtaining an image with high visibility regardless of shooting conditions.

SUMMARY OF THE INVENTION It is an object of the present invention to provide an image processing apparatus, an image processing method, and an image processing program capable of obtaining an image with high visibility regardless of the shooting conditions. and

The image processing apparatus of the present embodiment includes a dividing unit that divides an image into a plurality of pixels, a replacing unit that replaces pixels equal to or greater than a saturation threshold among the plurality of pixels with replacement pixels, and among the plurality of pixels, a first median calculator that calculates a first median value of pixels less than the saturation threshold; and a conversion unit that converts to pixels based on the median value.

According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and an image processing program capable of obtaining an image with high visibility regardless of shooting conditions.

1 is an external configuration diagram showing an example of a robot equipped with an image processing apparatus according to an embodiment; FIG. 3 is a functional block diagram showing an example of internal components of a control unit; FIG. FIG. 4 is a diagram showing an example of each step of image processing according to the present embodiment; It is a figure which shows an example of each calculation of the image processing of this embodiment. 4 is a flowchart showing an example of image processing according to the embodiment; FIG. 5 is a conceptual diagram showing an example of an image subjected to image processing according to the present embodiment and an image of a comparative example;

FIG. 1 is an external configuration diagram of a robot 10 as an example of an unmanned mobile object (robot, UAV (unmanned aerial vehicle) 10) equipped with an image processing apparatus according to the present embodiment.

The robot 10 is installed at, for example, a construction site, a civil engineering site, or the like, and has a function of performing monitoring while moving within the installation site. The robot 10 has a body portion 11 , a caterpillar portion 12 and a head portion 13 . A caterpillar portion 12 is connected to the lower portion of the main body portion 11, and a head portion 13 is connected to the upper portion of the main body portion 11 (connecting portions are omitted in the drawing).

A controller 11X including functional components of the robot 10 is stored in the main body 11 . The controller 11X includes, for example, a CPU (Central Processing Unit). Functional components included in the control unit 11X will be described later in detail. 10 has a function of applying various image processing to the image captured by the camera 10 . That is, the control unit 11X includes the image processing device (dynamic range correction device, gradation correction device) of this embodiment.

The caterpillar portion 12 has a function of moving (running) within the installation site of the robot 10 . Movement control (running control) of the robot 10 by the caterpillar portion 12 is executed based on, for example, an autonomous movement operation command (autonomous running operation command) from the cloud or a server (not shown).

The head 13 has a photographing unit 13X for photographing the installation site of the robot 10 . Here, a case is illustrated in which photographing lenses (camera lenses) provided at positions corresponding to both eyes of the head 13 of the robot 10 constitute the photographing unit 13X. A subject image formed on an imaging device (not shown) through the imaging unit 13X is input to the control unit 11X as a captured image (signal).

In this way, the robot 10 that moves within the installation site is equipped with a camera to capture images and videos, grasp the installation site in real time, and transmit the captured images and the grasped situation to a cloud or a server (not shown). can be stored in The captured images and videos can be used, for example, for patrol monitoring in the installation site, remote monitoring from the office, remote response (communication) from the office to workers, work progress management, real-time monitoring on site, etc. can be done.

Here, the case where the image processing apparatus of the present embodiment is installed in an unmanned mobile body that performs monitoring while moving within the installation site has been described as an example, but the image processing apparatus of the present embodiment is applied to this. is not limited to The image processing apparatus of the present embodiment may be installed in a fixed imaging apparatus such as a security camera or a surveillance camera, or may be installed in a portable imaging apparatus such as a digital camera or various terminals. Further, the image processing apparatus of this embodiment may be one specialized for performing image processing on images transmitted by wire or wirelessly from an imaging apparatus or other apparatus, or images stored in various storage media. good. That is, the image processing apparatus of this embodiment has a degree of freedom, and various design changes are possible.

FIG. 2 is a functional block diagram showing an example of internal components of the control unit (image processing device, dynamic range correction device, gradation correction device) 11X. The image processing method and image processing program of the present embodiment are implemented by causing a computer (CPU) constituting the control section 11X to execute various processing steps (processing steps).

The control unit 11X has a division unit 11A, a replacement unit 11B, a first median calculation unit 11C, a second median calculation unit 11D, and a conversion unit 11E.

The dividing unit 11A divides the image captured by the imaging unit 13X into a plurality of pixels. FIG. 3A shows a plurality of pixels divided by the dividing section 11A. In FIG. 3A, nine pixels 1-1, 1-2, 1-3, 2-1, 2-2, 2-3, 3-1, 3-2, 3-3 are shown for simplicity of explanation. is illustrated as an example.

The replacing unit 11B replaces pixels having a saturation threshold value S1 or more among the plurality of pixels with the replacement pixels V1. FIG. 3B shows pixels replaced by the replacement unit 11B. FIG. 3B illustrates a case where two pixels 1-1 and 3-2 out of nine pixels are equal to or higher than the saturation threshold value S1 and the two pixels 1-1 and 3-2 are replaced with the replacement pixel V1. I am drawing. There is a degree of freedom in how to set each value of the saturation threshold S1 and the replacement pixel V1. For example, it may be changed in real time (dynamically) according to the luminance distribution.

The first median value calculator 11C calculates a first median value m1 of pixels da below the saturation threshold value S1 among the plurality of pixels. In the example of FIG. 3B, seven pixels 1-2, 1-3, 2-1, 2-2, 2-3, 3-1, and 3-3 correspond to the pixels da below the saturation threshold S1. The first median calculator 11C calculates the first median m1 by median filtering. More specifically, the first median calculator 11C calculates pixels da (seven pixels 1-2, 1-3, 2-1, 2-2, 2-3, 3-1, For 3-3), a two-dimensional median filtering process or a three-dimensional median filtering process of setting a target pixel and peripheral pixels and calculating a first median value m1 by calculation based on the target pixel and peripheral pixels is executed. At that time, the two pixels 1-1 and 3-2 that were replaced with the replacement pixel V1 because the saturation threshold S1 or higher were not used, or the pixels were interpolated by an arbitrary calculation method. You may use it as a complementary pixel.

The second median calculator 11D calculates a second median m2 different from the first median m1. For example, the second median value calculator 11D preferably calculates the second median value m2 as a value V1/2 that is half the pixel value of the replacement pixel V1. There is a degree of freedom in how the second median value calculator 11D calculates the second median value m2, and various design changes are possible. Theoretically, the first median value m1 calculated by the first median value calculation unit 11C and the second median value m2 calculated by the second median value calculation unit 11D may be the same value. However, even in this case, the first median value m1 and the second median value m2 are different median values calculated separately by the first median calculator 11C and the second median calculator 11D. can be equivalent to

The conversion unit 11E converts pixels da (seven pixels 1-2, 1-3, 2-1, 2-2, 2-3, 3-1, 3-3) below the saturation threshold value S1 among the plurality of pixels. to pixels db referenced to a second median value m2 different from the first median value m1.

More specifically, the conversion unit 11E converts pixels da (seven pixels 1-2, 1-3, 2-1, 2-2, 2-3, 3-1, 3 -3), converting the pixel da1 having a pixel value smaller than the first median value m1 into the pixel db1 having a pixel value smaller than the second median value m2, and converting the pixel da2 having a pixel value equal to or greater than the first median value m1 is converted to a pixel db2 having a pixel value equal to or greater than the second median value m2.

FIG. 3C shows pixels converted by the conversion unit 11E. In FIG. 3C, among pixels da (seven pixels 1-2, 1-3, 2-1, 2-2, 2-3, 3-1, 3-3) below the saturation threshold S1 of a plurality of pixels, Pixels 1-2, 1-3, 2-3, and 3-1 are pixels da1 with pixel values smaller than the first median value m1, and pixels 2-1, 2-2, and 3-3 are the first median value m1. The pixel da2 has a pixel value equal to or greater than the median value m1 of . A pixel da1 having a pixel value smaller than the first median value m1 of pixels 1-2, 1-3, 2-3, and 3-1 is converted to a pixel db1 having a pixel value smaller than the second median value m2. , pixels 2-1, 2-2, and 3-3, the pixel da2 having a pixel value equal to or greater than the first median value m1 is converted into the pixel db2 having a pixel value equal to or greater than the second median value m2.

Here, there is a degree of freedom in how to set the pixels db1 and db2 after conversion by the conversion unit 11E, and various design changes are possible. For example, it may be determined in advance based on a predetermined arithmetic expression using the second median value m2 as a parameter, or may be changed in real time (dynamically) according to the imaging environment (eg, luminance distribution).

FIG. 4 is a diagram showing an example of each calculation of image processing according to the present embodiment. In FIG. 4, R means the maximum pixel value in the image. Also, in FIG. 4, S2 means the pixel value of the replacement pixel V1, and here, the same case as R is exemplified (S2=R).

In FIG. 4, conditional expression (1) expresses that pixels having a saturation threshold value S1 or more among a plurality of pixels are replaced with replacement pixels V1. That is, if a pixel having a pixel value greater than S1 existing at coordinates (x, y) is defined as pixel ds, then pixel ds existing at coordinates (x, y) is converted to pixel v1 having pixel value S2. be.

In FIG. 4, among pixels da below the saturation threshold S1 of a plurality of pixels, a pixel da1 with a pixel value smaller than the first median value m1 is converted into a pixel db1 with a pixel value smaller than the second median value m2. This is expressed by conditional expression (2). That is, if a pixel having a pixel value smaller than m1 at coordinates (x, y) is defined as pixel da1, the pixel da1 at coordinates (x, y) has a pixel value m2/m1*da1(x, y).

In FIG. 4, among pixels da below the saturation threshold value S1 of a plurality of pixels, pixels da2 with pixel values equal to or greater than the first median value m1 are converted into pixels db2 with pixel values equal to or greater than the second median value m2. This is expressed by conditional expression (3). That is, if a pixel da2 exists at coordinates (x, y) and has a pixel value equal to or greater than m1, the pixel da2 at coordinates (x, y) has a pixel value of (S2-m2)/(S1 -m1)*(da2(x,y)-m1)+m2 with pixel db2.

FIG. 5 is a flowchart showing an example of image processing according to this embodiment. The execution subject of each processing process (each processing step) in FIG. 5 is a computer (CPU) that mainly constitutes the control unit (image processing apparatus) 11X.

In step ST1, an image captured by the imaging unit 13X is acquired. In step ST2, various initial settings necessary for image processing are performed.

At step ST3, addresses of saturated pixels ds, which are pixels having a saturation threshold value S1 or more among a plurality of pixels obtained by dividing the image obtained at step ST1, are extracted and stored.

In step ST4, a first median value m1 of remaining pixels da less than the saturation threshold value S1 among the plurality of pixels obtained by dividing the image obtained in step ST1 is calculated.

In step ST5, pixels da below the saturation threshold S1 among the plurality of pixels are converted into pixels db based on a second median value m2 different from the first median value m1. Specifically, among the pixels da less than the saturation threshold S1 of the plurality of pixels, the pixel da1 having the pixel value smaller than the first median value m1 is converted into the pixel db1 having the pixel value smaller than the second median value m2. A pixel da2 having a pixel value equal to or greater than the median value m1 of 1 is converted into a pixel db2 having a pixel value equal to or greater than the second median value m2. Here, in step ST5 in the first loop, the default second median value m2 can be used. As will be described later, the second median value m2 is set (updated) in step ST9, so in step ST5 in the second and subsequent loops, the second median value m2 set (updated) in step ST9 is can be used.

In step ST6, among the plurality of pixels, the saturated pixel ds, which is the pixel equal to or higher than the saturation threshold value S1, is replaced with the replacement pixel V1. Here, at step ST6 in the first loop, the default replacement pixel V1 can be used. As will be described later, since the replacement pixel V1 is set (updated) in step ST9, the replacement pixel V1 set (updated) in step ST9 can be used in step ST6 in the second and subsequent loops.

At step ST7, an image is displayed. At step ST8, it is determined whether or not the image displayed at step ST7 needs trimming. If it is determined that trimming is necessary (step ST8: Yes), the process proceeds to step ST9. If it is determined that trimming is unnecessary (step ST8: No), the process proceeds to step ST10.

In step ST9, the second median value m2 and the replacement pixel V1 are set (updated), and the process returns to step ST5. In this manner, steps ST5 to ST9 are repeated while setting (updating) the second median value m2 and the replacement pixel V1 until it is determined in step ST8 that trimming is unnecessary.

At step ST10, it is determined whether or not to save the image. If it is determined that the image should be saved (step ST10: Yes), the process proceeds to step ST11 to save the image, and then the process ends. If it is determined not to save the image (step ST10: No), the process ends without saving the image.

As described above, in the present embodiment, among a plurality of pixels, pixels equal to or greater than the saturation threshold S1 are replaced with replacement pixels V1, and among the plurality of pixels, the first median value m1 of pixels da below the saturation threshold S1 is Among the plurality of pixels, pixels da below the saturation threshold S1 are converted into pixels db based on a second median value m2 different from the first median value m1. Further, among the pixels da below the saturation threshold value S1 of the plurality of pixels, the pixel da1 having the pixel value smaller than the first median value m1 is converted into the pixel db1 having the pixel value smaller than the second median value m2, and the first A pixel da2 having a pixel value equal to or greater than the median value m1 is converted into a pixel db2 having a pixel value equal to or greater than the second median value m2.

As a result, an image with high visibility can be obtained regardless of the shooting conditions. In particular, by appropriately performing dynamic range correction or tone correction on a low tone area in an image, the low tone area can be enlarged and converted to make it easier to see. In other words, the Gaussian distribution centered on the median value (first median value) m1 of the pixel values in areas other than the saturated region is the median value (second median value) after conversion. value) converted to a Gaussian distribution centered at m2. Therefore, it is possible to obtain an image with high visibility by extending only the low gradation area from an image having a bias toward high gradation and low gradation.

For example, when an image is acquired indoors, the indoor image tends to have low gradation and is not suitable for recording due to the influence of high-intensity external light (incident light) such as sunlight or reflected light. In particular, at construction sites, civil engineering sites, etc., there are many areas where concrete or the like is exposed indoors, and there are many materials with low reflectance from the start, so images with low gradation are likely to occur. Also, at construction sites and civil engineering sites, the environment is the same as outdoors when there is no roof, but when there is a roof, there are no windows or walls. This often results in an image or video that includes dark areas in the background. The images and videos to be shot are assumed to be taken during the daytime, but because the lighting is weak in the field, the images and videos tend to include many dark areas and extremely bright areas. Under such circumstances, when acquiring images for work progress and process progress management, if even a part of the outside light or reflected light enters the angle of view, the low gradation area will be further grayed out due to its influence. Distortion may occur. Therefore, there is concern that it is not suitable as an image to be left as a record.

In this embodiment, the above technical problems are solved, and regardless of the shooting situation (for example, indoors and outdoors, brightness level, shooting location such as construction site and civil engineering site), visibility of low gradation area in particular is improved. An image with excellent image quality can be obtained.

Incidentally, in the above-mentioned Patent Document 1, the color representative value of a plurality of components obtained for each pixel block is processed to obtain the representative value of one component, and a certain range is enlarged so as not to destroy the histogram information. , the saturation region is set arbitrarily. On the other hand, the image processing (dynamic range correction, gradation correction) of the present embodiment includes a first median value m1 of pixel values other than the saturated region and a second median value different from the first median value m1. It is based on the value m2, and is not converted so that the information of the histogram is not destroyed. In some respects, it differs from the technology of Patent Literature 1 (it draws a line).

6A and 6B are conceptual diagrams showing an example of an image subjected to the image processing of this embodiment and an image of a comparative example. FIG. 6A is a conceptual diagram showing an example of an image of a comparative example, and FIG. 6B is a conceptual diagram showing an example of an image subjected to image processing according to this embodiment. It can be read that the image in FIG. 6B is superior to that in FIG. 6A in that extremely bright portions and extremely dark portions do not coexist, and the visibility of the low gradation region is particularly excellent.

Several embodiments are described above. However, it should be understood that the embodiments are not limited to the embodiments described above, but encompass various variations and alternatives of the embodiments described above. For example, it will be appreciated that various embodiments may be embodied with varying elements without departing from the spirit and scope thereof. Also, it will be understood that various embodiments can be implemented by appropriately combining a plurality of constituent elements disclosed in the above-described embodiments. Furthermore, various embodiments can be implemented by deleting some components from all the components shown in the embodiments or by adding some components to the components shown in the embodiments. It will be understood by those skilled in the art.

10 robot 11 main unit 11X control unit (image processing device, dynamic range correction device, gradation correction device)
11A division unit 11B replacement unit 11C first median calculation unit 11D second median calculation unit 11E conversion unit 12 caterpillar unit 13 head 13X photographing unit (photographing lens, camera lens)

Claims (6)

a dividing unit that divides an image into a plurality of pixels;
a replacing unit that replaces pixels equal to or greater than a saturation threshold among the plurality of pixels with replacement pixels;
a first median calculator that calculates a first median of pixels below the saturation threshold among the plurality of pixels;
a conversion unit that converts pixels below the saturation threshold among the plurality of pixels into pixels based on a second median value different from the first median value;
An image processing device comprising: The conversion unit converts pixels having a pixel value smaller than the first median value among pixels having a pixel value smaller than the saturation threshold among the plurality of pixels into pixels having a pixel value smaller than the second median value, and converting pixels with pixel values greater than or equal to the median of 1 to pixels with pixel values greater than or equal to the second median;
2. The image processing apparatus according to claim 1, wherein: a second median calculation unit that calculates the second median value as a half value of the pixel values of the replacement pixels;
3. The image processing apparatus according to claim 1, wherein: The first median calculation unit calculates the first median by median filtering,
4. The image processing apparatus according to any one of claims 1 to 3, characterized by: dividing the image into a plurality of pixels;
replacing pixels equal to or greater than a saturation threshold among the plurality of pixels with replacement pixels;
calculating a first median value of pixels below the saturation threshold among the plurality of pixels;
converting pixels below the saturation threshold among the plurality of pixels into pixels based on a second median value different from the first median value;
An image processing method characterized by comprising: dividing the image into a plurality of pixels;
replacing pixels equal to or greater than a saturation threshold among the plurality of pixels with replacement pixels;
calculating a first median value of pixels below the saturation threshold among the plurality of pixels;
converting pixels below the saturation threshold among the plurality of pixels into pixels based on a second median value different from the first median value;
An image processing program characterized by causing a computer to execute

Images