JP6655504B2 - Image processing apparatus, image processing system, moving object, and image processing method - Google Patents

Description

  The present invention relates to an image processing device, an image processing system, a moving object, and an image processing method.

  2. Description of the Related Art Conventionally, there has been known an image processing apparatus that combines images captured by a plurality of cameras to create one continuous combined image. For example, the image processing device described in Patent Literature 1 looks down on a vehicle and a road surface around the vehicle by combining images obtained by capturing four images of the front, rear, right, and left of the vehicle. Generate a composite image representing the situation.

  On the other hand, when the illuminance in the imaging range of the camera is low, such as in the evening or at night, an imaging device that irradiates a subject with near-infrared light to capture an image without giving dazzling or discomfort to the subject person is known. Have been. For example, the imaging device described in Patent Literature 2 transmits near-infrared light of 800 nm or more in which the relative sensitivities of pixels of each color are substantially equal, and blocks infrared light excluding near-infrared light in this wavelength range. It has a filter and images a subject irradiated with near-infrared light.

JP 2009-17020A JP-A-10-108206

  However, such an image pickup apparatus having a filter that transmits near-infrared light in a predetermined wavelength range and blocks infrared light excluding near-infrared light in a predetermined wavelength range, as in the above-described related art, has been proposed. When an image of a subject irradiated with infrared light is captured, the saturation is lower than that of an image of a subject not irradiated with near infrared light because the relative sensitivity of pixels of each color is almost equal in near infrared light. I do.

  Therefore, in an image processing apparatus that generates a synthesized image by synthesizing a plurality of images, an image captured by an imaging apparatus having a filter that transmits near-infrared light and blocks infrared light other than near-infrared light Is used, the saturation of an image of a subject irradiated with near-infrared light is different from that of an image of a subject not irradiated with near-infrared light. Therefore, the synthesized image obtained by synthesizing these images has different saturation for each region, and may be unnatural when viewed as a whole.

  Therefore, an object of the present invention focused on these points is that even when a part of the subject is irradiated with near-infrared light, the saturation is balanced, and the referrer feels uncomfortable. It is an object of the present invention to provide an image processing apparatus, an image processing system, a moving object, and an image processing method capable of obtaining an image for generating a composite image with less image quality.

An image processing apparatus that solves the above-described problem is configured to acquire a plurality of images captured by a plurality of cameras each including a filter that transmits visible light and selectively transmits infrared light in a predetermined wavelength range. And calculating the saturation information of each of the plurality of images obtained by the obtaining unit, and based on the respective pieces of saturation information, the saturation of pixels constituting at least one image of the plurality of images. And a processor configured to reduce a difference between the saturation information of the plurality of images to be equal to or less than a first threshold having a predetermined value . The processor generates a synthesized image by synthesizing the image with the reduced saturation and an image different from the at least one of the plurality of images.

An image processing system that solves the above problem includes a plurality of cameras including a filter that transmits visible light and selectively transmits infrared light in a predetermined wavelength range, and a plurality of images that are respectively captured by the plurality of cameras. An acquisition unit that acquires the image of the plurality of images, calculates saturation information of each of the plurality of images acquired by the acquisition unit, and, based on the respective saturation information, determines at least one image of the plurality of images. By reducing the saturation of the constituent pixels, the difference between the saturation information of the plurality of images is made equal to or less than a first threshold having a predetermined value, and the image having the reduced saturation and the plurality of images A processor that combines the at least one image with an image different from the at least one image to generate a combined image .

A moving body that solves the above-mentioned problem transmits a visible light, and has a plurality of cameras each having a filter that selectively transmits infrared light in a predetermined wavelength range, and a plurality of cameras each of which is imaged by the plurality of cameras. An acquiring unit for acquiring an image, calculating saturation information of each of the plurality of images acquired by the acquiring unit, and configuring at least one of the plurality of images based on the respective saturation information By reducing the saturation of the pixel to be performed, the difference between the saturation information of the plurality of images is set to be equal to or less than a first threshold having a predetermined value, and the image having the reduced saturation, and And an image processing apparatus having a processor that synthesizes an image different from the at least one image to generate a synthesized image .

An image processing method for solving the above problems is an image processing method executed by an image processing apparatus, wherein the image processing apparatus transmits visible light and selectively transmits infrared light in a predetermined wavelength range. Acquiring a plurality of images respectively captured by a plurality of cameras including a filter for causing the image processing apparatus to calculate saturation information of each of the plurality of acquired images, based on the respective saturation information Reducing the saturation of the pixels constituting at least one of the plurality of images to reduce the difference between the saturation information of the plurality of images to a first threshold having a predetermined value or less, A combined image is generated by combining the image with reduced saturation and an image different from the at least one of the plurality of images .

  According to an embodiment of the present invention, even when a part of a subject is irradiated with near-infrared light, a composite image with uniform saturation and less discomfort for a referrer is generated. It is possible to obtain an image for.

FIG. 1 is a functional block diagram of the image processing system according to the first embodiment. FIG. 2 is a top view of a moving body on which the imaging device, the image processing device, and the display device illustrated in FIG. 1 are mounted. FIG. 3 is a flowchart illustrating a processing flow of the image processing system according to the first embodiment. FIG. 4 is a functional block diagram of the image processing system according to the second embodiment. FIG. 5 is a flowchart illustrating a processing flow of the image processing system according to the second embodiment.

  Hereinafter, a first embodiment of the present invention will be described with reference to the drawings.

  As shown in FIG. 1, the image processing system 1 according to the first embodiment includes an imaging device (camera) 10a, 10b, 10c, 10d, an image processing device 20, a display device 30, and the like. The image processing device 20 is connected to the cameras 10a, 10b, 10c, 10d, and the display device 30 via a communication network to transmit and receive information. The image processing device 20 may be connected to a driving support device or the like via a communication network to receive driving support information. The driving support information is information indicating, for example, a target parking position, a predicted course line, a retreat start position, and the like for supporting a driver's operation. The image processing device 20 may be provided in the same housing as the display device 30. In that case, the communication interfaces 24 and 31 may not be provided, and the functions executed by the processor 22 and the display processor 32 may be executed by the same processor. The display device 30 may also serve as a display unit such as a driving support device.

  2, the cameras 10a, 10b, 10c, and 10d, the image processing device 20, and the display device 30 are mounted on the moving body 4. Here, the “moving body” means, for example, a vehicle running on a road, such as a passenger car, a truck, or a bus. In the following, any imaging device among the cameras 10a, 10b, 10c, and 10d is referred to as "imaging device (camera) 10".

  The camera 10 is installed so as to capture an image around the moving body 4. Explaining using the example shown in FIG. 2, the cameras 10a, 10b, 10c, and 10d are installed so as to capture images of the front, right, rear, and left of the moving body 4, respectively. Each camera is installed such that a part of the imaging range of the camera is common to a part of the imaging range of another camera.

  The imaging range of the camera 10a is A1, A2, and A3 shown in FIG. 2, and the imaging range of the camera 10b is A3, A5, and A8. That is, A3 which is a part of the imaging range of the camera 10a is a part of the imaging range of the camera 10b.

  Similarly, the imaging range of the camera 10c is A6, A7, and A8. That is, A8 which is a part of the imaging range of the camera 10b is a part of the imaging range of the camera 10c.

  The imaging range of the camera 10d is A1, A4, and A6. That is, A1 which is a part of the imaging range of the camera 10d is a part of the imaging range of the camera 10a.

  The camera 10 includes an imaging lens 11, a filter 12, an imaging element 13, a signal processing unit 14, and the like. Specifically, as shown in FIG. 1, the camera 10a includes an imaging lens 11a, a filter 12a, an imaging element 13a, a signal processing unit 14a, and the like. Similarly, the camera 10b includes an imaging lens 11b, a filter 12b, an imaging element 13b, a signal processing unit 14b, and the like. The camera 10c includes an imaging lens 11c, a filter 12c, an imaging device 13c, a signal processing unit 14c, and the like. The camera 10d includes an imaging lens 11d, a filter 12d, an imaging device 13d, a signal processing unit 14d, and the like. Is done.

  The imaging lens 11 is a lens that condenses incident light so that an image is formed on a light receiving surface of the imaging element 13. The imaging lens 11 may be configured by, for example, a fish-eye lens or an ultra-wide-angle lens. The imaging lens 11 may be composed of a single lens, or may be composed of a plurality of lenses.

  The filter 12 can be arranged on the image side of the imaging lens 11 so that the surface of the filter 12 is substantially perpendicular to the optical axis of the imaging lens 11. The filter 12 transmits visible light and selectively transmits near-infrared light in a predetermined wavelength range. The predetermined wavelength range is a range of wavelengths of near-infrared light in which the relative sensitivities of the imaging elements 13 of the respective colors are equal, and includes, for example, 860 nm.

  Pixels forming an image, which are generated by the light passing through the filter 12 being formed on the image sensor 13, are represented in color by visible light. Then, as described above, near-infrared light in a predetermined wavelength range arrives at the image sensor 13 together with visible light. Since the signal intensity of each color pixel due to the near-infrared light is substantially equal, the saturation of the pixel represented by the color is lower than when no near-infrared light arrives.

  For example, when near infrared light is applied to only a part of the range captured by each of the cameras 10a, 10b, 10c, and 10d, the wavelength range described above is applied only to the camera 10 that captures the range irradiated with the infrared light. Is transmitted through the filter 12 and reaches the image sensor 13. Therefore, a first image of the range of the cameras 10a, 10b, 10c, and 10d irradiated with near-infrared light, and an image of the range of the cameras 10a, 10b, 10c, and 10d not irradiated with near-infrared light. There is a case where the saturation of the second image is different from that of the second image so as to cause a sense of incongruity when viewed from human eyes. In particular, when the processor 22 generates one combined image by combining the first image and the second image, the area related to the first image and the second image in one combined image are generated. Since the saturation is different from the region according to the above, the user who has visually recognized the composite image has a sense of discomfort.

  The imaging device 13 is an imaging device that captures an image formed by the imaging lens 11. The image sensor 13 includes a charge-coupled device (CCD) image sensor and a complementary metal oxide semiconductor (CMOS) image sensor. The imaging device 13 includes a color filter for capturing a color image. In the color filters, filters of a plurality of colors such as red, green, blue, or cyan, magenta, and yellow are arranged for each light receiving element. The color filters may be arranged in a Bayer arrangement.

  The signal processing unit 14 is a processor that processes an image, for example, a dedicated microprocessor formed to execute a specific function, or a processor that executes a specific function by reading a specific program.

  The signal processing unit 14 generates an image signal representing an image formed by the image sensor 13. Further, the signal processing unit 14 may perform arbitrary processing such as distortion correction and gamma correction on the image.

  The communication interface 15 transmits an image signal representing an image processed by the signal processing unit 14 to the image processing device 20.

  The image processing device 20 includes an acquisition unit 21, a processor 22, a memory 23, a communication interface 24, and the like.

  The acquisition unit 21 is an interface with the camera 10 that acquires an image represented by the image signal by receiving the input of the image signal generated by the signal processing unit 14.

  The processor 22 is a processor that processes an image acquired by the acquisition unit 21, and is, for example, a dedicated microprocessor formed to execute a specific function or a general-purpose microprocessor that executes a specific function by reading a specific program. It is a CPU (Central Processing Unit).

  The processor 22 is configured so that the difference between the brightness information of the plurality of images captured by the plurality of cameras 10a, 10b, 10c, and 10d, and the plurality of images acquired by the acquisition unit 21 is equal to or smaller than a predetermined threshold (third threshold). adjust. The brightness information is a value indicating a characteristic regarding the brightness of the entire image. The brightness information is calculated from the brightness (brightness) of each pixel constituting the image, and is, for example, a statistical value such as an average value, a median value, a mode value, and a sum of the brightness of each pixel. Since the plurality of cameras 10a, 10b, 10c, and 10d have sensitivity to near-infrared light, near-infrared light as well as visible light contributes to brightness information.

  Specifically, the processor 22 determines whether or not the difference between the brightness information of the plurality of images captured by the plurality of cameras 10a, 10b, 10c, and 10d and acquired by the acquisition unit 21 is greater than a third threshold value. Is determined. Then, when the difference in the brightness information is larger than the third threshold, the brightness of the pixels constituting one or more images is changed so that the difference in the brightness information of all the images is equal to or less than the third threshold. I do. If the difference in the brightness information is less than the third threshold, the processor 22 does not change the brightness of any of the pixels.

  Here, the third threshold value is a value that allows the brightness of a plurality of images to be considered to be substantially the same as seen by human eyes when the difference between the brightness information is equal to or less than this value. It is determined in advance.

  The method by which the processor 22 changes the brightness information of each image is arbitrary. For example, the processor 22 may adjust the brightness information of another image so as to match the brightness information of the other image with the image having the highest brightness information among the plurality of images. The brightness of the pixels forming the image having the highest brightness information may be reduced, or the brightness of the pixels forming the image having the highest brightness information may be reduced so as to match the brightness information with another image. Further, the processor 22 may calculate an average value of the brightness information of the plurality of images, and may change the brightness of the pixels constituting each image so that the brightness information of the plurality of images becomes the average value. .

  The processor 22 calculates saturation information of a plurality of images captured by the cameras 10a, 10b, 10c, and 10d, respectively, and the brightness information of which is adjusted as described above. The saturation information is a value indicating a characteristic regarding the vividness of the entire image. The saturation information is calculated from the saturation of each pixel constituting the image, and is, for example, a statistical value such as an average value, a median value, a mode value, and a sum of the saturation of each pixel. The saturation of each pixel is calculated from the RGB values of each pixel.

  In addition, the processor 22 controls the difference between the saturation information of the plurality of images to be within a predetermined range based on the saturation information of the plurality of images respectively captured by the plurality of cameras 10a, 10b, 10c, and 10d. In addition, the saturation of pixels constituting at least one image is reduced. Here, a process in which the processor 22 reduces the saturation of a pixel based on a plurality of pieces of saturation information will be described in detail.

  The processor 22 calculates a difference between the saturation information of the plurality of images, and based on the difference between the saturation information of the first image and the second image different from the first image among the plurality of images. It is determined whether or not the saturation of the pixels forming the image is to be reduced.

  For example, the processor 22 sets the image with the highest brightness information among the images acquired by the acquisition unit 21 as the first image, and calculates the saturation information based on a difference between the first image and a second image different from the first image. Is calculated. In the case where the left area of the moving object 4 surrounded by the two-dot broken line ellipse in FIG. 2 is illuminated by the streetlight constituted by the halogen light, the camera 10d among the plurality of cameras 10a, 10b, 10c, and 10d. The brightness information of the captured image Id is the highest. Further, since the light irradiated by the halogen light contains much light having a wavelength in the near-infrared region, the saturation information of the image Id captured by the camera 10d is lower than that when the light is not irradiated by the halogen light. ing.

  Accordingly, the processor 22 determines that the saturation information SPd of the first image Id having the highest brightness information and the images Ia, Ib, and Ic different from the first image Id (in the present example, these images are the second images, respectively). The difference from each of the chroma information SPa, SPb, and SPc (corresponding to an image) is calculated. Specifically, the processor 22 calculates a difference | SPd−SPa | between the saturation information SPd of the first image Id and the saturation information SPa of the image Ia captured by the camera 10a. The processor 22 also calculates the difference | SPd−SPb | between the saturation information SPd of the image Id and the saturation information SPb of the image Ib captured by the camera 10b, and the saturation information SPd of the image Id, and captures the image by the camera 10c. The difference | SPd−SPc | between the obtained image Ic and the saturation information SPc is calculated. Then, the processor 22 determines whether or not any of the calculated differences in the saturation information is greater than a predetermined threshold (first threshold). Then, the processor 22 determines that the saturation of the pixels constituting at least one of the plurality of images is reduced when any of the differences in the saturation information is equal to or greater than the first threshold. Here, the first threshold value is a value at which the vividness of a plurality of images can be considered to be substantially the same as seen by the human eyes when the difference between the saturation information is equal to or less than this value. It is determined in advance.

  The method of calculating the difference between the saturation information of a plurality of images is not limited to this, and may be calculated as follows. For example, the processor 22 calculates a difference between the saturation information in all combinations of an arbitrary first image captured by the plurality of cameras 10a, 10b, 10c, and 10d and a second image different from the first image. It is calculated, and if any of the differences is equal to or greater than the first threshold value, it is determined that the saturation of pixels constituting at least one of the plurality of images is to be reduced.

  Specifically, the processor 22 calculates a difference | SPa−SPb | between the saturation information SPa of the image Ia captured by the camera 10a and the saturation information SPb of the image captured by the camera 10b. Similarly, the processor 22 uses the saturation information SPc of the image captured by the camera 10c and the saturation information SPd of the image captured by the camera 10d to generate | SPa-SPc |, | SPa-SPd |, | SPb-SPc |, | SPb-SPd |, and | SPc-SPd | are calculated, respectively. Then, the processor 22 determines that the saturation of the pixels constituting at least one of the plurality of images is reduced when any of the differences is larger than the first threshold.

  Here, the process in which the processor 22 lowers the saturation of the pixels constituting the image will be described in detail.

  The processor 22 reduces the saturation such that the saturation information of the second image different from the first image is matched with the first image having the lowest saturation information among the images Ia, Ib, Ic, and Id. . When the saturation information Spd of the image Id among the images Ia, Ib, Ic, Id is the lowest, the difference between the saturation information Spa, Spb, Spc of each of the images Ia, Ib, Ic and the saturation information Spd of the image Id. Is less than or equal to the first threshold value, the saturation of the pixels constituting the images Ia, Ib, Ic is reduced. At this time, the saturation may be reduced so that the saturation reduction rate of each pixel may be the same, or the saturation may be reduced based on another arbitrary algorithm.

  Further, the processor 22 reduces the saturation of the pixels forming at least one of the plurality of images to reduce the saturation information of the plurality of images to a first threshold value or less, and the plurality of images are reduced. A composite image is generated by compositing.

  Further, the processor 22 outputs a synthesized image generated by synthesizing a plurality of images to the communication interface 24. In addition, the processor 22 may output to the communication interface 24 an image obtained by superimposing the driving assistance information indicating the target parking position, the predicted course line, the retreat start position, and the like on the composite image, received from the driving assistance device.

  The communication interface 24 receives, from each of the cameras 10a, 10b, 10c, and 10d, image information representing an image captured by the plurality of cameras 10a, 10b, 10c, and 10d. Further, the communication interface 24 receives driving support information from the driving support device. In addition, the communication interface 24 transmits to the display device image information representing the composite image output by the processor 22 or an image in which driving assistance information is superimposed on the composite image. The communication interface 24 can be a physical connector, a wireless communication device, or the like. Physical connectors include electrical connectors, optical connectors, and electromagnetic connectors. The wireless communication device includes a wireless communication device conforming to each standard including Bluetooth (registered trademark) and IEEE 802.11, and an antenna.

  The display device 30 includes a communication interface 31, a display processor 32, a display panel 33, and the like.

  The communication interface 31 receives, from the image processing device 20 via the communication interface 24, a composite image or image information representing an image in which driving assistance information is superimposed on the composite image. The communication interface 31, like the communication interface 24, can be a physical connector, a wireless communication device, or the like.

  The display processor 32 causes the display panel 33 to display the image information received by the communication interface 31.

  Next, an image processing method of the image processing system 1 according to the first embodiment will be described with reference to FIG.

  First, the cameras 10a, 10b, 10c, and 10d each capture an image of a subject around the moving body 4 and generate images Ia, Ib, Ic, and Id (step S11).

  When the images Ia, Ib, Ic, and Id are generated in step S11, the acquisition unit 21 of the image processing device 20 outputs the image information representing the images Ia, Ib, Ic, and Id to the respective cameras 10a, 10b, 10c, and Obtained from 10d (step S12).

  After acquiring the image information in step S12, the processor 22 calculates the brightness information of each image based on the brightness of the pixels constituting the images Ia, Ib, Ic, Id (step S13).

  When the brightness information is calculated in step S13, the processor 22 changes the brightness of at least one pixel constituting the image so that the brightness information difference between the images is equal to or less than the third threshold value. Is adjusted (step S14).

  When the brightness information is adjusted in step S14, the processor 22 calculates saturation information for each of the plurality of images whose brightness information has been adjusted (step S15).

  When the saturation information is calculated in step S15, it is determined whether or not the difference between the saturation information of the plurality of images is equal to or less than a first threshold (step S16).

  If it is determined in step S16 that the difference between the saturation information of the plurality of images is larger than the first threshold, the saturation of the pixels forming at least one of the plurality of images is reduced (step S17). Specifically, a second image different from the first image such that a difference from the first image having the lowest saturation information among the images Ia, Ib, Ic, and Id is equal to or less than a first threshold. The saturation of the pixels constituting each is reduced.

  When the difference in the saturation information of the plurality of images is reduced to the first threshold value or less by reducing the saturation of the pixel in step S17, the processor 22 combines the plurality of images to generate a combined image. (Step S18). At this time, the processor 22 may generate an image in which the driving assistance information received from the driving assistance device, including the target parking position for parking, the predicted course line, the retreat start position, and the like, received from the driving assistance device are superimposed.

  When the combined image or the image in which the driving assistance information is superimposed on the combined image is generated in step S18, the processor 22 outputs the image information representing the generated image and transmits the image information to the display device 30 via the communication interface 24. (Step S19).

  When the image information is output to the display device 30 in step S19, the display device 30 displays the image (step S20).

  In the first embodiment, even if the saturation of one of the plurality of images captured by the cameras 10a, 10b, 10c, and 10d is reduced by near-infrared light, the saturation of each of the plurality of images is reduced. Since the saturation of at least one pixel constituting the image is reduced based on the information, the saturation information of all the images can be made the same. Therefore, when a synthesized image is generated by synthesizing these images, the viewer can refer to the images without feeling a difference in coloring for each region.

  In the first embodiment, the saturation of any one of the plurality of images adjusted so that the difference in the brightness information is equal to or less than the third threshold is reduced. Therefore, for example, when only some of the images are brightened by a streetlight that emits light including a large amount of near-infrared light, the brightness of all the images is substantially the same as seen from the eyes of the reference person. And the saturation information can be made comparable. At night or in the evening, when only a part of the imaging range of the plurality of cameras 10a, 10b, 10c, and 10d is illuminated by a streetlight such as a halogen light containing a large amount of near-infrared light, the subject in the imaging range is The captured image is brighter than other images and has a lower saturation. Therefore, when the brightness information is adjusted for a plurality of images as described above, and the saturation information is further set to the same level, when a synthesized image is generated by synthesizing these images, the synthesized image is The referring person can refer to the image without feeling any difference in brightness and vividness between the regions.

  Hereinafter, a second embodiment of the present invention will be described with reference to the drawings.

  As shown in FIG. 4, the image processing system 2 according to the second embodiment includes cameras 10a, 10b, 10c, and 10d, an image processing device 20, and a display, like the image processing system 1 according to the first embodiment. It is configured to include the device 30 and the like. Further, the image processing system 2 is mounted on the moving body 4 similarly to the image processing system 2 according to the first embodiment. Further, the image processing system 2 according to the first embodiment is different in that the image processing system 2 further includes illumination devices 40a, 40b, 40c, and 40d that irradiate the imaging ranges of the cameras 10a, 10b, 10c, and 10d with near-infrared light. Different from the processing system 2. In the following description, any lighting device among the lighting devices 40a, 40b, 40c, and 40d is referred to as “lighting device 40”.

  In FIG. 4, the configurations of the cameras 10b, 10c, and 10d are omitted, but the camera 10b includes an imaging lens 11b, a filter 12b, an imaging element 13b, a signal processing unit 14b, and the like, as in the first embodiment. It consists of. The camera 10c includes an imaging lens 11c, a filter 12c, an imaging device 13c, a signal processing unit 14c, and the like. The camera 10d includes an imaging lens 11d, a filter 12d, an imaging device 13d, a signal processing unit 14d, and the like. Is done.

  The illumination device 40 irradiates at least a part of the range captured by the camera 10 with near-infrared light. The lighting device 40 includes a communication interface 41, an irradiation control unit 42, a light emitting unit 43, and the like. Specifically, the lighting device 40a includes a communication interface 41a, an irradiation control unit 42a, a light emitting unit 43a, and the like. Also, although omitted in FIG. 4, the lighting device 40b is similarly configured to include a communication interface 41b, an irradiation control unit 42b, a light emitting unit 43b, and the like. The lighting device 40c includes a communication interface 41c, an irradiation control unit The lighting device 40d includes a communication interface 41d, an irradiation control unit 42d, a light emitting unit 43d, and the like.

  The communication interface 41 receives intensity information indicating the intensity of illumination transmitted from the image processing device 20 via the communication interface 24. The communication interface 41, like the communication interface 24, can be a physical connector, a wireless communication device, or the like.

  The irradiation control unit 42 controls the intensity of irradiation by the light emitting unit 43 based on the intensity information received by the communication interface 41.

  The light emitting unit 43 irradiates the area captured by the camera 10 with the light emission intensity controlled by the irradiation control unit 42. The light emitting unit 43 includes a light source of near-infrared light. The light source of the near-infrared light can be, for example, a near-infrared LED (light emitting diode).

  In the second embodiment, the processor 22 of the image processing device 20 captures images using the plurality of cameras 10a, 10b, 10c, and 10d, respectively, based on the difference in brightness information of the plurality of images acquired by the acquisition unit 21. And the intensity of the lighting devices 40a, 40b, 40c, 40d.

  Specifically, the processor 22 determines the intensities of the lighting devices 40a, 40b, 40c, and 40d such that the difference between the brightness information of the plurality of images is equal to or less than a third threshold. Then, the processor 22 transmits intensity information representing the intensity determined for each of the lighting devices 40a, 40b, 40c, and 40d to each of the lighting devices 40a, 40b, 40c, and 40d via the communication interface 24.

  Other configurations and operations in the second embodiment are the same as those in the first embodiment. Therefore, the same or corresponding components are denoted by the same reference characters and description thereof is omitted.

  Next, an image processing method of the image processing system 2 according to the second embodiment will be described with reference to FIG.

  First, the cameras 10a, 10b, 10c, and 10d each capture an image of a subject around the moving body 4 to generate images Ia1, Ib1, Ic1, and Id1 (step S31).

  When the images Ia1, Ib1, Ic1, and Id1 are generated in step S11, the acquisition unit 21 of the image processing device 20 outputs image information representing the images Ia1, Ib1, Ic1, and Id1 to the cameras 10a, 10b, and 10c, respectively. 10d (step S32).

  After acquiring the image information in step S12, the processor 22 calculates the brightness information of each image based on the brightness of the pixels constituting the images Ia1, Ib1, Ic1, and Id1 (step S33).

  When the brightness information is calculated in step S33, the processor 22 determines the intensity of each of the lighting devices 40a, 40b, 40c, and 40d such that the difference between the brightness information of each image is equal to or less than a third threshold. (Step S34).

  When the intensity is determined in step S34, the processor 22 transmits intensity information representing the intensity to the lighting devices 40a, 40b, 40c, and 40d (step S35).

  When the intensity information is transmitted in step S35, the lighting devices 40a, 40b, 40c, and 40d respectively cause at least a part of the imaging range of each of the cameras 10a, 10b, 10c, and 10d to emit near-infrared light based on the intensity information. Irradiation is performed (step S36).

  In a state where at least a part of the imaging range of the cameras 10a, 10b, 10c, and 10d is irradiated with the near-infrared light in step S36, the cameras 10a, 10b, 10c, and 10d re-examine the objects around the moving body 4. Images are respectively captured to generate images Ia2, Ib2, Ic2, and Id2 (step S37). By the processing in steps S33 to S36 described above, the difference between the brightness information of the images Ia2, Ib2, Ic2, and Id2 generated here is equal to or smaller than the third threshold.

  When the images Ia2, Ib2, Ic2, Id2 are generated in step S37, the obtaining unit 21 obtains the generated images Ia2, Ib2, Ic2, Id2 (step S38).

  When the images Ia2, Ib2, Ic2, and Id2 are obtained, the processor 22 performs the same processing in steps S39 to S44 as in steps S15 to S20 of the first embodiment.

  As described above, in the second embodiment, one of the plurality of images captured by the plurality of cameras 10a, 10b, 10c, and 10d has a reduced saturation due to near-infrared light. Also, since the saturation of the pixels constituting at least one image is reduced based on the saturation information of each of the plurality of images, the saturation information of all the images can be made the same. Therefore, when a combined image is generated by combining these images, the same effect as in the first embodiment can be obtained in that the referrer can refer to the image without feeling a difference in coloring for each region. .

  In the second embodiment, based on brightness information of a plurality of images respectively captured by the plurality of cameras 10a, 10b, 10c, and 10d, the imaging ranges of the plurality of cameras 10a, 10b, 10c, and 10d are determined. The lighting devices 40a, 40b, 40c, and 40d are controlled so that the brightness is substantially the same. Therefore, at night or in the evening, the illuminating devices 40a, 40b, 40c, and 40d emit light when only a part of the imaging range of the plurality of cameras 10a, 10b, 10c, and 10d is illuminated by street lights or the like. , 40c, and 40d are controlled, and the plurality of images respectively captured by the plurality of cameras 10a, 10b, 10c, and 10d have substantially the same brightness information. If the saturation information is the same in such a plurality of images, if a synthesized image is generated by synthesizing these images, a person who refers to the synthesized image will not be able to determine both the brightness and the vividness. An image can be referred to without feeling a difference for each region.

  Subsequently, a third embodiment of the present invention will be described.

  An image processing system 3 according to the third embodiment is mounted on a moving body 4 similarly to the image processing system 1 according to the first embodiment, and includes imaging devices (cameras) 10a, 10b, 10c, 10d, and an image processing device. 20 and a display device 30.

  The processor 22 according to the first embodiment adjusts the saturation information of the second image different from the first image to the first image having the lowest saturation information among the images Ia, Ib, Ic, and Id. Decrease saturation. The processor 22 according to the third embodiment determines that if any one or more of the saturation information Spa, Spb, Spc, and Spd of each of a plurality of images is equal to or less than a predetermined threshold (second threshold), the image Ia , Ib, Ic, and Id become the monochrome images, respectively. In this point, the processor 22 in the third embodiment is different from the processor 22 in the first embodiment.

  Other configurations and operations in the third embodiment are the same as those in the first embodiment, and thus the same or corresponding components are denoted by the same reference characters and description thereof will be omitted.

  As described above, in the third embodiment, even if any one of the plurality of images captured by the cameras 10a, 10b, 10c, and 10d has a reduced saturation due to near-infrared light. The color saturation of pixels constituting at least one image is reduced based on the saturation information Spa, Spb, Spc, Spd of each of the plurality of images. Therefore, the saturation information of all the images can be made equal. Therefore, when a synthesized image is generated by synthesizing these images, an effect similar to that of the first embodiment is obtained in that a person who refers to the synthesized image can refer to the image with less discomfort. .

  In the third embodiment, at least one of the saturation information Spa, Spb, Spc, and Spd of each of the plurality of images Ia, Ib, Ic, and Id is set to a predetermined threshold (a second threshold). In the following cases, all of the plurality of images respectively captured by the cameras 10a, 10b, 10c, and 10d are monochrome images. Therefore, the difference between the saturation information of a plurality of images can be further reduced. Therefore, when a combined image is generated by combining these images, a person who refers to the combined image can further reduce the discomfort of the referring person due to a difference in saturation.

  Although the above embodiments have been described as representative examples, it will be apparent to those skilled in the art that many changes and substitutions can be made within the spirit and scope of the present invention. Therefore, the present invention should not be construed as being limited by the above-described embodiments and examples, and various modifications and changes can be made without departing from the scope of the claims. For example, a plurality of constituent blocks described in the embodiment and the examples can be combined into one, or one constituent block can be divided.

  In the above-described embodiment, the image processing system 1 includes the four cameras 10a, 10b, 10c, and 10d. However, the number of the cameras 10 is not limited to four, but may be two, three, or five or more. I just need. That is, the image processing system 1 can include a plurality of cameras.

1 Image processing system 4 Moving object 10, 10a, 10b, 10c, 10d Camera 11, 11a, 11b, 11c, 11d Imaging lens 12, 12a, 12b, 12c, 12d Filter 13, 13a, 13b, 13c, 13d Image sensor 14 , 14a, 14b, 14c, 14d Signal processing unit 15, 15a, 15b, 15c, 15d Communication interface 20 Image processing device 21 Acquisition unit 22 Processor 23 Memory 24 Communication interface 30 Display device 31 Communication interface 32 Display processor 33 Display panel 40 , 40a, 40b, 40c, 40d Lighting device 41, 41a, 41b, 41c, 41d Communication interface 42, 42a, 42b, 42c, 42d Irradiation control unit 43, 43a, 43b, 43c, 43d Light emitting unit

Claims (13)

An acquisition unit that transmits visible light, and acquires a plurality of images respectively captured by a plurality of cameras including a filter that selectively transmits infrared light in a predetermined wavelength range,
Calculating the saturation information of each of the plurality of images acquired by the acquisition unit; and decreasing the saturation of a pixel constituting at least one of the plurality of images based on the respective saturation information. A processor for setting a difference between the saturation information of the plurality of images to be equal to or less than a first threshold having a predetermined value ;
Equipped with a,
The processor generates the synthesized image by synthesizing the image with the reduced saturation and an image different from the at least one image of the plurality of images,
Image processing device.   The processor, based on a difference between saturation information of a first image of the plurality of images and saturation information of a second image different from the first image of the plurality of images, The image processing device according to claim 1, wherein the saturation of the pixels constituting the image having the higher saturation information of the image and the second image is reduced. The processor and saturation information of the first image, when the difference between the chroma information of the second image is greater than the first threshold value, among the first image and the second image The image processing apparatus according to claim 2, wherein the saturation of the pixel having the higher saturation information is reduced.   The processor may include, for each of the plurality of images, the image in which the saturation information of the pixel having the high saturation information is reduced in saturation and the image in which the saturation information is low among the plurality of images, respectively. The image processing apparatus according to claim 3, wherein the saturation of pixels forming an image having high saturation information is reduced so that a difference in information is equal to or less than the first threshold.   The processor, when any one or more of the saturation information of each of the plurality of images is equal to or less than a second threshold having a predetermined value, so that all of the plurality of images are monochrome images. The image processing apparatus according to claim 1, wherein the saturation of pixels constituting each of the plurality of images is reduced.   The processor adjusts a difference between brightness information of the plurality of images acquired by the acquisition unit to be equal to or less than a third threshold having a predetermined value, and adjusts a color of the plurality of images in which the brightness information is adjusted. The image processing apparatus according to claim 1, wherein degree information is calculated. A plurality of cameras including a filter that transmits visible light and selectively transmits infrared light in a predetermined wavelength range,
An acquisition unit that acquires a plurality of images respectively captured by the plurality of cameras, calculates saturation information of each of the plurality of images acquired by the acquisition unit, and, based on the respective saturation information, By reducing the saturation of pixels constituting at least one of the plurality of images, the difference between the saturation information of the plurality of images is set to be equal to or less than a first threshold having a predetermined value, and the saturation is reduced. An image processing apparatus including: a reduced image; and a processor configured to combine a different image from the at least one image of the plurality of images to generate a combined image ;
An image processing system comprising: Further comprising a plurality of lighting devices that respectively illuminate the imaging range of the plurality of cameras,
The image processing system according to claim 7, wherein the processor calculates brightness information of each of the plurality of images acquired by the acquisition unit, and controls the plurality of lighting devices based on the brightness information. Wherein the processor difference in each of the plurality of image brightness information is determined by whether more than the first threshold value, if the difference is greater than the first threshold value, the difference is less than the first threshold value At least one of the plurality of lighting devices is controlled so as to irradiate the imaging range, and when the difference is equal to or less than the first threshold, all of the plurality of lighting devices perform the imaging. The image processing system according to claim 8, wherein control is performed so as not to irradiate the range.   The image processing system according to claim 8, wherein the illumination device emits light including infrared light in a predetermined wavelength range.   When the saturation of the pixel is reduced, the processor combines the image with the reduced saturation with an image different from the at least one image of the plurality of images to generate a combined image. The image processing system according to claim 7. A plurality of cameras having a filter that transmits visible light and selectively transmits infrared light in a predetermined wavelength range,
An acquisition unit that acquires a plurality of images respectively captured by the plurality of cameras, calculates saturation information of each of the plurality of images acquired by the acquisition unit, and, based on the respective saturation information, By reducing the saturation of pixels constituting at least one of the plurality of images, the difference between the saturation information of the plurality of images is set to be equal to or less than a first threshold having a predetermined value, and the saturation is reduced. A moving object including: an image processing apparatus including: a reduced image; and a processor configured to combine an image different from the at least one image among the plurality of images to generate a combined image. . An image processing method executed by an image processing device,
The image processing apparatus transmits visible light, and acquires a plurality of images respectively captured by a plurality of cameras including a filter that selectively transmits infrared light in a predetermined wavelength range,
The image processing apparatus calculates saturation information of each of the plurality of acquired images, and calculates a saturation of a pixel included in at least one of the plurality of images based on the saturation information. By reducing the difference between the saturation information of the plurality of images to a first threshold having a predetermined value or less,
Generating a synthesized image by synthesizing the image with the reduced saturation and an image different from the at least one image of the plurality of images;
Image processing method.

Images