JP6341396B1 - Motion monitoring device, server device, and motion monitoring system - Google Patents

Abstract

The present invention enables a color image of a moving object to be faithfully reproduced and to output a color image with excellent visibility of the moving object even when shooting in a state where there is not enough ambient light such as at night. A color camera that captures with ambient light, a monochrome camera that captures with infrared light, a color image signal output from the color camera, and a monochrome image signal output from the monochrome camera are processed. A signal processing unit 21, a storage unit 16 for storing alignment information generated from a color image and a monochrome image captured in a state where there is sufficient ambient light, and a color image captured in a state where there is not enough ambient light In addition, the alignment unit 24 performs alignment for matching the position of the subject in the monochrome image based on the alignment information, and acquires color information from the aligned color image, and uses the color information to obtain the monochrome image. And an image composition unit 25 that performs color conversion for coloring. [Selection] Figure 3

Description

  The present invention receives a moving object monitoring device that outputs an image of a monitoring area in which a moving object to be monitored appears, and a color image and a monochrome image obtained by capturing the monitoring area in the moving object monitoring device from the moving object monitoring device via a network. The present invention relates to a server device that performs monitoring, and a moving body monitoring system that transmits an image of a monitoring area taken from the moving body monitoring device to the server device via a network.

  Surveillance systems that monitor a situation of a moving object such as a person to be monitored by installing a camera for photographing a monitoring area are widely used. In such a monitoring system, a camera that irradiates the subject with infrared light and photographs the subject may be used so that the monitoring can be continued even at night.

  While shooting with such infrared light can obtain a clear image, it is a monochrome image, and thus has a problem that it is difficult to identify the subject. For this reason, there is a demand for a technique that can improve subject identification even in images taken at night.

  In response to such a demand, conventionally, a subject is photographed in color by visible light, and a subject is photographed in monochrome by infrared light. A photographed image by visible light is used as a background image, and a red image is displayed on the background image. A technique for superimposing a person image cut out from a photographed image by external light is known (see Patent Document 1).

JP 2009-010675 A

  However, this conventional technique can improve the overall visibility of images taken at night, but does not take into account the important color reproduction of moving objects. In other words, since the moving body image is cut out from the captured image of infrared light, the color of the moving body cannot be determined. In particular, a photographed image using infrared light is captured in a state in which the luminance is inverted. For example, a person's blue clothes appear in white, so that a moving object to be monitored is misidentified. For this reason, it cannot be used at all for the purpose of monitoring a moving object.

  Therefore, the present invention provides a moving object monitoring apparatus capable of outputting a color image with excellent visibility of a moving object by faithfully reproducing the color of the moving object even when shooting in a state where there is not enough ambient light such as at night. It is a main object to provide a server device and a moving object monitoring system.

  The moving object monitoring apparatus of the present invention is a moving object monitoring apparatus that outputs a composite image by synthesizing a color image and a monochrome image obtained by capturing a monitoring area where a moving object to be monitored appears, and captures the monitoring area with ambient light. A color camera, a monochrome camera that captures the monitoring area with infrared light, a color image signal output from the color camera, and a signal processing unit that processes a monochrome image signal output from the monochrome camera; A storage unit that stores registration information generated based on the color image and the monochrome image captured in a state where there is sufficient ambient light, the color image captured in a state where there is not enough ambient light, and Alignment is performed based on the alignment information so that the positions of the subject images in the monochrome images coincide with each other. A location registration unit, and acquires the color information from the alignment already said color image, an image combining unit that performs color conversion to color the monochrome image using the color information, and configurations with.

  The server device of the present invention is a server device that receives a color image and a monochrome image obtained by capturing a monitoring area where a moving object to be monitored appears in the moving object monitoring device from the moving object monitoring device via a network. In the moving body monitoring device, a communication unit that receives a color image captured by a color camera that captures the monitoring area with ambient light, and a monochrome image captured by a monochrome camera that captures the monitoring area with infrared light; A storage unit that stores registration information generated based on the color image and the monochrome image captured in a state where there is sufficient ambient light, the color image captured in a state where there is not enough ambient light, and the Alignment that matches the position of the subject image in each monochrome image A registration unit that performs color conversion from the color image that is obtained from the registered color image and that uses the color information to color the monochrome image. And

  The moving body monitoring system according to the present invention is a moving body monitoring system that transmits a color image and a monochrome image obtained by capturing a monitoring area in which a moving body to be monitored appears in the moving body monitoring device to the server device via the network. The moving object monitoring apparatus is a color camera that captures the monitoring area with ambient light, a monochrome camera that captures the monitoring area with infrared light, and a color image signal output from the color camera, And a signal processing unit for processing a monochrome image signal output from the monochrome camera, and the color image photographed with sufficient ambient light and registration information generated based on the monochrome image are stored. A storage unit; and the color image and the monochrome image captured in a state where there is not enough ambient light And aligning the position of the image of the subject imaged on the basis of the alignment information, obtaining color information from the aligned color image, and using the color information to obtain the color information An image composition unit that performs color conversion for coloring a monochrome image and a communication unit that transmits the composite image acquired by the image composition unit to the server device are provided.

  According to the present invention, on the basis of registration information generated from a color image and a monochrome image captured with sufficient ambient light as in the daytime, the image is captured without sufficient ambient light as at night. Since the color image and the monochrome image thus aligned are aligned, the alignment can be performed with high accuracy. Color information is obtained from a color image that has been accurately aligned, and the color information is used to color a monochrome image, so that the color of the moving object is faithfully reproduced and the color of the moving object is excellent in visibility. Can be output.

Overall configuration diagram of a moving object monitoring system according to the present embodiment Explanatory drawing which shows the imaging | photography condition by the camera apparatus 1 The block diagram which shows schematic structure of the camera apparatus 1 Functional block diagram showing a schematic configuration of the signal processing unit 21 Explanatory drawing which shows the point of the resolution conversion performed in the resolution conversion part 54. Explanatory drawing showing a histogram before and after resolution conversion Explanatory drawing which shows the processing mode set by the signal processing control part 53 Flow chart showing a procedure of processing performed in the signal processing control unit 53 Flow chart showing the procedure of white balance correction performed by the gradation color tone correction unit 55 Explanatory drawing which shows the condition of the gradation correction performed by the gradation color correction unit 55 The block diagram of the principal part which concerns on the process performed in the alignment information production | generation part 23, the alignment part 24, and the image synthetic | combination part 25 Explanatory drawing which shows the outline | summary of the process performed in the alignment information generation part 23, the alignment part 24, and the image synthetic | combination part 25. Flow chart showing calibration procedure Explanatory drawing showing the situation of image matching Flow chart showing the processing procedure during operation Explanatory drawing showing the shooting situation of moving objects with different shapes Flow chart showing processing procedure during operation in the second embodiment Explanatory drawing which shows schematic structure of the camera apparatus 1 and server apparatus 2 which concern on 3rd Embodiment. Explanatory drawing which shows schematic structure of the camera apparatus 1 and server apparatus 2 which concern on 4th Embodiment.

  A first invention made to solve the above-described problem is a moving object monitoring apparatus that outputs a composite image by combining a color image and a monochrome image obtained by photographing a monitoring area where a moving object to be monitored appears. A color camera that captures the monitoring area with light, a monochrome camera that captures the monitoring area with infrared light, a color image signal output from the color camera, and a monochrome image signal output from the monochrome camera A signal processing unit for processing, a storage unit for storing alignment information generated based on the color image and the monochrome image captured in a state where there is sufficient ambient light, and a state where there is not enough ambient light Alignment for matching the positions of the images of the subject in the captured color image and the monochrome image, respectively. A registration unit that performs color conversion from the color information that is obtained from the registered color image and uses the color information to color the monochrome image. The configuration.

  According to this, based on registration information generated from a color image and a monochrome image captured with sufficient ambient light as in the daytime, the image was captured without sufficient ambient light as at night. Since color images and monochrome images are aligned, alignment can be performed with high accuracy. Color information is obtained from a color image that has been accurately aligned, and the color information is used to color a monochrome image, so that the color of the moving object is faithfully reproduced and the color of the moving object is excellent in visibility. Can be output.

  In the second aspect of the invention, the storage unit stores a plurality of pieces of alignment information for each type of moving object having a different shape, and the alignment unit detects a moving object from the monochrome image, and the type of the moving object. And the alignment is performed based on the alignment information corresponding to the type of the moving object.

  According to this, it is possible to accurately align the color image and the monochrome image according to the shape of the moving object.

  In addition, the third invention further includes an alignment information generation unit that generates the alignment information, and the storage unit stores the alignment information generated by the alignment information generation unit in the signal processing unit. The information is stored as calibration information together with information on processing conditions.

  According to this, it is possible to appropriately perform alignment of a color image and a monochrome image by performing, as calibration, processing for generating alignment information at an appropriate timing when the apparatus is installed or during operation.

  According to a fourth aspect of the present invention, the signal processing unit adds a signal value for each of a plurality of adjacent pixels in the color image to reduce the number of pixels of the color image, and the monitoring area. And a signal processing control unit that controls the operation of the resolution conversion unit based on a shooting environment.

  According to this, in a situation where there is even a small amount of ambient light, the actual color information of the subject is included in the signal value of each pixel. Therefore, the actual color of the subject can be clearly obtained by adding the signal values of a plurality of pixels. The appearing color image can be output. In addition, since the operation of the resolution conversion unit is controlled based on the shooting environment in the monitoring area, an appropriate color image can be output regardless of the shooting environment. Further, the color conversion in which the actual color of the subject appears clearly is obtained by the resolution conversion, and the color information of the moving object that appeared at night can be obtained from the color image with high accuracy.

  The fifth invention is a server device that receives a color image and a monochrome image obtained by capturing a monitoring area in which a moving object to be monitored appears in the moving object monitoring device from the moving object monitoring device via a network. In the moving body monitoring apparatus, a communication unit that receives a color image captured by a color camera that captures the monitoring area with ambient light, and a monochrome image captured by a monochrome camera that captures the surveillance area with infrared light, and an environment A storage unit that stores registration information generated based on the color image and the monochrome image captured with sufficient light, and the color image and monochrome image captured with sufficient ambient light. Alignment for matching the positions of the subject images in the images is based on the alignment information. A positioning unit that performs you are, acquires color information from the alignment already said color image, an image combining unit that performs color conversion to color the monochrome image using the color information, and configurations with.

  According to this, as in the first aspect of the invention, even when shooting in a state where there is not enough ambient light, such as at night, the color of the moving object is faithfully reproduced and a color image with excellent visibility of the moving object is output. be able to.

  The sixth invention is a moving object monitoring system that transmits a color image and a monochrome image obtained by capturing a monitoring area where a moving object to be monitored appears in the moving object monitoring device to the server device via the network. The moving body monitoring device includes a color camera that captures the monitoring area with ambient light, a monochrome camera that captures the monitoring area with infrared light, a color image signal output from the color camera, and the A signal processing unit that processes a monochrome image signal output from the monochrome camera, and a storage unit that stores the color image captured in a state where there is sufficient ambient light and registration information generated based on the monochrome image To the color image and the monochrome image taken in the absence of ambient light. Alignment for matching the position of the subject image based on the alignment information and color information from the color image that has been aligned are acquired, and the monochrome information is obtained using the color information. An image composition unit that performs color conversion to be colored and a communication unit that transmits the composite image acquired by the image composition unit to the server device are provided.

  According to this, as in the first aspect of the invention, even when shooting in a state where there is not enough ambient light, such as at night, the color of the moving object is faithfully reproduced and a color image with excellent visibility of the moving object is output. be able to.

  In addition, according to a seventh aspect of the invention, the server device includes an alignment information generation unit that generates the alignment information, and a communication unit that transmits the alignment information to the moving object monitoring device. Is configured to update the storage unit in accordance with the alignment information received from the server device.

  According to this, the user can update the alignment information used in the moving object monitoring apparatus by remote operation.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is an overall configuration diagram of the moving object monitoring system according to the first embodiment.

  The moving body monitoring system includes a camera device 1 (moving body monitoring device), a server device 2 (image storage device), and a browsing device 3. The camera device 1, the server device 2, and the browsing device 3 are connected via a network.

  The camera device 1 captures a monitoring area set in a facility, a road, or the like, and outputs a captured image in which a moving object such as a person existing in the monitoring area is captured. The server device 2 stores the captured images acquired from the camera device 1. The browsing device 3 is a PC, a tablet terminal, a smartphone, or the like, and by accessing the server device 2, the user can browse the captured images stored in the server device 2.

  Next, the camera device 1 will be described. FIG. 2 is an explanatory diagram showing a shooting situation by the camera device 1.

  The camera device 1 includes a color camera 11 and a monochrome camera 12. The color camera 11 and the monochrome camera 12 photograph a subject existing in the monitoring area, that is, a moving object such as a person, a building, a site of a facility, a road, and the like.

  The color camera 11 includes an infrared light cut filter, captures a subject in color with visible light, and outputs a color image. The monochrome camera 12 includes a visible light cut filter, captures a subject in monochrome with infrared light, and outputs a monochrome image. Note that when photographing with the monochrome camera 12, the infrared light projector 13 (see FIG. 3) irradiates the subject with near infrared light.

  Here, when shooting is performed with the color camera 11 at night, in the time of sunset or sunrise, and when there is not enough ambient light, the moving object or background that is the subject appears dark in the captured color image. There is a problem that it is difficult to distinguish the color of a moving object, for example, the color of a person's clothes or the color of a vehicle body. In addition, there is a problem that a monochrome image photographed with near-infrared light by the monochrome camera 12 is captured with the luminance reversed, for example, a person's blue clothes appear white. For this reason, misidentification of a moving body occurs.

  Therefore, in the present embodiment, by performing signal processing on the color image signal output from the color camera 11, the actual color of the subject can be clearly seen even when the image is captured in a state where there is not enough ambient light. A high-quality color image that appears can be generated.

  Furthermore, in the present embodiment, image synthesis is performed to synthesize a color image and a monochrome image that are shot in a state where there is not enough ambient light. In this image composition, color information is acquired from a color image, and a process for coloring a monochrome image using the color information is performed. In the present embodiment, a high-definition monochrome image is obtained by photographing with near-infrared light. By coloring the monochrome image using color information acquired from the color image, the color of the moving object is high-definition and faithful. It is possible to generate a color image that is reproduced at night.

  At this time, the color camera 11 and the monochrome camera 12 have parallax, and due to the influence of this parallax, the position of the subject appearing in the color image and the monochrome image is shifted. Therefore, when the color image and the monochrome image are synthesized as they are, the background color is changed. Problems such as appearing in the moving object region or the moving object color appearing in the background region occur.

  Therefore, in the present embodiment, alignment processing for matching the position of the subject between the color image and the monochrome image is performed.

  Although FIG. 2 shows an example in which the monitoring area is outdoors, indoors may be the monitoring area. In this case, the brightness of the ambient light in the monitoring area is changed by turning on / off the lighting fixtures in addition to the sunshine.

  Next, a schematic configuration of the camera device 1 will be described. FIG. 3 is a block diagram illustrating a schematic configuration of the camera device 1.

  In addition to the color camera 11 and the monochrome camera 12, the camera device 1 includes an infrared light projector 13, a communication unit 14, a control unit 15, and a storage unit 16.

  The infrared light projector 13 projects near-infrared light onto the subject when the monochrome camera 12 captures the subject.

  The communication unit 14 communicates with the server device 2 via a network. In the present embodiment, the composite image generated by the control unit 15 is transmitted to the server device 2. The processed color image and monochrome image generated by the control unit 15 may be transmitted to the server device 2.

  At this time, shooting information relating to the installation location, camera attributes, shooting time, shooting conditions, and the like is added to the image as attribute information and transmitted. The camera attributes relate to color and monochrome, identification information (MAC address, etc.) of the camera device 1 and the like. The shooting conditions relate to exposure time, gain, and the like.

  In addition, character recognition processing may be performed on a monochrome image to acquire character information in the monochrome image, and the character information may be added to the image and transmitted.

  The storage unit 16 stores a color image, a monochrome image, a composite image, and the like generated by the control unit 15. The storage unit 16 stores a program executed by the control unit 15. Further, the storage unit 16 functions as a calibration memory, and performs calibration related to imaging conditions of the color camera 11 and the monochrome camera 12 and signal processing conditions for color image and monochrome image signals performed by the control unit 15. Information (setting values of various parameters, etc.) is stored.

  The control unit 15 includes a signal processing unit 21, an LED control unit 22, an alignment information generation unit 23, an alignment unit 24, and an image composition unit 25. The control unit 15 includes a processor, and each unit of the control unit 15 is realized by executing a program stored in the storage unit 16.

  The signal processing unit 21 processes image signals output from the color camera 11 and the monochrome camera 12, respectively.

  The LED control unit 22 controls the LED that is the light source of the infrared light projector 13.

  Next, the signal processing unit 21 will be described. FIG. 4 is a functional block diagram illustrating a schematic configuration of the signal processing unit 21.

  The signal processing unit 21 includes a synchronization signal generation unit 31, a monochrome signal processing unit 32, and a color signal processing unit 33.

  The synchronization signal generation unit 31 generates a synchronization signal for synchronizing the color camera 11 and the monochrome camera 12. With this synchronization signal, the color camera 11 and the monochrome camera 12 can photograph the subject at the same timing.

  The monochrome signal processing unit 32 includes a camera interface 41, a gradation correction unit 42, and a gamma correction 43.

  In the camera interface 41, an image signal of a monochrome image output from the monochrome camera 12 is input.

  The gradation correction unit 42 performs gradation correction on the image signal of the monochrome image input to the camera interface 41.

  The gamma correction unit 56 performs gamma correction on the image signal output from the gradation correction unit 42 to correct the gradation of the image to the optimum characteristic according to the characteristics of the display device.

  The color signal processing unit 33 includes a camera interface 51, a signal level detection unit 52, a signal processing control unit 53, a resolution conversion unit 54, a tone tone correction unit 55, a gamma correction unit 56, and a Y component generation. A unit 57, a UV component generation unit 58, and an averaging reduction unit 59.

  In the camera interface 51, an image signal of a color image output from the color camera 11 is input.

  The signal level detection unit 52 detects the signal level based on the image signal of the color image input to the camera interface 51. This signal level represents the brightness of the entire image that is the imaging environment of the monitoring area, that is, the brightness of the ambient light in the monitoring area, and is detected based on the maximum value of luminance and the distribution status (histogram). .

  The signal processing control unit 53 refers to the signal level acquired by the signal level detection unit 52 and sets the degree of resolution conversion (reduction rate) performed by the resolution conversion unit 54. Further, the signal processing control unit 53 sets the degree of reduction (reduction ratio) performed by the averaging reduction part 59 according to the degree of resolution conversion. Note that the degree of resolution conversion includes the case where the resolution conversion unit 54 is stopped and resolution conversion is not performed, and the degree of averaging reduction is the case where the operation of the averaging reduction unit 59 is paused. The case where averaging reduction is not performed is also included. Here, the degree of resolution conversion (reduction ratio) performed by the resolution converter 54 is set based on the signal level acquired by the signal level detector 52 as the imaging environment of the monitoring area. The unit 52 may be omitted, and a control table for setting the degree of resolution conversion for each night time zone (1 or more) may be held according to daytime and nighttime settings determined for each day. . Further, instead of the signal level detection unit 52, an illuminance sensor or the like may be provided, and the shooting environment may be determined based on the sensor output.

  The resolution conversion unit 54 performs resolution conversion for the image signal of the color image input to the camera interface 51 to integrate the signal values of a plurality of adjacent pixels to reduce the number of pixels.

  The gradation color tone correction unit 55 performs gradation correction and color tone correction on the image signal of the color image output from the resolution conversion unit 54. As the gradation correction, for example, gain adjustment for brightening the image is performed. As the color tone correction, for example, white balance correction that suppresses the influence of the hue of the ambient light is performed.

  The gamma correction unit 56 performs gamma correction on the image signal output from the tone color tone correction unit 55 to correct the tone of the image to an optimum characteristic according to the characteristics of the display device.

  The Y component generation unit 57 generates a Y component image signal (luminance signal) from the image signal output from the gamma correction unit 56. The UV component generation unit 58 generates U component and V component image signals (color difference signals) from the image signal output from the gamma correction unit 56.

  The averaging reduction unit 59 averages the signal values of a predetermined number of pixels with respect to the image signals output from the Y component generation unit 57 and the UV component generation unit 58, respectively, thereby making the color image a predetermined size. Process to reduce.

  Next, resolution conversion performed by the resolution conversion unit 54 will be described. FIG. 5 is an explanatory diagram showing a procedure for resolution conversion. FIG. 6 is an explanatory diagram showing a histogram before and after resolution conversion.

  In the imaging element of the color camera 11, as shown in FIG. 5A, pixels of each color of R, B, and G are arranged in a Bayer pattern.

  As shown in FIG. 5B, the resolution conversion unit 54 adds the signal values of a predetermined number of adjacent pixels for pixels of the same color, and calculates the total value as shown in FIG. As shown in FIG. 4, the signal value of one pixel is used.

  In the example shown in FIG. 5, the signal values of a total of 16 pixels of 4 × 4 are added. As a result, the photographing sensitivity is increased 16 times. Further, the resolution is reduced to 1/16, and the data amount is reduced to 1/16.

  In FIG. 5, R is shown, but the same applies to B and G.

  By performing such resolution conversion, as shown in FIG. 6A, the signal value is biased to a dark range before the resolution conversion, whereas as shown in FIG. After the conversion, the signal value is spread over a wide range.

  As described above, in this embodiment, resolution conversion is performed to reduce the number of pixels of a color image by adding signal values of a plurality of pixels. As a result, in a situation where there is even a small amount of ambient light due to streetlights or building lighting, the actual color information of the subject is included in the signal value of each pixel. It is possible to output a color image in which the actual color of the image clearly appears. In particular, the moving object can be easily identified. For example, in the case of a person, the color of clothes clearly appears, and in the case of a vehicle, the color of the vehicle body appears clearly, thereby avoiding misidentification of the moving object.

  By the way, since the color image generated by the camera device 1 is transmitted to the server device 2 via the network, it is desired to reduce the data amount of the color image for the purpose of reducing the communication load.

  Here, it is conceivable to perform compression processing such as JPEG, but when compression processing is performed on a color image taken in a state where there is not enough ambient light such as at night, there is sufficient ambient light as in the daytime. The compression noise which did not occur in the color image photographed in this state appears remarkably, and the image quality is greatly lowered.

  On the other hand, in the present embodiment, by performing resolution conversion, the number of pixels of the color image is reduced, so that the data amount of the color image can be reduced. Further, compression processing may be performed on a color image that has undergone resolution conversion, and in this case, compression noise is greatly reduced compared to when compression processing is performed without performing resolution conversion. be able to.

  Next, processing performed by the signal processing control unit 53 will be described. FIG. 7 is an explanatory diagram showing processing modes set by the signal processing control unit 53. FIG. 8 is a flowchart showing a procedure of processing performed by the signal processing control unit 53.

  In the signal processing control unit 53, the signal level acquired by the signal level detection unit 52 is compared with a plurality of threshold values, and the degree of resolution conversion performed by the resolution conversion unit 54 is stepwise based on the comparison result. change.

  In the example shown in FIG. 7, three levels (minimum, intermediate, and maximum) are set as the degree of resolution conversion by dividing into three processing modes based on the signal level. Thereby, appropriate resolution conversion can be performed so that the signal value of each pixel is not saturated.

  The first processing mode is performed when the day is bright. In the first processing mode, the level of resolution conversion is minimized, and the reduction rate of resolution conversion is 1, that is, resolution conversion is not performed.

  The second processing mode is carried out when the state is dim at sunset or sunrise. In the second processing mode, the resolution conversion level is intermediate and the resolution conversion reduction rate is 1/4. That is, the resolution conversion is performed to make the resolution 1/4 by adding the signal values of a total of 4 pixels of 2 × 2.

  The third processing mode is performed when the night is dark. In the third processing mode, the resolution conversion level is maximized and the resolution conversion reduction ratio is 1/16. That is, resolution conversion is performed so that the resolution is 1/16 by adding the signal values of a total of 16 pixels of 4 × 4.

  Further, the signal processing control unit 53 averages and reduces the unit 59 according to the degree of resolution conversion so that a color image of the same size is finally obtained regardless of the degree of resolution conversion performed by the resolution conversion unit 54. Sets the degree of averaging reduction performed in.

  That is, when the resolution conversion reduction ratio is 1 in the first processing mode, the level of averaging reduction is maximized, and the reduction ratio of averaging reduction is 1/16. Also, when the resolution conversion reduction rate is 1/4 in the second processing mode, the level of average reduction is intermediate, and the reduction rate of average reduction is 1/4. When the reduction rate of resolution conversion is 1/16 in the third processing mode, the level of average reduction is minimum, and the reduction rate of average reduction is 1. That is, averaging reduction is not performed. Thereby, a color image reduced to 1/16 in the same manner in all processing modes is obtained.

  Specifically, as shown in FIG. 8, the signal level L acquired by the signal level detection unit 52 is compared with two threshold values a and b (a <b), and the signal level L is set to the threshold value a. It is determined whether or not the signal level L is less than the threshold value b (ST102). As a result, the level of resolution conversion and averaging reduction is determined for each of the three processing modes.

  That is, when the signal level L is equal to or higher than the threshold value b (No in ST102), that is, in the daytime bright state, the first processing mode is set, and the resolution conversion level is set to the minimum (ST103). ), The level of averaging reduction is set to the maximum (ST104).

  In addition, when the signal level L is equal to or higher than the threshold value a and lower than the threshold value b (Yes in ST102), that is, when the signal level L is dark in the sunset or sunrise time zone, the second processing mode is set. The resolution conversion level is set to the middle (ST105), and the averaging reduction level is set to the middle (ST106).

  On the other hand, when the signal level L is less than the threshold value a (Yes in ST101), that is, in the dark state at night, the third processing mode is set, and the resolution conversion level is set to the maximum (ST107). ), The level of averaging reduction is set to the minimum (ST108).

  In the example shown in FIG. 7 and FIG. 8, the case is divided into three cases (first to third processing modes) according to the signal level, but the case is divided into two cases or four or more cases. It may be. Further, although the resolution conversion reduction ratios are set to 1, 1/4 and 1/16, the resolution conversion is performed so that the resolution is 1/16 by adding the signal values of a total of 64 pixels of 8 × 8. For example, resolution conversion with various reduction ratios is possible.

  Next, white balance correction performed by the gradation color tone correction unit 55 will be described. FIG. 9 is a flowchart showing a procedure of white balance correction.

  The tone color tone correction unit 55 performs white balance correction on the image signal of the color image output from the resolution conversion unit 54. In the white balance correction, the color tone is corrected by regarding the brightest (high luminance) area as white. For this reason, in an image in which a light such as a streetlight or a vehicle headlight is captured at night, the light area is the brightest, and thus the color tone is corrected by regarding this area as white. At this time, if the light of the light is not white, a color cast that causes the overall color of the image to shift occurs.

  Therefore, in the present embodiment, white balance correction is performed by excluding the light region.

  Specifically, first, it is determined whether or not the signal level acquired by the signal level detection unit 52 is less than a predetermined threshold value (ST201). This threshold value distinguishes between nighttime and daytime.

  Here, when the signal level is lower than the threshold value (Yes in ST201), that is, at night, the light area in the color image is detected, and the pixels included in the light area are totaled. Exclude from the target (ST202).

  Next, the signal values of each pixel to be aggregated are summed for each RGB color to calculate the total value (RSUM, GSUM, BSUM) of each color (ST203). Then, the output value (Rout, Gout, Bout) of each color is calculated by multiplying the input value (Rin, Gin, Bin) of each color by the gain of each color based on the total value of each color (ST204). At this time, correction based on G is performed.

  On the other hand, when the signal level is equal to or higher than the threshold value (No in ST201), that is, in the daytime, the total value of each color is calculated for all the pixels in the color image (ST203). The output value of each color is calculated (ST204).

  Next, the gradation correction performed by the gradation color tone correction unit 55 will be described. FIG. 10 is an explanatory diagram showing the state of gradation correction performed by the gradation color tone correction unit 55.

  The tone color tone correction unit 55 performs tone correction (gain adjustment) to brighten the color image by adding a gain to the image signal of the color image output from the resolution conversion unit 54.

  The example shown in FIG. 10 is a nighttime color image in which streetlights are reflected. As shown in FIG. 10A, the original image is entirely dark and the subject is difficult to see.

  Here, when a large gain is uniformly applied to the image signal of the original image, as shown in FIG. 10B, the entire image becomes brighter, and the subject can be easily seen in an area away from the light, but halation is caused. Since it becomes prominent, the subject is less likely to be seen in the area near the light.

  On the other hand, when a uniform small gain is given to the image signal of the original image, as shown in FIG. 10C, the halation is reduced, but the entire image is slightly brightened and the subject is difficult to see. Not much improvement.

  Therefore, in the present embodiment, gradation correction optimized according to the region is performed. That is, a large gain is given to a dark area far from the light, and a small gain is given to a bright area near the light.

  As a result, as shown in FIG. 10D, the subject can be easily seen in the area away from the light, and the halation is reduced, so that the subject can be easily seen in the area near the light. In this way, by giving different gains depending on the region, it is possible to obtain an optimal image that is not affected by halation.

  Next, processing performed by the alignment information generation unit 23, the alignment unit 24, and the image composition unit 25 will be described. FIG. 11 is a block diagram of a main part relating to processing performed by the alignment information generation unit 23, the alignment unit 24, and the image composition unit 25. FIG. 12 is an explanatory diagram showing an overview of processing performed by the alignment information generation unit 23, the alignment unit 24, and the image synthesis unit 25.

  As shown in FIG. 11, the registration information generation unit 23 performs the registration in the registration unit 24 based on the monochrome image output from the monochrome signal processing unit 32 and the color image output from the color signal processing unit 33. A transformation matrix (positioning information) used for geometric transformation (positioning) is generated (see FIG. 12). In this embodiment, image matching is performed between a monochrome image and a color image, a plurality of feature points corresponding to the monochrome image and the color image are acquired, and a transformation matrix is estimated using the feature points.

  The processing of the alignment information generation unit 23 is performed as a calibration before operation, and a color image and a monochrome image captured simultaneously by the color camera 11 and the monochrome camera 12 in a state where there is sufficient ambient light as in the daytime. It is done using.

  Here, if the color camera 11 and the monochrome camera 12 are separate monocular cameras, the relative positional relationship between the color camera 11 and the monochrome camera 12 changes according to the installation conditions. Even if the color camera 11 and the monochrome camera 12 are accommodated in the same housing so that the positional relationship between the color camera 11 and the monochrome camera 12 does not change, the shooting conditions, For example, the shooting distance from the color camera 11 and the monochrome camera 12 to the position where the moving object appears differs. Therefore, a conversion matrix (positioning information) is acquired for each camera device 1 as calibration before operation.

  In addition, a color image taken in a state where there is not enough ambient light such as at night is dark, unclear and noisy. In such a color image, image matching with a monochrome image is appropriately performed. I can't. For this reason, a transformation matrix is acquired using a color image and a monochrome image captured in a state where there is sufficient ambient light as in the daytime.

  Note that a color image captured with visible light and a monochrome image captured with near-infrared light may have different luminance gradients, so use a technique that is robust against luminance gradients such as the phase correlation method. Perform image matching.

  The conversion matrix generated by the alignment information generation unit 23 is stored in the storage unit 16 as calibration data.

  The alignment unit 24 uses a conversion matrix (alignment information) acquired by the alignment information generation unit 23 during operation to perform geometric conversion, for example, projection, on the color image output from the color signal processing unit 33. Conversion is performed (see FIG. 12). Thereby, the position of the image of the subject appearing in the color image can be matched with the monochrome image.

  The image composition unit 25 synthesizes the aligned color image output from the alignment unit 24 and the monochrome image output from the monochrome signal processing unit 32 to generate a composite image (see FIG. 12). In this image composition, color conversion is performed in which color information is acquired from a color image that has been aligned and a monochrome image is colored using the color information. For this image composition, a known image composition technique may be used.

  Next, calibration will be described. FIG. 13 is a flowchart showing a calibration processing procedure. FIG. 14 is an explanatory diagram showing a situation of image matching.

  This calibration is performed as an initial setting when the camera apparatus 1 is installed. Further, since the relative positional relationship between the color camera 11 and the monochrome camera 12 changes due to the influence of vibration, strong wind, etc., it may be performed periodically during operation.

  As shown in FIG. 13, in calibration, the alignment information generation unit 23 first acquires a color image and a monochrome image photographed in the daytime (ST301).

  Next, image matching is performed between the color image and the monochrome image (ST302). In this image matching, a plurality of feature points corresponding to monochrome images and color images are acquired (see FIG. 14).

Next, a transformation matrix used in geometric transformation is estimated using the acquired feature points (ST303). Specifically, a transformation matrix H as shown in the following equation is obtained. Here, projective transformation is performed as geometric transformation.
S: scale (enlargement, reduction), θ: rotation, Tx, Ty: translation amount

Next, geometric conversion is performed on the color image using the acquired conversion matrix (ST304). At this time, the coordinates (x, y) of the color image are converted into coordinates (x ′, y ′) as in the following equation.
Note that this equation is also used in the geometric transformation during operation.

  Next, the color image subjected to geometric transformation and the monochrome image are compared, and it is determined whether or not the position of the subject matches (ST305). Here, when the positions of the subjects match (Yes in ST305), the acquired conversion matrix is stored in the storage unit 16. On the other hand, when the positions of the subjects do not match (No in ST305), another color image and a monochrome image are acquired, and the above process is repeated.

  Note that the geometric transformation parameters may be finely adjusted by a user operation input according to the determination result of the subject position.

  Next, processing during operation will be described. FIG. 15 is a flowchart showing a processing procedure during operation.

  At the time of operation, first, the alignment unit 24 acquires a color image and a monochrome image taken at night (ST401). Next, geometric transformation is performed on the color image using the transformation matrix obtained by calibration (ST402). Next, the image composition unit 25 synthesizes the color image and the monochrome image that have undergone the geometric transformation (ST403).

(Second Embodiment)
Next, a second embodiment will be described. Note that points not particularly mentioned here are the same as in the above embodiment. FIG. 16 is an explanatory diagram illustrating a shooting state of moving objects having different shapes.

  If the shape of the moving object is different, the distance from the color camera 11 and the monochrome camera to the surface of the moving object changes. For example, as shown in FIG. 16A, the moving body is the same between the case where the moving body is a passenger car with a bonnet and the case where the moving body is a one-box car without a bonnet as shown in FIG. The distance L from the color camera 11 and the monochrome camera 12 to the surface of the moving object is different even in the state of.

  As described above, when the distance L from the color camera 11 and the monochrome camera to the surface of the moving object changes according to the shape of the moving object, how the moving object shifts due to the parallax between the color camera 11 and the monochrome camera 12 changes. .

  Therefore, in the present embodiment, in the calibration, a plurality of transformation matrices (alignment information) are created for each type of moving object having a different shape, and the type of moving object is determined during operation to correspond to the type of moving object. A geometric transformation is performed using a transformation matrix. Here, the type of moving object may be determined using a high-definition monochrome image.

  In addition, in order to obtain a conversion matrix for each type of moving object, the moving object is actually arranged in a monitoring area or an appropriate place under the same shooting conditions as the monitoring area, and the moving object is placed in the color camera 11 and the monochrome camera. The conversion matrix may be acquired based on the obtained color image and monochrome image. Moreover, you may make it arrange | position a marker instead of a moving body.

  Next, a processing procedure during operation in the second embodiment will be described. FIG. 17 is a flowchart showing a processing procedure during operation.

  During operation, first, the alignment unit 24 acquires a color image and a monochrome image taken at night (ST501). Then, a moving object is detected from the monochrome image (ST502).

  Next, the type of the detected moving object is discriminated, and a conversion matrix corresponding to the type of moving object is selected. In the example illustrated in FIG. 17, it is sequentially determined whether or not the moving object types A to F are satisfied (ST503 to ST508), and a transformation matrix corresponding to the moving object types A to F is selected (ST509 to ST514). If none of these apply, a general-purpose transformation matrix is selected (ST515).

  Next, geometric transformation is performed on the color image using the selected transformation matrix (ST516). Next, the image synthesizing unit 25 synthesizes the color image and the monochrome image that have been subjected to the geometric transformation (ST517).

(Third embodiment)
Next, a third embodiment will be described. Note that points not particularly mentioned here are the same as in the above embodiment. FIG. 18 is an explanatory diagram showing a schematic configuration of the camera device 1 and the server device 2 according to the third embodiment.

  In the above-described embodiment, in the calibration, a process of generating alignment information (conversion matrix) used for alignment (geometric conversion) for matching the positions of the images of the subject captured in the color image and the monochrome image, In the present embodiment, the server apparatus 2 performs the processing for generating the alignment information.

  The server device 2 includes a communication unit 61, a control unit 62, and a storage unit 63.

  The communication unit 61 communicates with the camera device 1 via a network. In the present embodiment, at the time of calibration, an image transmission request relating to a color image and a monochrome image used for generating alignment information is transmitted to the camera device 1, and the color transmitted from the camera device 1 in response to the image transmission request. Receive images and monochrome images. Further, the alignment information generated by the control unit 62 is transmitted to the camera device 1. In addition, a composite image periodically transmitted from the camera device 1 is received during operation.

  The storage unit 63 stores color images and monochrome images received by the communication unit 61, registration information generated by the control unit 62, and the like. The storage unit 63 stores a program executed by the control unit 62.

  The control unit 62 includes an image transmission request unit 71 and an alignment information generation unit 72. The control unit 62 includes a processor, and each unit of the control unit 62 is realized by executing a program stored in the storage unit 63.

  The image transmission request unit 71 instructs the camera device 1 to simultaneously take a color image and a monochrome image in a state where there is sufficient ambient light as in the daytime, and to transmit the color image and the monochrome image. An image transmission request is transmitted from the communication unit 61.

  When the communication unit 61 receives the color image and the monochrome image transmitted from the camera device 1 in response to the image transmission request, the alignment information generation unit 72 performs alignment of the camera device 1 based on the color image and the monochrome image. Alignment information (transformation matrix) used for alignment (geometric transformation) performed by the unit 24 is generated. The alignment information generated by the alignment information generation unit 72 is transmitted from the communication unit 61 to the camera device 1.

  Upon receiving the image transmission request transmitted from the server device 2, the camera device 1 captures a color image and a monochrome image and transmits them to the server device 2. In addition, when the camera apparatus 1 receives the alignment information transmitted from the server apparatus 2, the camera apparatus 1 performs an update process of the storage unit 16 according to the received alignment information.

  Thereby, the user can perform initial setting and update of the alignment information used in the camera device 1 by remote operation.

  In the present embodiment, the registration information is generated by the server device 2 that stores the image transmitted from the camera device 1, but the device that stores the image and the device that generates the alignment information are: It may be provided separately.

(Fourth embodiment)
Next, a fourth embodiment will be described. Note that points not particularly mentioned here are the same as in the above embodiment. FIG. 19 is an explanatory diagram showing a schematic configuration of the camera device 1 and the server device 2 according to the fourth embodiment.

  In the above-described embodiment, the alignment between the color image and the monochrome image and the image synthesis of the color image and the monochrome image are performed by the camera device 1. However, in this embodiment, the alignment and the image are performed. The composition may be performed by the server device 2.

  That is, the control unit 62 of the server device 2 includes an alignment unit 73 and an image composition unit 74. The alignment unit 73 performs geometric conversion on the color image transmitted from the camera device 1 using the conversion matrix (alignment information) acquired by the alignment information generation unit 72. The image composition unit 74 combines the aligned color image output from the alignment unit 73 and the monochrome image transmitted from the camera device 1 to generate a composite image. The processing performed by the alignment unit 73 and the image composition unit 74 is the same as that of the alignment unit 24 and the image composition unit 25 of the camera device 1 in the above embodiment.

  As described above, the embodiments have been described as examples of the technology disclosed in the present application. However, the technology in the present disclosure is not limited to this, and can be applied to embodiments in which changes, replacements, additions, omissions, and the like have been performed. Moreover, it is also possible to combine each component demonstrated by said embodiment into a new embodiment.

  For example, in the above-described embodiment, an example in which the moving object to be monitored is mainly a person has been described, but the moving object to be monitored is not limited to a person, and may be an animal or a vehicle.

  In the above-described embodiment, the control is performed based on the signal level representing the brightness of the ambient light. However, the brightness of the ambient light changes regularly due to the change in the daylight hours according to the season and time. Therefore, it is possible to control based on time information. However, since the brightness of the ambient light changes according to the weather, the control based on the signal level can be performed with higher accuracy.

  In the above-described embodiment, various image processing (signal processing) such as resolution conversion is performed by the camera device. However, all or part of the image processing may be performed by the server device. However, since the resolution conversion and the averaging reduction processing reduce the communication load by reducing the amount of image data, it is desirable to perform the processing by the camera device.

  The moving object monitoring device, the server apparatus, and the moving object monitoring system according to the present invention are excellent in visibility of moving objects by faithfully reproducing the color of moving objects even when shooting in a state where there is not enough ambient light such as at night. A moving object monitoring device that has an effect of outputting a color image and outputs an image obtained by photographing a monitoring area where a moving object to be monitored appears, a color image and a monochrome image obtained by photographing the monitoring area in the moving object monitoring device, The present invention is useful as a server device that receives a moving object monitoring device via a network, and a moving object monitoring system that transmits an image of a monitoring area from the moving object monitoring device to the server device via the network.

1 Camera device (moving object monitoring device)
2 Server device (image storage device)
DESCRIPTION OF SYMBOLS 11 Color camera 12 Monochrome camera 14 Communication part 15 Control part 16 Storage part 21 Signal processing part 23 Registration information generation part 24 Registration part 25 Image composition part 61 Communication part 63 Storage part 72 Registration information generation part 73 Positioning part 74 Image composition unit

Claims (7)

A moving object monitoring apparatus that combines a color image and a monochrome image obtained by capturing a monitoring area where a moving object to be monitored appears and outputs a combined image,
A color camera for photographing the monitoring area with ambient light;
A monochrome camera for photographing the monitoring area with infrared light;
A signal processing unit for processing a color image signal output from the color camera and a monochrome image signal output from the monochrome camera;
A storage unit for storing alignment information generated based on the color image and the monochrome image captured in a state where there is sufficient ambient light;
An alignment unit that performs alignment based on the alignment information to align the positions of the subject images captured in the color image and the monochrome image captured in a state where there is not enough ambient light;
An image composition unit that obtains color information from the color image that has been aligned, and performs color conversion to color the monochrome image using the color information;
A moving body monitoring device comprising: The storage unit stores a plurality of the alignment information for each type of moving body having a different shape,
The alignment unit detects a moving object from the monochrome image, determines a type of the moving object, and performs alignment based on the alignment information corresponding to the type of the moving object. Item 8. The moving object monitoring apparatus according to Item 1. Furthermore, it comprises an alignment information generator for generating the alignment information,
The storage unit stores the alignment information generated by the alignment information generation unit as calibration information together with information on processing conditions of the signal processing unit. The moving body monitoring device according to claim 1. The signal processing unit
Adding a signal value for each of a plurality of adjacent pixels in the color image to reduce the number of pixels of the color image; and
A signal processing control unit that controls the operation of the resolution conversion unit based on the imaging environment of the monitoring area;
The moving body monitoring apparatus according to claim 1, wherein the moving body monitoring apparatus comprises: A server device that receives a color image and a monochrome image obtained by capturing a monitoring area where a moving object to be monitored appears in a moving object monitoring device via the network,
In the moving body monitoring device, a communication unit that receives a color image captured by a color camera that captures the monitoring area with ambient light, and a monochrome image captured by a monochrome camera that captures the monitoring area with infrared light;
A storage unit for storing alignment information generated based on the color image and the monochrome image captured in a state where there is sufficient ambient light;
An alignment unit that performs alignment based on the alignment information to align the positions of the subject images captured in the color image and the monochrome image captured in a state where there is not enough ambient light;
An image composition unit that obtains color information from the color image that has been aligned, and performs color conversion to color the monochrome image using the color information;
A server device comprising: A moving body monitoring system for transmitting a color image and a monochrome image obtained by capturing a monitoring area where a moving body to be monitored appears in a moving body monitoring device to a server device from the moving body monitoring device via a network,
The moving object monitoring device includes:
A color camera for photographing the monitoring area with ambient light;
A monochrome camera for photographing the monitoring area with infrared light;
A signal processing unit for processing a color image signal output from the color camera and a monochrome image signal output from the monochrome camera;
A storage unit for storing alignment information generated based on the color image and the monochrome image captured in a state where there is sufficient ambient light;
An alignment unit that performs alignment based on the alignment information to align the positions of the subject images captured in the color image and the monochrome image captured in a state where there is not enough ambient light;
An image composition unit that obtains color information from the color image that has been aligned, and performs color conversion to color the monochrome image using the color information;
A communication unit that transmits the composite image acquired by the image composition unit to the server device;
A moving object monitoring system comprising: The server device
An alignment information generator for generating the alignment information;
A communication unit for transmitting the alignment information to the moving object monitoring device;
With
The said moving body monitoring apparatus performs the update process of the said memory | storage part according to the said alignment information received from the said server apparatus, The moving body monitoring system of Claim 6 characterized by the above-mentioned.