JP2019033470A - Image processing system, imaging apparatus, image processing apparatus, control method, and program - Google Patents

Abstract

To solve the problem in which there is a possibility that a problem may occur in performing correction processing on an image in which different input and output characteristics (gamma curves) are applied to a plurality of partial areas in the image.SOLUTION: By acquiring area information indicating at least one of a first area to which a first input and output characteristic is applied and a second area to which a second input and output characteristic is applied, the first area and the second area are distinguished from each other and displayed, and a correction intended by a user is performed to improve color reproducibility of a photographed image.SELECTED DRAWING: Figure 5

Description

  The present invention relates to a technique for applying correction processing for each region to a captured image obtained by combining a plurality of images.

  In recent years, monitoring systems using network cameras have become widespread. Since network cameras are used in a wide range of fields as surveillance cameras in large-scale public institutions and mass retailers, it is necessary to expand the dynamic range for environments with large illuminance differences, such as indoors and outdoors, or environments with different lighting. It has become.

  It is also required to display colors more accurately. For example, Patent Document 1 describes a method of recognizing a subject to classify different areas and determine imaging conditions including an exposure condition and an image processing condition for the divided areas. Patent Document 2 proposes to calculate and correct white balance for each of a plurality of regions.

Japanese Patent Laid-Open No. 2006-192606 Japanese Patent Laying-Open No. 2015-192152

  However, there is a possibility that a malfunction may occur when performing correction processing on an image in which different input / output characteristics (gamma curves) are applied to a plurality of partial areas in the image due to expansion of the dynamic range. . Specifically, when the user performs adjustment processing such as white balance for the entire image without being conscious of the input / output characteristics applied to each partial region, it is difficult to obtain preferable color reproduction.

  In order to solve the above problems, an image processing system according to the present invention is an image processing system having an imaging device and an image processing device, the imaging device including an imaging unit that acquires a captured image, and the imaging unit. Setting means for setting a first area to which the first input / output characteristic is applied and a second area to which the second input / output characteristic is applied, and the first area or the second area, Output means for outputting area information indicating at least one of the display, the image processing apparatus acquires the area information from the imaging apparatus, and displays the first area and the second area separately Control means is provided.

  According to the present invention, by making it easy for a user to identify a region to which a different gamma curve or the like is applied, correction processing such as white balance adjustment can be easily applied to each region, and preferable color reproducibility can be obtained. it can.

It is a schematic diagram which shows a system configuration. It is a block diagram which shows the structure of a camera. It is a block diagram which shows the structure of a client. It is a block diagram which shows the detail of a structure of the image process part of a camera. It is a flowchart of a gamma adjustment process. (A) It is a schematic diagram which shows the example of a scene. (B) It is a schematic diagram for demonstrating a histogram. (C) It is a schematic diagram for demonstrating a gamma curve. (D) It is a schematic diagram which shows a map. It is a schematic diagram which shows the outline of GUI. It is a flowchart which shows the outline of a process of an image process part. (A)-(d) It is a schematic diagram which shows the example of the scene from which exposure differs, respectively. (A) It is a schematic diagram for demonstrating an image composition ratio. (B) It is a schematic diagram which shows a map. (A) It is a schematic diagram which shows the outline of GUI. (B) It is a table | surface shown about the camera setting at the time of applying priority mode. It is a flowchart which shows the process for setting a white balance. It is a schematic diagram which shows the example of the several parameter each set to each structure of an image process part. It is a block diagram which shows the detail of a structure of an image process part. It is a flowchart which shows the outline of a process of an image process part. It is a schematic diagram which shows the outline of GUI.

  Embodiments of the present invention will be described below in detail with reference to the drawings.

<< Embodiment 1 >>
Hereinafter, Embodiment 1 of the present invention will be described with reference to FIGS. In the following, a network camera will be described as an embodiment of the present invention.

  FIG. 1 is a diagram illustrating an example of a schematic configuration of a network camera system as an image processing system according to the first embodiment. As shown in FIG. 1, the network camera system 100 includes a network camera 110 (hereinafter referred to as a camera) as an imaging device, a viewer client 120 (hereinafter referred to as a client), and a network 130. The camera 110 and the client 120 are communicably connected to each other via a network 130. Note that the photographing apparatus is not limited to a network camera, and other types having a photographing function such as a digital single lens reflex camera, a mirrorless single lens camera, a compact digital camera, a camcorder, a tablet terminal, a PHS, a smartphone, a feature phone, and a portable game machine. It may be replaced with a portable device.

  The camera 110 distributes image data including a captured (captured) image via the network 130. The client 120 accesses the camera 110, makes initial settings for the camera 110, sets the imaging parameters of the camera 110, sets the distribution settings, and so forth, so that desired image data can be acquired. Then, the client 120 processes the image data distributed from the camera 110, stores the distributed image data, processes the stored image data, and the like, and processes the image based on the processed image data. Is displayed.

  The network 130 connects the camera 110 and the client 120 so that they can communicate with each other. For example, the network 130 includes a plurality of routers, switches, cables, and the like that satisfy a communication standard such as Ethernet. In the present embodiment, the communication standard, scale, and configuration of the network 130 are not limited as long as communication (for image distribution and camera setting) between the camera 110 and the client 120 can be performed without any problem. Therefore, the network 130 can be applied from the Internet to a wired LAN and a wireless LAN.

  FIG. 2 is a block diagram showing a configuration of the camera 110 in the present embodiment. The imaging optical system 201 includes an objective lens, a zoom lens, a focus lens, an optical diaphragm, and the like, and collects light information of a subject on an imaging element unit 202 described later. The imaging element unit 202 is an element that converts light information collected by the imaging optical system 201 into a current value, and includes a CCD or CMOS sensor, and acquires color information by combining with a color filter or the like. Here, the image sensor unit 202 is basically an image sensor that can set an arbitrary exposure time for each pixel.

  The CPU 203 is involved in the processing of each component connected to the bus 210. For example, the CUP 203 sequentially reads and interprets instructions stored in a ROM (Read Only Memory) 204 or a RAM (Random Access Memory) 205, and executes processing according to the result. The imaging system control unit 206 controls the imaging optical system 201 as instructed if there is an instruction such as driving the focus lens to adjust the focus or adjusting the diaphragm from the CPU 203.

  More specifically, aperture drive control is based on an exposure value calculated based on an AE function specified by a shooting mode set by the user, such as a program AE (Automatic Exposure), shutter speed priority AE, or aperture priority AE. Done.

  The CPU 203 performs AF (Autofocus) control in conjunction with AE control. An active method, a phase difference detection method, a contrast detection method, or the like is applied to the AF control. In addition, since it is sufficient to use a known general technique for the configuration and control of this type of AE and AF, detailed description thereof is omitted here.

  The image signal digitized by the image sensor unit 202 is input to the image processing unit 207. The image processing unit 207 performs image processing described later to generate a luminance signal Y and color difference signals Cb / Cr.

  The encoder unit 208 converts the image data processed by the image processing unit 207 to Jpeg or H.264. H.264, H.C. An encoding process for converting to a predetermined format such as H.265 is performed.

  The communication unit 209 communicates with the client 120 according to a camera control protocol such as onvif, and distributes photographed image data to the client 120 via the network 130. The communication using the camera control protocol is to receive a camera operation command, a camera setting command, a function inquiry, or the like from the client 120 and transmit necessary data other than the response and image data.

  FIG. 3 is a block diagram showing a schematic configuration of the client 120 in the present embodiment. The CPU 301 controls the operation of the client 120 in an integrated manner. The ROM 302 is a non-volatile memory that stores a control program necessary for the CPU 301 to execute processing. A RAM 303 functions as a main memory, work area, and the like for the CPU 301. That is, the CPU 301 loads a necessary program or the like from the ROM 302 to the RAM 303 when executing the process, and implements various functional operations by executing the loaded program or the like, or performs the processes shown in FIGS. 5 and 8 to be described later. Or do it.

  The HDD 304 is a large-capacity secondary storage unit and stores, for example, various data necessary for the CPU 301 to perform processing, image data, various information, and the like. The HDD 304 stores various data, image data, various information, and the like obtained by the CPU 301 performing processing using a program or the like.

  The operation input unit 305 is an input unit including operation devices such as a power button, a keyboard, and a mouse, and functions as a reception unit that receives various settings (image processing settings and priority settings for each area, which will be described later) from the user. . The communication unit 306 performs communication processing between the client 120 and the network 130. Specifically, image data captured by the camera 110 is received via the network 130. In addition, a camera operation command is transmitted to the camera 110, and necessary data other than the response and image data are received.

  The display unit 307 includes a GUI (details will be described later) and a display for inputting various control parameters of the camera 110. The display unit 307 may be configured to perform display control so that a GUI or the like described later is displayed on an external display. Also. Some or all of the functions of each element of the client 120 can be realized by the CPU 301 executing a program. However, at least a part (GPU, DMA, etc.) of each element of the client 120 may operate separately from the CPU 301 as dedicated hardware. In this case, the dedicated hardware operates based on the control of the CPU 301.

  FIG. 4 is a block diagram showing details of the configuration of the image processing unit 207 of this embodiment. The image processing unit 207 is roughly divided into two blocks, a development processing unit 400 and a dynamic range expansion processing unit 410, and is connected to the memory 420 via the local bus 430.

  The development processing unit 400 includes an optical correction unit 401 that corrects the imaging optical system 201 such as lens aberration correction and peripheral light amount correction for the image data input from the image sensor unit 202, and correction of sensor defects and defects. And a sensor correction unit 402 that performs correction of the imaging element unit 202 such as offset adjustment, and a gain adjustment unit 403 that performs gain adjustment with a digital value together with the gain of the sensor. Further, the development processing unit 400 includes an NR processing unit 404 that performs noise reduction processing, a WB adjustment unit 405 that performs white balance adjustment, a gamma correction unit 406 that performs gamma correction, a sharpness processing unit 407 that performs sharpness processing, a contrast adjustment processing, A color processing unit 408 that performs color processing such as saturation adjustment processing and color conversion processing, etc., has a configuration for performing image content correction processing. The output of the development processing unit 400 is temporarily stored in the memory 420.

  The dynamic range expansion processing unit 410 includes a histogram analysis processing unit 411, a map creation processing unit 412, a gamma adjustment unit 413, and a WDR synthesis processing unit 414, which will be described later. The map information created in the map creation process 412 is also stored in the memory 420. A description of each functional module included in the dynamic range expansion processing unit 410 will be given later.

  The attribute generation unit 409 outputs attribute information to each component of the image processing unit 207 (development processing unit 400) in order to change the content of the image processing based on the luminance of the pixel. Each configuration is configured such that a processing parameter when processing image data can be changed with reference to attribute information output from the attribute generation unit 409.

For example, the brightness threshold value Y ^th is set in the attribute generation unit 409. Then, the attribute generation unit 409 compares the luminance value for each processing pixel with the luminance threshold value ^Yth, and adds information indicating whether the value is larger than the threshold value to the luminance information of the pixel as attribute information. For example, attribute information may be a boolean value, if the luminance value of the pixel is greater than the Y ^th "1", holds the value of smaller if "0". The optical correction unit 401 to the color processing unit 408 refer to the attribute information and set a processing parameter corresponding to the attribute information.

  Next, the operation flow of the image processing unit 207 will be described with reference to FIG. In step S <b> 501, the image processing unit 207 receives image data from the image sensor unit 202. Next, in step S502, various processes are performed on the image data received by each component of the development processing unit 400.

  In step S <b> 503, the image processing unit 207 determines whether a map creation instruction has been received from the client 120. In this embodiment, this instruction is a command sent from the client when the user starts an image adjustment operation such as white balance. For example, it is a command that means that the map information is requested from the client 120 when there is a region where the content of the development processing differs depending on the brightness. If this command has not been received, gamma adjustment processing is performed in step S507 described later, and the processing ends. On the other hand, if there is a map creation instruction, the process proceeds to step S504.

  In step S504, the histogram analysis processing unit 411 performs histogram analysis. The histogram will be described with reference to FIG. FIG. 6A shows an example of a shooting scene, and a range including the outside of the window 601 indicated by shading and the room 602 is shot. The outside of the window 601 and the room 602 are affected by different light sources, and it is assumed that the luminance difference between the image area of the window 601 and the image area of the room 602 is large. In the case of a scene having such a luminance difference, as shown in FIG. 6B, the histogram includes a histogram having a peak 702 corresponding to the room 602 and a histogram having a peak 701 corresponding to the window 601. It seems that is sandwiched.

  In step S504, the histogram analysis unit 411 generates a histogram for the luminance value for each pixel from the image data, and detects whether the generated histogram has two peaks or only one peak. In step S505, the process branches depending on the number of peaks that are the analysis results.

  When the number of detected peaks is one or less, gamma adjustment processing 507 described later is performed and the processing of the image processing unit 207 is ended. If there are two peaks, map creation processing is performed in step S506. Further, when there are two peaks, the histogram analysis unit 411 sets the luminance value of the peak valley 703 in the attribute generation unit 409. It may be configured to detect three or more peaks, and the region may be divided by an amount corresponding to the detected peak, or if the number of pixels included is small (the size of the region to which the peak belongs). It may be configured to be ignored (if it is small).

  In step S506, the map creation processing unit 412 creates a map. For an image in which two peaks are detected, the map shows on the image which region the histogram belongs to. First, the map creation processing unit 412 divides the image into a plurality of luminance areas. In this embodiment, the resolution of the processed image 604 is 1920 × 1080 and is divided into 64 × 36 blocks 603. This block is divided into a block 603 in which a pixel having a luminance value larger than the luminance value of the valley 703 of the histogram exceeds 2/3 of the number of pixels in the block, and a block 603 that does not, and is shown as a map. A map corresponding to the image shown in FIG. 6 is shown in FIG. Since the window portion is bright, it can be seen that the region is divided between the window 601 portion and the other portions. The dynamic range expansion processing unit 410 outputs the map information and the number of peaks detected in step S505 to the memory 420, and the communication unit 209 notifies the client 120.

  In step S507, the gamma adjustment unit 413 performs gamma adjustment processing. When the map creation process in step S506 is not performed, the adjustment gamma is a gamma curve (gamma curve) indicated by a dotted line 801 in FIG. Here, when the map is created, the adjustment gamma is adjusted so as to have characteristics as indicated by solid lines 802 (first input / output characteristics) and 803 (second input / output characteristics) in FIG. . As shown in FIG. 6C, the gradation characteristics are discontinuous as the gamma characteristics. The luminance area is divided using this discontinuous luminance as a boundary. That is, in FIG. 6B, the brightness is adjusted so as to lower the brightness of the area after the valley 703, and the process is to secure the entire dynamic range. The output value is once lowered at the luminance value corresponding to the valley 703. By adjusting the luminance with such a gamma curve, the dynamic range is expanded and the visibility of a bright area is improved. As described above, the processing of the image processing unit 207 is completed.

  Next, processing after the camera 110 receives a setting parameter transmitted from the client 120 described later will be described. The setting parameter received from the client 120 is reflected as an image processing parameter of each image processing module (configurations 401 to 409 in FIG. 4), and is switched according to a threshold set by the attribute generation unit 409. Specifically, a luminance threshold value corresponding to the value 703 of the valley of the histogram is set as a luminance threshold value in the attribute generation unit 409, and each image processing parameter is changed with this value as a boundary. Therefore, even when the subject moves, the same gamma correction is continuously applied to the same subject. Similarly, the white balance process for bright pixels is also maintained.

  Here, the white balance processing will be described in detail. White balance has two operation modes: automatic setting and manual setting. This operation mode is designated by the client 120, and the operation of the client will be described later.

  When the white balance automatic mode is designated, the area information indicating the white balance measurement area is also sent from the client 120 to the camera 110. In the manual mode, white balance color temperature information and light source information are received from the client 120. The manual mode and the automatic mode are set such that when two areas are detected in the histogram analysis in step S504, settings corresponding to the respective areas can be made, so that the camera 110 receives the two setting information. become.

  When the automatic mode is selected, a pixel that is considered to be white is extracted from the measurement region designated by the client 120 (for example, a pixel other than noise and having a pixel value closest to the white value is selected. Extract). Next, by calculating a color conversion coefficient that approximates the extracted white pixel value to the white value, and applying the color conversion process based on the calculated coefficient to the entire processing target pixel, Adjust the white balance. On the other hand, if the measurement area is not designated by the client 120, pixels that are considered to be white are extracted from each area divided by the map, a coefficient is calculated for each area, color conversion is performed, and white balance is obtained. Make adjustments.

  In the manual mode, the white balance is adjusted based on the color temperature and light source information specified from the client 120 for each area divided by the map. The configuration is such that the white balance is set by changing the processing parameters for the pixels that do not exceed the pixels that do not.

  Furthermore, image processing other than white balance adjustment, for example, noise reduction processing, sharpness processing, contrast adjustment, and saturation adjustment can be similarly changed according to the brightness threshold set in the attribute generation unit 409.

  FIG. 13 shows how the parameters of each component of the image processing unit 207 are switched according to the threshold value set by the attribute generation unit 409. For each configuration of each image processing unit 207, the processing content can be set by comparison with the luminance threshold value set in the attribute generation unit 409. The white balance is set to Auto in a dark area below the threshold value and set to Manual (5500 K) in a bright area above the threshold value. Similarly, noise reduction, sharpness, etc. are set as shown in FIG.

  For example, for noise reduction, a parameter is set such that noise reduction is performed weaker in a bright area above the threshold than in an area below the threshold. For sharpness, contrast adjustment, and saturation adjustment, parameters are set such that each process is executed more strongly in a bright area that is equal to or greater than the threshold value than in an area that is less than the threshold value. As described above, more preferable color reproducibility can be obtained by switching the parameter for the region synthesized as a bright region.

  Next, processing of the client 120 will be described. In the client 120, a moving image distributed from the camera 110 can be displayed, and settings related to shooting and network settings can be performed on the camera 110.

  Here, the setting of the user's white balance will be described with reference to FIG. First, in step S1201, when the client 120 transitions to the white balance setting mode, whether the number of peaks detected in step S504 in the image processing unit 207 and the map created in step S505 are updated for the camera 110. Check if. When there is an update, in step S1202 and step S1203, the communication unit 209 of the camera 110 distributes the peak number detected in step S504 in the image processing unit 207 and the map created in step S505, and the communication unit 306 of the client 120 Receive.

  Next, in step S1204, the communication unit 306 receives the distribution image of the 110 camera 110. Further, in step S1205, the display unit 307 superimposes and displays the received map on the captured image distributed from the camera 110 on a display screen such as a monitor or display used by the user. At this time, areas with different input / output characteristics on the map may be surrounded by a frame, or may be filled with different colors, or either one of them blinks or a character string corresponding to the input / output characteristics when clicking is selected. A symbol may be displayed.

  FIG. 7 shows an example of a display screen on which the user selects a white balance adjustment region and a white balance adjustment method (operation mode). This figure is a schematic diagram of an example of a room with a window, with a bright window on the left and a person walking outside. It also shows an example of indoor people and plants. The user can select a white reference extraction region for white balance adjustment for each region divided by the displayed map. In FIG. 7, as areas to which different gamma curves are applied, a bright area 1001 outside the window and a room 1002 are displayed as a map so that a dark area 1002 can be distinguished from the outside of the window. Yes. Therefore, it is possible to select a white balance adjustment method corresponding to each region.

  Here, a plurality of white balance adjustment modes can be selected for the designated area, and a mode for automatically processing (Auto) and a mode for manually selecting a color temperature such as a light source or an environment type can also be selected. . In the example illustrated in FIG. 7, the user selects one from the selection column 1006 and the selection column 1007 for each of the region 1 corresponding to the region 1001 and the region 2 corresponding to the region 1002 from the various white balance adjustment modes displayed. Can be specified one by one. The white balance adjustment mode includes “AWB (auto white balance)”, “Fluorescent 3000K, 4000K, 6500K (fluorescent lamp and color temperature Kelvin)”, “Indoor (indoor)”, and “Outdoor (outdoor)”. Of course, it may be selectable in a pull-down format. Note that each area “Auto” is set as an initial setting, and “Manual” is set by selecting the selection field 1006.

  Further, in the case of the automatic processing mode, an arbitrary position on the image (for example, the area 1003 and the area 1004 in FIG. 7) is operated by drag & drop or the like as a white balance measurement area for each area divided by the map. It can also be specified. For this reason, for example, the white balance correction accuracy is increased by designating a region (reference region) as a white reference in advance by the user. In FIG. 7, white balance correction accuracy is preferable by specifying a rectangle 1004 when a white wall is reflected.

  When the user designates the white balance measurement area, the area information indicating the measurement area is transmitted to the camera 110 as a setting value of the WB adjustment unit 405 of the image processing unit 207. The shape of the area designated by the user and transmitted to the camera 110 may be any shape other than a rectangle. In particular, when the user does not designate a white balance area, Auto may be notified to the camera 110, and the camera 110 may automatically perform white balance adjustment for each map.

  As described above, according to the present invention, by applying white balance correction for each gamma curve, it is possible to accurately reproduce colors for areas with different illuminance differences or environments.

  Furthermore, in the present embodiment, since the processing is changed not in the position area but in the brightness area where the brightness is high and low, for example, when the angle of view of the camera changes such as panning and tilting, and at night traffic monitoring Even in such a scene where a bright subject moves in the screen, colors can be accurately reproduced for areas with different illuminance differences or environments.

<< Embodiment 2 >>
Next, Embodiment 2 of the present invention will be described. In the first embodiment, a bright portion in one image is subjected to histogram analysis, an image is generated using a different gamma adjustment value (gamma curve) for each partial region divided by luminance, and white balance is adjusted between the bright portion and the dark portion. Different embodiments have been described. In the present embodiment, an embodiment in which a plurality of images with different exposures are combined to generate a combined image will be described.

  In addition, while the same code | symbol is attached | subjected to the structure and process which have the same function as Embodiment 1, the description is abbreviate | omitted about the thing which does not change structurally and functionally.

  In the present embodiment, two frames (captured images) captured under imaging conditions with different exposures are acquired, and a frame corresponding to the bright subject side is analyzed by histogram analysis, development processing, and gamma adjustment processing using the method of the first embodiment. After that, all the frames are combined and output as a combined image. Of course, the number of images to be combined with different exposures is not limited to two, and three or more captured images may be combined. In the present embodiment, an example in which the entire shooting area is combined for simplicity is described. However, the present invention is not limited to this, and at least a part of the captured image may be combined.

  First, the operation flow of the image processing unit 207 will be described with reference to FIG. Steps S501 and S502 are the same as those in the first embodiment. In this embodiment, in step S1101, the image processing unit 207 determines whether an image for a frame necessary for composition has been received and developed. In the present embodiment, two images obtained by photographing with different exposure conditions are acquired. As an exposure condition, the shutter speed may be changed, the gain of the image sensor may be changed, or both the shutter speed and the gain may be changed. If the required number of sheets has been received, the process proceeds to step S503. If the required number has not been reached, the process returns to step S501.

  Next, in step S504, it is determined whether or not there is a map creation instruction from the client 120 as in step S503 in the first embodiment. Here, map creation in the second embodiment will be described. If map creation is necessary, a histogram analysis is performed in step S504 as in the first embodiment, and the number of peaks is determined in step S505. Here, the frame for which the histogram analysis processing unit 411 performs the histogram analysis is a high EV frame (described later). Since the histogram analysis method is the same as that of the first embodiment, the description thereof is omitted. Here, the description of the high EV / low EV frame will be supplemented. As shown in FIG. 9, a frame (hereinafter referred to as a high EV frame) shot with a bright object adjusted for exposure (hereinafter, a high EV frame) is shown in FIG. 9A, and a frame shot with a dark object adjusted for exposure (hereinafter referred to as a low EV frame). As shown in FIG. In the low EV frame FIG. 9 (c), the outside of a bright window is whitened, while an exposure close to an appropriate value is obtained in a dark room. Here, EV represents an exposure value (Exposure Value).

  If the number of peaks is two in step S505, a map is created in step S506. In this embodiment, since it is assumed that two frames with different exposures are combined, the map is a map corresponding to a combination ratio of a plurality of images. Specifically, from the low luminance side, the area where the low EV frame ratio is 100%, the mixed area of the low EV frame and the high EV frame, the area where the high EV frame ratio is 100%, and the high EV frame ratio are 100%, A plurality of peaks are detected, and the map has four categories of areas having different gamma characteristics. Of course, the map creation may be performed for a high EV frame or a low EV frame. FIG. 10B shows an example of a map created in the present embodiment in the shooting scene of FIG.

  A shaded dark area 1401 is an area where the low EV frame ratio is 100%. The widest area 1402 is a mixed area of a low EV frame and a high EV frame. Since an indoor person is reflected brightly, the area 1403 is an area having a high EV frame ratio of 100%. An area 1404 in which the outside of the window is reflected is the brightest and has a high EV frame ratio of 100%, and a plurality of peaks are detected and the areas have different gamma characteristics. The map created here is transmitted to the client 120.

  Thereafter, the processing in step S507 is the same as that in the first embodiment. FIG. 9B shows an image obtained by performing histogram analysis, developing, and gamma adjustment processing on the image taken with exposure adjusted to the bright object shown in FIG. By adjusting the gamma, the brightness of the bright area outside the window is lowered to an appropriate brightness.

  Next, in step S1103, the WDR composition processing unit 414 performs composition processing of the image after the gamma adjustment of the high EV frame and the low EV frame. A synthesis process performed by the WDR synthesis processing unit 414 will be described with reference to FIG. In FIG. 10A, the horizontal axis represents the reference luminance, and the vertical axis represents the combination ratio when the images are added and combined. A combination ratio indicated by a solid line 1301 represents a combination ratio of a low EV frame with respect to the reference luminance, and a combination ratio indicated by a one-dot chain line 1302 represents a combination ratio of a high EV frame with respect to the reference luminance. Only a low EV frame is used for an area darker than the reference luminance threshold Y1 when combining, and only a high EV frame is used for an area brighter than the reference luminance threshold Y2. Also, in the intermediate region between the boundaries Y1 and Y2 near the reference luminance threshold, the image switching can be made smooth by gradually changing the composition ratio. In this embodiment, a high EV frame is used as the reference luminance.

  FIG. 9D shows an image after the synthesis process. By combining FIG. 9A and FIG. 9C, an image having a wide dynamic range that can be expressed from a bright area to a dark area is obtained. The processing in the client 120 is the same as that in the first embodiment except that the number of map information areas is four.

  As described above, even in WDR shooting, by applying white balance correction for each gamma curve, it is possible to accurately reproduce colors for areas with different illuminance differences or environments.

  Furthermore, in the present embodiment, since the processing is changed not in the position area but in the brightness area where the brightness is high and low, for example, a scene where the angle of view of the camera changes such as pan or tilt, or night traffic monitoring Even in such a scene where a bright subject moves in the screen as described above, it is possible to more accurately reproduce the color for areas with different illuminance differences or environments.

<< Embodiment 3 >>
Next, a third embodiment of the present invention will be described. In the first embodiment and the second embodiment, a different gamma adjustment value is set for each of the ranges divided based on the luminance to perform white balance adjustment, whereas in the present embodiment, each of the ranges divided based on the luminance is set. Different gamma adjustment values and priority modes are applied. In addition, while the same code | symbol is attached | subjected to the structure and process which have the same function as Embodiment 1, 2, the description is abbreviate | omitted about what does not change structurally and functionally.

  The operation of the client will be described. FIG. 11A shows an example of a display screen (GUI) for setting by the user. The user can select a mode that the client 120 wants to prioritize based on the map information superimposed on the display screen, such as “motion priority”, “noise priority”, or “resolution priority”. In this embodiment, the map has a maximum of four areas. For example, in the case of changing the shutter speed, in this embodiment, since two images are combined, areas having different gamma characteristics between the high EV frame and the high EV frame. Since it does not make sense to divide, the area displayed to the user may be reduced depending on the setting item.

  An example of the camera setting change when the priority mode selected by the user is applied is shown in FIG. As an example, a case where “motion priority” is selected in the area 601 as shown in FIG.

  As shown in FIG. 11B, the shutter speed of the frame corresponding to the region 601 is shorter than that when the other is selected. Thereby, an image with little motion blur can be acquired. In the first embodiment, it is also possible to use this method because motion blur can be similarly reduced by reducing the noise reduction intensity.

  Further, when “resolution” priority is set for the region 602, the sharpness adjustment such as edge enhancement or the gamma curve adjustment as shown in FIG. A high image can be obtained. In addition, you may correct | amend etc. which strengthen contrast by weakening noise reduction.

  According to the present invention, by applying the priority mode for each different gamma curve, by setting in detail an image with low motion blur, low noise, high sharpness, etc. for each region with different illuminance differences or environments, A user's desired image can be obtained.

  Further, in the above-described embodiment, the description is mainly made on the areas to which different gamma curves are applied (first area and second area), but the present invention is also applied to areas to which different exposure settings are applied. It can also be applied. Specifically, when different exposure correction is applied for each region of the composite image, the correction processing shown in the above-described embodiment may be set for each region and applied.

<< Embodiment 4 >>
Next, a fourth embodiment of the present invention will be described. In the first and second embodiments, a different gamma adjustment value is set for each of the ranges divided based on the luminance to perform white balance adjustment, whereas in this embodiment, focus adjustment is performed. In addition, while the same code | symbol is attached | subjected to the structure and process which have the same function as Embodiment 1, 2, the description is abbreviate | omitted about what does not change structurally and functionally.

  In the present embodiment, a description will be given based on an embodiment in which a plurality of images with different exposures in the second embodiment are combined to generate a combined image, but the present invention may be applied to one image based on the first embodiment.

  First, FIG. 14 shows details of the configuration of the image processing unit 207 of the present embodiment. In the present embodiment, the development processing unit 400 of the image processing unit 207 has a focus evaluation value acquisition unit 1701. A focus evaluation value acquisition unit 1701 is an acquisition unit that acquires a focus evaluation value of a designated area, and acquires a focus evaluation value for each of a plurality of images with different exposures.

  Next, the operation flow of the imaging system control unit 206 and the image processing unit 207 will be described with reference to FIG. In this embodiment, after the image reception in S501, the focus evaluation value acquisition unit 1701 acquires the focus evaluation value (step S1501). The next steps S502 and S1101 are the same as those in the second embodiment, and a description thereof will be omitted.

  Next, in S1502, focus processing is performed using the focus evaluation value acquired in S1501. As the focus evaluation value, the evaluation value of the area designated by the user is held for each image with different exposure. In this step, focus is driven based on the focus evaluation value of the exposure image specified by the user.

  Here, the user designated area and the focus evaluation value will be described in detail. The user selects an area used for focus determination from the area areas 1401 to 1404 of the map shown in FIG.

  Since the area 1401 is an area having a low EV frame ratio of 100%, when the user selects this area, the focus evaluation value is calculated based on the low EV frame. Since the area 1403 and the area 1404 are areas having a high EV frame ratio of 100%, when the user selects this area, the focus evaluation value is calculated based on the high EV frame.

  An area 1402 is a mixed area of a low EV frame and a high EV frame. When this area is designated, either a high EV frame or a low EV frame may be used. In this embodiment, the focus evaluation value of a frame having a high focus evaluation value is used.

  Subsequent steps S503 to S1103 are the same as those in the second embodiment, and a description thereof will be omitted.

  With the configuration described above, the user can specify an area to be focused, so that it is possible to select a region to be focused and exposure, and to improve the visibility with respect to a desired area. Furthermore, even when the subject is moving, since the focus evaluation value is acquired by specifying the exposure, a stable value can be acquired without changing the evaluation value due to the movement of the subject.

  The S507 gamma adjustment processing implemented in the first and second embodiments can be set to be executable without depending on the present embodiment.

<< Other Embodiments >>
As other embodiments, the white balance adjustment is performed in the first and second embodiments, the image parameter adjustment is performed in the third embodiment, and the focus adjustment is performed in the fourth embodiment. The plurality of adjustment functions described above may be selectively performed.

  FIG. 16 shows an example of a display screen (GUI) for setting by the user. The user can select a plurality of modes such as “noise priority”, “focus”, and “white balance” that the client 120 wants to adjust based on the map information superimposed on the display screen. However, the focus can be set effectively only with one map information.

  As a result, the above-described adjustment process can be selectively set collectively for the exposure area that the user is paying attention to, and a desired image can be easily obtained.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (22)

An image processing system having an imaging device and an image processing device,
The imaging device
An imaging means for acquiring a captured image;
Setting means for setting a first area to which the first input / output characteristic is applied and a second area to which the second input / output characteristic is applied to the captured image acquired by the imaging means;
Output means for outputting area information indicating at least one of the first area and the second area,
The image processing apparatus includes:
An image processing system comprising: display control means for acquiring the region information from the imaging device and displaying the first region and the second region separately.   The imaging unit acquires a plurality of captured images captured under different imaging conditions, and the setting unit sets the first region and the second region based on luminance values of the plurality of captured images. The image processing system according to claim 1.   The imaging condition is an exposure condition, and the imaging apparatus generates a composite image with an expanded dynamic range based on the brightness of the first area and the brightness of the second area set by the setting unit and the plurality of captured images. The image processing system according to claim 2, further comprising a synthesizing unit for acquiring.   The input / output characteristic is a gamma curve, and the first gamma curve corresponding to the first input / output characteristic and the second gamma curve corresponding to the second input / output characteristic are discontinuous. The image processing system according to any one of claims 1 to 3, wherein the image processing system is characterized in that:   5. The image processing apparatus further comprises accepting means for accepting application of correction processing for each of the first area and the second area displayed by the display control means. The image processing system according to any one of the above.   The image processing system according to claim 5, wherein the imaging unit further includes a development processing unit that performs the correction process received by the receiving unit.   The image processing system according to claim 5, wherein the correction process is a process of adjusting white balance.   The image processing system according to claim 5, wherein the correction processing includes at least one of noise reduction processing, sharpness processing, and color processing.   The image processing apparatus further includes a receiving unit that receives a processing instruction for calculating an evaluation value of a degree of focus for either the first region or the second region displayed on the display control unit. The image processing system according to any one of claims 1 to 4, wherein the image processing system is characterized in that: An imaging means for acquiring a captured image;
Setting means for setting a first area to which the first input / output characteristic is applied and a second area to which the second input / output characteristic is applied to the captured image acquired by the imaging means;
An imaging apparatus comprising: output means for outputting area information indicating at least one of the first area and the second area.   Correction processing means having a plurality of white balance setting values and switching the plurality of white balance setting values and adjusting the white balance based on the luminance value of the first area or the second area; The imaging apparatus according to claim 10. Acquisition means for acquiring, from the imaging device, area information indicating at least one of a first area to which the first input / output characteristic is applied and a second area to which the second input / output characteristic is applied with respect to the captured image;
An image processing apparatus comprising: display control means for distinguishing and displaying the first area and the second area.   The image processing apparatus according to claim 11, further comprising a receiving unit configured to receive a setting for adjusting a white balance for each of the first area and the second area.   The image processing apparatus according to claim 11, further comprising a receiving unit configured to receive a setting of a reference area for white balance adjustment for each of the first area and the second area.   The image processing apparatus according to claim 11, further comprising a reception unit configured to receive setting of shooting priority for each of the first area and the second area.   The image processing apparatus according to claim 11, further comprising an accepting unit that accepts setting of an image processing parameter for each of the first area and the second area.   The image processing apparatus according to claim 11, wherein the first area and the second area are areas having different exposure settings. A control method of an image processing system having an imaging device and an image processing device,
The imaging device is
An acquisition step of acquiring a captured image;
A setting step for setting a first region to which the first input / output characteristic is applied and a second region to which the second input / output characteristic is applied to the captured image acquired in the acquisition step;
An output step of outputting region information indicating at least one of the first region or the second region; and
The image processing apparatus is
An acquisition step of acquiring the region information from the imaging device;
A control method for an image processing system, comprising: performing a display control step of distinguishing and displaying the first area and the second area. A method for controlling an imaging apparatus,
An acquisition step of acquiring a captured image;
A setting step for setting a first region to which the first input / output characteristic is applied and a second region to which the second input / output characteristic is applied to the captured image acquired in the acquisition step;
And a step of outputting region information indicating at least one of the first region and the second region. A method for controlling an image processing apparatus connected to an imaging device in a communicable manner,
An acquisition step of acquiring region information from the imaging device;
A control method for an image processing apparatus, comprising: a display control step of displaying the first area and the second area separately. Computer
An imaging means for acquiring a captured image;
Setting means for setting a first area to which the first input / output characteristic is applied and a second area to which the second input / output characteristic is applied to the captured image acquired by the imaging means;
A program that functions as output means for outputting area information indicating at least one of the first area and the second area. Computer
Acquisition means for acquiring, from the imaging device, area information indicating at least one of a first area to which the first input / output characteristic is applied and a second area to which the second input / output characteristic is applied with respect to the captured image;
A program comprising: functioning as display control means for distinguishing and displaying the first area and the second area.

Images