JP2012244480A - All-round monitored image display processing system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To clearly display details of each frame unit of an entire 360-degree monitored video photographed by a monocular camera and to clearly display details of a clipped image clipped from the 360-degree monitored video.SOLUTION: A blocking brightness extension processing module 20 performs a blocking brightness extension process on each pixel of an image which is either a 360-degree all-round input image (Pa) of an elliptical surface stored in an image buffer memory 13 or a clipped image (object image) stored in an object image buffer memory 19 and is output for display.

Description

  The present invention relates to an omnidirectional monitoring image display processing system having a display processing function of cutting out an image of a partial area from an omnidirectional image captured by a monocular camera and displaying the image as a monitoring image.

  Various 360 ° monitoring systems have been developed and put into practical use in which a 360 ° lens is attached to a monocular camera, and an image captured by the camera is output as a monitoring image over the entire circumference of the camera. Since this type of conventional 360 ° monitoring system is an NTSC video in which the entire circumference of the camera fits in one frame, the resolution of the partial video is low, and all of the 360 ° full circumference video is visually monitored. There were limitations and detailed monitoring was difficult.

  In addition, even if the monitoring video is set to high resolution, the 360 ° all-round video often includes places with significantly different brightness, and it is not possible to visually monitor all the parts with greatly different brightness. It was difficult.

  In addition, even when an image of a partial region is cut out from a 360 ° all-round video and the cut-out image is displayed as a monitoring image, the brightness of the cut-out point is often greatly different, and a portion with a greatly different brightness is cut out. There was a problem that clear visual monitoring of images was not possible.

  Conventionally, as this type of omnidirectional monitoring camera system, there is a monitoring camera technique (Patent Document 1) that records only a specific moving object image with high image quality, or by converting wide-angle image data into an image with a normal angle of view that can be viewed. There was a surveillance camera technology (Patent Document 2) to display.

JP 2006-121320 A JP 2005-303340 A

  As described above, in a 360 ° monitoring system in which a 360 ° lens is attached to a monocular camera and an image captured by the camera is output as a monitoring image over the entire circumference of the camera, places with greatly different brightness may be included. In many cases, it has been difficult to visually monitor all of the light and dark areas that differ greatly. In addition, even when a part of an image is cut out from the 360 ° all-round video and the cut-out image is displayed as a monitoring image, the monitoring target existing in a very small area in the 360 ° all-round video is displayed. It was impossible to display clearly, and it was difficult to display details of the cut-out monitoring image.

  The present invention has been made in view of the above circumstances, and for each 360-degree monitoring video captured by a monocular camera, the entire video can be clearly displayed in detail and a part of the 360-degree monitoring video is cut out. An object of the present invention is to provide an omnidirectional monitoring image display processing system capable of clearly displaying a cut-out monitoring image in detail when the cut-out image is displayed as a monitoring image.

  The present invention is an omnidirectional monitoring image display processing system capable of displaying a monitoring view according to position designation information designated by a user with respect to a 360 ° all-round monitoring video imaged by a monocular camera. A monitoring video input unit that captures video in frame units, and a cutout process that cuts out an image of a partial area designated by the position designation information with respect to the monitoring video in frame units captured by the monitoring video input unit, Monitoring image processing means for performing correction processing for correcting distortion on the cut-out image, blocking luminance expansion processing means for performing blocking luminance expansion processing on a pixel basis on the cut-out image corrected for distortion, Display output means for displaying the clipped image subjected to the blocking luminance expansion processing as a monitoring image; It is characterized by.

  Further, the present invention is an omnidirectional monitoring image display processing system capable of displaying a monitoring view according to position designation information and view designation information designated by a user with respect to a 360 ° all-round monitoring video imaged by a monocular camera, A monitoring video input unit that captures 360 ° all-round monitoring video in frame units, an input image storage unit that stores the monitoring video in frame units captured by the monitoring video input unit as an input image, and the monitoring video input unit captures A monitoring image processing unit that performs a clipping process for cutting out an image of a partial region designated by the position designation information with respect to a monitoring image in frame units, and performs a correction process for correcting distortion for the cut image And a cut-out image storage unit that stores the cut-out image corrected for distortion as a view designation monitoring image A blocking luminance expansion processing unit that performs a block-by-pixel blocking luminance expansion process on the view designation monitoring image stored in the cut-out image storage unit, and a view designation monitoring image subjected to the luminance expansion processing in advance. View creation processing means for creating a view screen in a monitoring form according to the view format specified by the view designation information from a plurality of view formats with different prepared monitoring forms, and a view screen created by the view creation processing means Display output means for displaying.

  According to the present invention, it is possible to clearly display the entire video of the 360 ° monitoring video of each frame unit captured by the monocular camera in detail, and cut out a part of the 360 ° monitoring video and extract the cut-out image as the monitoring image. Can be displayed clearly in detail.

The block diagram which shows the structure of the omnidirectional monitoring image display processing system which concerns on embodiment of this invention. The block diagram which shows the function structure of the image process part which concerns on the said embodiment. The flowchart which shows the procedure of the process which concerns on the said embodiment. The block diagram which shows the component of the blocking luminance expansion | extension process means which concerns on the said embodiment. Operation | movement explanatory drawing for demonstrating the brightness | luminance expansion process operation | movement of the brightness | luminance expansion | extension process part which concerns on the said embodiment. Operation | movement explanatory drawing for demonstrating the blocking luminance expansion | extension process operation | movement which concerns on the said embodiment. Operation | movement explanatory drawing for demonstrating the blocking luminance expansion | extension process operation | movement which concerns on the said embodiment. Operation | movement explanatory drawing for demonstrating the convolution process operation | movement which concerns on the said embodiment. Operation | movement explanatory drawing for demonstrating the convolution process operation | movement which concerns on the said embodiment. Operation | movement explanatory drawing for demonstrating the bilinear interpolation process operation | movement which concerns on the said embodiment. The block diagram which shows the structure of the brightness expansion process part which concerns on the said embodiment. The flowchart which shows the process sequence of the brightness expansion process part which concerns on the said embodiment. The figure which shows an example of the omnidirectional (360 degrees all around) monitoring image which concerns on the said embodiment. The figure which shows the display form of the monitoring image (PTZ (Pan-Tilt-Zoom) view which designates and displays arbitrary positions) concerning the said embodiment. The figure which shows the display form of the monitoring image (preview view which switches and displays the registered position) which concerns on the said embodiment. The figure which shows the display form of the monitoring image (multi-point view which displays a several point simultaneously) which concerns on the said embodiment. The figure which shows the display form of the monitoring image (panoramic view which displays 360 degrees all the circumferences simultaneously) concerning the said embodiment.

  Embodiments of the present invention will be described below with reference to the drawings. In this embodiment, a means for realizing the display processing function will be described by taking, as an example, a cut-out image according to a monitoring view format and a cut-out position specified by the user. In this embodiment, image data for one frame (one screen) captured from a monocular camera is simply referred to as an image or a monitoring video.

  An omnidirectional monitoring image display processing system according to an embodiment of the present invention provides a 360 ° lens to a high-pixel (high-definition) monocular camera (one camera) having a number of pixels of 1.3 mega (M) pixels or more. (Fisheye lens) attached to the camera, and for various types of monitoring according to the monitoring view setting parameters specified by the user (preset information, etc.), the position information of the monitoring view specified by the user, and the monitoring view instructions. Applicable to view display. In this processing, the 360 ° all-round high-definition surveillance video (omnidirectional surveillance image) captured by the monocular camera is used as an input image, and distortion is corrected for the input image and a cut-out image cut out from the input image. After that, light / dark correction is performed by blocking luminance expansion, and display output is performed.

FIG. 1 shows the configuration of an omnidirectional monitoring image display processing system according to an embodiment of the present invention.
As shown in FIG. 1, an omnidirectional monitoring image display processing system according to an embodiment of the present invention includes a monocular camera (ip camera) 11, a capture unit 12, an image buffer memory 13, a resizing unit 14, and a monitoring unit. An input device 16 that gives a view creation instruction, a view parameter storage unit (preset information buffer memory) 17, a region-of-interest image creation processing unit 18, a cut-out image buffer memory 19, a blocking luminance expansion processing unit 20, a display view A creation processing unit 21 and a display unit (display device) 22 are provided.

  The monocular camera 11 and the capture unit 12 constitute monitoring video input means for capturing 360 ° all-round monitoring video in frame units.

  The monocular camera 11 constitutes a high-definition surveillance camera having a pixel number of 1.3 M pixels or more with a 360 ° fisheye lens 11A attached to the imaging window, and a 360 ° all-round high-definition surveillance video of an ellipsoid is displayed as a surveillance image (Pa ). Here, a 360 ° all-round high-definition monitoring video of horizontal × vertical = 1280 × 960 pixels, in which the long and short sides of the ellipsoid are matched to the width and height of one frame, is output as a monitoring image (Pa). .

  The capture unit 12 has a processing function of capturing a monitor image (Pa) in frame units (1280 × 960 pixels) captured by the camera 11 as an input image and storing it in the image buffer memory 13.

  The image buffer memory 13 constitutes an input image storage unit that stores the monitoring video (Pa) in units of frames captured by the capture unit 12 as an input image. Here, 360 ° all-round high-definition monitoring video (Pa) having horizontal × vertical = 1280 × 960 pixel configuration (Quad-VGA) is stored as an input image. Here, the latest input image (live video) captured by the monocular camera 11 is input as the input image (Pa) to be displayed, but the present invention is not limited to this. For example, as shown in FIG. A time-series recorded video stored in the storage medium 10 may be input as an input image (Pa).

  The resizing unit 14 has a 640 × 480 pixel configuration (VGA) in which the input image (Pa) of the 1280 × 960 pixel configuration (Quad-VGA) captured by the capture unit 12 is matched to the pixel configuration of the display unit (display device) 22. ) Is resized to the display input image and transferred to the blocking luminance expansion processing unit 20.

  Here, the input device 16 is used as a user operation input device for instructing creation of a monitoring view, and view designation information (Ca) for designating a view format and position designation information for a monitoring view by user operation. (Cb) is received, and the received view designation information (Ca) is sent to the attention area image creation processing unit 18, the blocking luminance expansion processing unit 20, and the view creation processing unit 21, which will be described later. The view position designation information (Cb) is stored in the view parameter storage unit 17 as preset information. The view designation information (Ca) is information for designating the display form (view format) of the monitoring view. In this embodiment, the PTZ view shown in FIG. 14, the preset view shown in FIG. 15, the multipoint view shown in FIG. Among the panoramic views shown in FIG. 17, a monitoring view in an arbitrary view format is designated as a display target (monitoring screen). The monitoring view position designation information (Cb) is preset information in various monitoring view formats for designating the rectangular position and area of the image to be cut out from the 360 ° all-round input image (Pa) for each of the monitoring views. Yes, it is used for the image cutout process of the attention area image creation processing unit 18.

  The view parameter storage unit 17 stores preset information (PTZ view position information, preset view position information, multi-point view position information) in various monitoring view formats input from the input device 16 as monitoring view parameters. To do. The view parameter storage unit 17 stores preset information as external parameters designated in advance from the input device 16, and each monitoring view (PTZ view, preset view, multipoint view, Update the preset information of (Panorama View).

  The attention area image creation processing unit 18 and the cut-out image buffer memory 19 store preset information (position information) of the designated view stored in the view parameter storage unit 17 when the monitoring view is designated by the view designation information (Ca). Is performed on the input image (Pa) stored in the image buffer memory 13 to correct distortion aberration on the cut image. The monitoring image processing means for executing the processing is configured. Here, the distortion aberration is corrected for the cut-out image (q) cut out from the input image (Pa), and 640 × 480 pixels are adapted to the display pixel configuration based on each preset information suitable for the processing of the view creation processing unit 21. The cut-out image buffer memory 19 stores the cut-out image of the configuration (VGA). A known technique is applied to the distortion correction process at this time. When the monitoring view is designated by the view designation information (Ca) in the cutout image after distortion correction stored in the cutout image buffer memory 19, the blocking luminance expansion processing unit 20 performs brightness correction by the blocking luminance expansion processing. Then, a drawing image (monitoring view) creation process designated by the user is performed in the view creation processing unit 21.

  The blocking luminance expansion processing unit 20 performs block-by-pixel blocking on an image to be displayed and output with the input image (Pa) stored in the image buffer memory 13 and the cut-out image stored in the cut-out image buffer memory 19 as processing targets. A processing unit (blocking luminance expansion processing unit) that performs luminance expansion processing is configured. In this embodiment, when a monitoring view is designated by the view designation information (Ca), pixel level contrast correction by blocking luminance expansion processing is performed on the clipped image stored in the clipped image buffer memory 19 for monitoring. When the view for use is not designated (when the view designation information (Ca) is not inputted), pixel-level brightness correction by blocking luminance expansion processing is performed on the 360 ° all-round input image (Pa). carry out. The pixel-level blocking luminance expansion processing by the blocking luminance expansion processing unit 20 corrects a greatly different portion of brightness and darkness with respect to the monitoring target image at the pixel level, and corrects the monitoring target image to an image having a dynamic range that is easy to monitor. Is done.

  The blocking luminance expansion processing unit 20 inputs an input image (here, a VGA 360-degree all-round input image resized by the resizing unit 14 or a VGA cut-out image stored in the cut-out image buffer memory 19). ), The image area (or the set attention area) is divided into a plurality of blocks, and for each of the divided blocks, a block unit based on the luminance values of a plurality of blocks obtained by convolving the luminance values of surrounding blocks. A block processing unit that calculates a luminance expansion parameter for each block and creates a block-by-block luminance conversion table based on the luminance expansion parameter, and a block unit based on the block-based luminance conversion table created by the block processing unit. For each pixel in each block, the luminance conversion test of surrounding blocks is performed for each block. Interpolation processing means for performing bilinear interpolation with reference to the image data, and image conversion means for creating a luminance expansion output image for the input image based on the pixels of each block subjected to bilinear interpolation by the interpolation processing means. It is provided, and with respect to an image having a large difference in brightness and darkness existing in a part of the entire screen, a portion having a large difference in brightness and darkness is corrected at a pixel level to correct an image easy to monitor. The configuration of the blocking luminance expansion processing unit 20 and the luminance expansion processing will be described later with reference to FIGS.

When the monitoring view is designated by the view designation information (Ca) input from the input device 16, the view creation processing unit 21 performs the blocking luminance expansion processing by the blocking luminance expansion processing unit 20 (see FIG. 13). The monitoring view designated by the user is created based on the (circumference image) and displayed on the display unit (display device) 22. When the view is not specified, the 360 ° all-round monitor image corrected by the blocking luminance expansion processing unit 20 is displayed and output on the display unit (display device) 22.
The display unit (display device) 22 constitutes a display output unit that displays the monitoring target image subjected to the luminance expansion processing. In this embodiment, the display screen of the display device is composed of 640 × 480 pixels, the 360 ° all-round input image obtained by resizing the input image (Pa) stored in the image buffer memory 13 by the resizing unit 14, and the cut-out image buffer The cutout image stored in the memory 19 is selected as a display target, and one of the images is selected, or the display area is divided and simultaneously displayed and output. In this embodiment, when the monitoring view is not designated (when the view designation information (Ca) is not inputted), it is stored in the image buffer memory 13 (resized by the resizing unit 14). When the 360 ° all-round input image (Pa) is displayed and the user designates the monitoring view (when the view designation information (Ca) is in the operating state), the image is stored in the cut-out image buffer memory 19. A designated view using the clipped image is displayed.

  FIG. 2 shows the software (SW) configuration of the above-described omnidirectional monitoring image display processing system.

  The capture unit 12 captures, as an input image (Pa), a monitoring image in frame units (1280 × 960 pixels) captured by the camera 11 or a time-series recorded video in which the camera video is stored in the mass storage medium 10. With respect to this input image (Pa), the region-of-interest image creation processing unit 18 performs segmenting processing of a partial image (monitoring image) according to the preset information.

  The attention area image creation processing unit 18 uses the preset information (center position coordinates of the rectangular area) stored in the view parameter storage unit 17 (preset information buffer memory) to store the input image stored in the image buffer memory 13. (Pa) is cut out, distortion is corrected for the cut out image (q) cut out from the input image (Pa), and display based on each preset information suitable for the processing of the view creation processing unit 21 The cut-out image buffer memory 19 stores a cut-out image having a 640 × 480 pixel configuration (VGA) according to the pixel configuration.

  The blocking luminance expansion processing unit 20 processes the input image (Pa) stored in the image buffer memory 13 and the cut-out image stored in the cut-out image buffer memory 19 as a processing target in units of pixels. The blocking luminance expansion process is performed. When the monitoring view is designated by the view designation information (Ca), pixel level contrast correction is performed on the clipped image stored in the clipped image buffer memory 19 by blocking luminance expansion processing, and the monitor view is designated. If not, pixel level brightness correction is performed on the input image (Pa) stored in the image buffer memory 13 by blocking luminance expansion processing.

  When the monitoring view is designated by the view designation information (Ca), the view creation processing unit 21 allows the user to apply a monitoring image that has been subjected to fine dynamic range image correction in the pixel level blocking luminance expansion processing. A designated monitoring view (PTZ view shown in FIG. 14, preset view shown in FIG. 15, multipoint view shown in FIG. 16, or panoramic view shown in FIG. 17) is created and output to the display unit (display device) 22. . When the monitoring view is not designated, the 360 ° circumference monitoring image that is resized by the resizing unit 14 and expanded by the blocking luminance expansion processing unit 20 is output to the display unit (display device) 22.

  FIG. 3 is a flowchart showing the processing procedure of the omnidirectional monitoring image display processing system. Note that the registration of view parameters (storage of preset information for each monitoring view in the view parameter storage unit 17) for creating each monitoring view (PTZ view, preset view, multi-point view, panoramic view) has already been performed. Suppose that it is done.

  The capture unit 12 captures, as an input image (Pa), a monitoring image in frame units (1280 × 960 pixels) captured by the camera 11 or a time-series recorded video in which the camera video is stored in the large-capacity storage medium 10 ( Step S11).

  The region-of-interest image creation processing unit 18 performs a partial image cutout process according to the preset information on the input image (Pa). In addition, the resizing unit 14 performs a resizing process to a 640 × 480 pixel configuration (VGA) on the 360 ° all-round input image (Pa).

  When the monitoring view is designated by the view designation information (Ca), the attention area image creation processing unit 18 designates the view format (PTZ view, preset view, multipoint view, panoramic view) by the user. Correction processing that uses preset information according to the specified view format, performs cutout processing according to the preset information for an input image (Pa) of 360 degrees all around, and corrects distortion aberration for the cutout image Then, the resizing process according to the pixel configuration of the display device 22 is performed, and the clipped image subjected to the process is stored in the clipped image buffer memory 19 (step S12).

  The blocking luminance expansion processing unit 20 performs blocking luminance expansion processing for each pixel on the cutout image stored in the cutout image buffer memory 19 and corrects the complementary luminance at the pixel level for portions that are greatly different in light and dark (step S1). S13).

  The view creation processing unit 21 creates a configuration screen of the designated view using the clipped image subjected to the blocking luminance expansion processing by the blocking luminance expansion processing unit 20, and displays it on the display device 22 as a monitoring view (step S14). .

  If the view format is not specified by the user, the 360-degree all-round input image (Pa) resized by the resizing unit 14 is subjected to brightness correction at the pixel level by the blocking luminance expansion processing unit 20, and the view creation processing unit 21 is displayed on the display device 22 via 21 as a monitoring view.

  Configuration examples of the above-described 360 ° all-round input image (Pa) and the monitoring views (PTZ view, preset view, multi-point view, panoramic view) using the cut-out image cut out from the input image (Pa) This is shown in FIGS.

  Here, an omnidirectional monitoring image display processing system is realized in which a monitoring view according to a view format and position designation information designated by a user can be displayed on a 360 ° all-round surveillance video photographed by a monocular camera. For example, when the user designates a view format according to the view designation information (Ca), the monitoring view display function by the user designation displays a surveillance view for a cut-out image according to the view format designated by the user. The display function can be realized. Even in the display of the monitoring view specified by the user, blocking luminance expansion processing is performed on the clipped image to correct the dynamic range for easy monitoring, and the clipped monitored image is clearly displayed as a monitor view in detail. .

  Here, the display mode selection function of the monitoring image subjected to the blocking luminance expansion processing will be described with reference to FIGS. FIG. 13 is a view showing an example of a 360 ° all-round monitoring image (all-round image) according to the above embodiment, and FIGS. 14 to 17 are created by the view creation processing unit 21 using the all-round image shown in FIG. 13 as a basic image. It is a figure which shows the structural example of the display form (monitoring view) of the various monitoring images processed. FIGS. 14 to 17 show display examples of monitoring views displayed according to the view format designated by the user for the basic image (circumference image) shown in FIG. 14 shows a display form of a PTZ (Pan-Tilt-Zoom) view that is displayed by designating an arbitrary position. FIG. 15 shows a display form of a preset view that is displayed by switching registered positions. FIG. 16 is a diagram showing a display form of a multi-point view that simultaneously displays a plurality of registered points, and FIG. 17 is a figure showing a display form of a panoramic view that panoramicly displays the entire 360 ° circumference.

  In this embodiment, a view creation processing unit 21 is provided between the blocking luminance expansion processing unit 20 and the display unit (display device) 22 as a component for creating each of the monitoring views. In addition to the view creation processing unit 21, a view parameter storage that stores, as preset information, an input device 16 on which a user performs an operation input, and position designation information of various views input by the operation input of the input device 16. A portion 17 is provided.

  The view creation processing unit 21 follows the view designation information (Ca) input from the input device 16 and the view position designation information (Cb) input from the input device 16 and registered as preset information in the view parameter storage unit 17. Based on the basic image (all-round image) shown in FIG. 14 subjected to the blocking luminance extension processing by the blocking luminance extension processing unit 20, the monitor images of various display forms (view formats) shown in FIGS. 15 to 18 specified by the user are displayed. A process of creating and displaying on the display unit (display device) 22 as a monitoring view is performed.

  The input device 16 accepts input of view designation information (Ca) and view position designation information (Cb) by the user's operation, and sends the received view designation information (Ca) to the view creation processing unit 21. Is stored in the view parameter storage unit 17 as preset information. Here, preset information (PTZ view position information, preset view position information, multipoint view position information) can be registered in the view parameter storage unit 17 in units of view formats. Note that q1 to q4 shown in the basic image of FIG. 13 are rectangular cut-out image areas designated by the user's position designation information (Cb).

  The blocking luminance expansion processing unit 20 performs the blocking luminance expansion processing on the 360 ° all-round monitoring image resized by the resizing unit 14 or the cut-out image according to the position specification specified by the parameter, and the blocking luminance expansion processing image Is sent to the view creation processing unit 21.

  The view creation processing unit 21 performs the view for PTZ in the view format shown in FIG. 14 (designates an arbitrary position) according to the view designation information (Ca) instructed by the input device 16 for the image subjected to the blocking luminance expansion processing. PTZ view to be clipped and displayed, or preset view in the view format shown in FIG. 15 (preset view to switch and display the registered position), or view format multipoint view (multiple views shown in FIG. 16) A multi-view view that cuts and displays points simultaneously) or a panoramic view in the view format shown in FIG. 17 (a panoramic view that displays a panorama at the same time by dividing the entire 360 ° circumference by half) Output as.

  The display unit (display device) 22 displays and outputs the monitoring view created by the view creation processing unit 21. This monitoring view can be changed in the display of the monitoring view by previously rewriting the parameter (position information) stored (registered) in the view parameter storage unit 17 (from the input device 16 in advance). Preset information is stored in the view parameter storage unit 17 as designated external parameters, and preset information of each monitoring view (PTZ view, preset view, multi-point view, panoramic view) according to the position designation information from the input device 16 Update).

  With the monitoring view display function described above, it is possible to clearly display the entire video of the 360 ° monitoring video for each frame taken by the monocular camera as a panoramic view, for example, in detail, and 360 °. A part of the monitoring video can be cut out, and the cut-out image can be clearly displayed in detail as, for example, a PTZ view, a preset view, or a multi-point view.

  The all-round image (360 ° all-round monitoring image) or the cut-out image according to the parameter position designation subjected to the blocking luminance expansion processing in this way is displayed as a monitoring view on the display unit (display device) 22 in an arbitrary view format. Display output is possible. As a result, a clear monitoring image of an arbitrary region can be provided to the user in a view format desired by the user for all 360 ° all-round monitoring images.

  Note that the present invention is not limited to the above-described embodiment as it is, and can be embodied by modifying the constituent elements without departing from the scope of the invention in the implementation stage. In addition, various inventions can be formed by appropriately combining a plurality of components disclosed in the embodiment. For example, some components may be deleted from all the components shown in the embodiment. Furthermore, you may combine a component suitably in different embodiment. In addition, the method described in the above embodiment can be stored in a storage medium and distributed as a program that can be executed by a computer.

  Here, the configuration of the blocking luminance expansion processing unit 20 and the luminance expansion processing will be described with reference to FIGS.

  As described above, the blocking luminance expansion processing unit 20 inputs the input image to be subjected to the luminance expansion processing (the VGA 360-degree all-round input image resized by the resizing unit 14 or the VGA stored in the cut-out image buffer memory 19). For the clipped image), the image area (or the set attention area) is divided into a plurality of blocks, and for each of the divided blocks, based on the luminance values of the plurality of blocks obtained by convolving the luminance values of the surrounding blocks. A block processing unit that calculates a luminance expansion parameter for each block and creates a luminance conversion table for each block based on the luminance expansion parameter, and a block-based luminance conversion table created by the block processing unit. For each pixel in each of the divided blocks, the brightness change of the surrounding blocks is made for each block. Interpolation processing means for performing bilinear interpolation with reference to a table, and image conversion means for creating a luminance expansion output image for the input image based on the pixels of each block subjected to bilinear interpolation by the interpolation processing means It is provided and configured.

  The blocking luminance expansion processing unit 20 shown in FIG. 4 uses the image captured by the monocular camera 11 as an input image, performs luminance expansion processing (gamma correction) on the input image, and distributes the input image to the luminance value distribution of the input image. Convert from the state to the optimum gradation. Here, the luminance expansion processing is performed on the image of the attention area set for the input image of 0 to 255 gradations of VGA (640 × 480 pixels).

  In the luminance expansion processing in the luminance expansion processing unit 1, the luminance expansion parameter for luminance expansion necessary for gamma correction is divided with reference to the theory of gamma (γ) correction, and the area of interest in the input image is divided into a plurality of blocks. There is provided means for automatically calculating based on the average luminance value of the block unit image.

  The blocking luminance expansion processing unit 20 divides the set attention area into a plurality of blocks for the input image to be subjected to the luminance expansion processing, and convolves the luminance values of the surrounding blocks with respect to each of the divided blocks. A block processing unit that calculates a luminance expansion parameter for each block based on the luminance value of the block and creates a luminance conversion table for each block based on the luminance expansion parameter, and a block unit created by the block processing unit For each pixel of each divided block based on the luminance conversion table, an interpolation processing means for performing bilinear interpolation with reference to the luminance conversion table of the surrounding block for each block, and bilinear interpolation by the interpolation processing means Luminance expansion output image for the input image based on the pixels of each block subjected to Configured by including an image conversion means for generating.

  The luminance expansion processing operation of the blocking luminance expansion processing unit 20 shown in FIG. 4 will be described with reference to FIGS. FIG. 5 is an operation explanatory diagram for explaining the luminance expansion processing operation of the blocking luminance expansion processing unit 20, and FIGS. 6 and 7 are operation descriptions for explaining the operational principle of the luminance expansion processing in the block processing means described above. FIG. 8, FIG. 8 and FIG. 9 are operation explanatory views for explaining the operation principle of the convolution processing in the block processing means, and FIG. 10 is an operation for explaining the operation principle of the bilinear interpolation processing in the interpolation processing means. It is explanatory drawing.

  The block processing means 4a of the blocking luminance expansion processing unit 20 shown in FIG. 5 first divides a preset attention area 3 into a plurality of blocks for an input image (640 × 480 pixels) 2 photographed with a monocular camera. . As a specific example, the input image is divided into 32 × 32 pixel units, 16 × 16 pixel units, 8 × 8 pixel units, or the like, and attention area 3 (entire image or set attention area) is divided into a plurality of blocks. Divide into For example, the screen of the attention area 3 is divided into 16 × 16 pixel unit blocks (40 × 30 blocks). For each block division area, the average luminance is obtained using the luminance value of the surrounding block as a variable element.

  This operation principle is shown in FIGS. Here, as shown in FIG. 6 and FIG. 7, the 16 × 16 pixel pattern obtained by dividing the screen into blocks is converted into the value of one small block from the block 4 pattern shifted by 1/2 block (8 × 8 pixels). Think. Here, for a block (8 × 8 pixels) in which four patterns of blocks are overlapped by a 1/2 block shift, the average of the luminance of the overlapping blocks, the average of the square of the luminance, etc. are obtained, and based on this value Then, a luminance conversion table for luminance expansion, which is applied to a preset luminance expansion processing method, is created for each block. Further, the block processing means 4a calculates a luminance sum for each block for each of the divided blocks, and uses the calculated luminance sum for each block to weight and convolve each block with the block and its surrounding blocks. The total luminance of all the processed blocks is calculated, and the average luminance of each block is calculated. 8 and 9 show images of convolution counting and convolution processing at this time. Here, as shown in FIG. 8, “4” for the target block for which the average luminance is to be calculated, and “2” for the four blocks above, below, left, and right among the blocks near the target block. The convolution count of “1” is given to each block in the four diagonal directions, and the luminance value of each block is convolved with the image of data accumulation shown in FIG. Can be requested. Based on the average luminance value for each block, a luminance expansion parameter and a luminance conversion table for each block to be applied to a preset luminance expansion processing method are created.

  Next, in the interpolation processing unit 4b of the blocking luminance expansion processing unit 20, for each pixel of each block using the luminance conversion table according to the luminance expansion parameter obtained in units of the block, the interpolation unit 4b performs the corresponding block for each block. Then, bilinear interpolation is performed using the luminance conversion table of the surrounding four blocks. In this bilinear interpolation, as shown in FIG. 10, the conversion value of the target pixel is determined and set by linear interpolation (bilinear interpolation) of conversion values of the surrounding four blocks (A, B, C, D). Here, the block position where the pixel of interest is located and the positional difference between the pixel of interest and the corresponding block are calculated with a decimal point, and the value of the luminance conversion table of the block and surrounding blocks is multiplied by the coefficient of the block position difference to perform linear interpolation. The luminance conversion value is obtained by the principle of (bilinear interpolation).

  Based on the pixels of each block subjected to bilinear interpolation by the interpolation processing unit 4b, a luminance expansion output image 5 for the input image is created.

  As described above, the block processing unit 4a of the blocking luminance expansion processing unit 20 obtains the luminance expansion parameter and the luminance conversion table for each image block, and the interpolation processing unit 4b of the blocking luminance expansion processing unit 20 uses the bilinear block. By obtaining a converted luminance value by interpolation and creating a luminance expansion output image 5 for the input image based on the pixels of each block subjected to bilinear interpolation (reflecting luminance expansion parameters of surrounding blocks) The bright and dark images locally existing in a part of the attention area can be clearly displayed in gradation without causing a luminance difference at the block boundary.

  An example that further embodies the above embodiment is shown in FIGS. FIG. 11 is a block diagram illustrating a functional configuration of the blocking luminance expansion processing unit 20 according to the embodiment, and FIG. 12 is a flowchart illustrating a processing procedure of the blocking luminance expansion processing unit 20.

  The blocking luminance expansion processing unit 20 has the functions of the block processing means (4a) and the interpolation processing means (4b) described above, generates a luminance expansion parameter and a luminance conversion table for each image block by the block processing function, and performs interpolation. The converted luminance value of the pixel of each block is obtained by bilinear interpolation using surrounding blocks by the processing function.

  The configuration shown in FIG. 11 shows an example in which the block division luminance expansion processing function according to the embodiment of the present invention is applied to a non-linear luminance expansion processing method using a non-linear curve (convex curve, concave curve).

  In this nonlinear luminance expansion processing method, it is determined whether an average luminance value calculated by an average luminance calculation unit 222 described later is equal to or less than the central luminance value (127) of 256 gradations, and when it is equal to or less than the central luminance value A reference table with a convex curve is created as a luminance conversion table for luminance expansion for the input image. When the central luminance value is exceeded, a reference table with a concave curve is created as a luminance conversion table for luminance expansion for the input image. This nonlinear luminance expansion processing function is applied to the already-known nonlinear luminance expansion processing method according to Japanese Patent No. 4340303, and the gist thereof will be described later.

  As shown in FIG. 11, the blocking luminance expansion processing unit 20 includes a block reference luminance setting unit 221, an average luminance calculation unit 222, a power parameter calculation unit 223, a reference table determination unit 224, an image conversion unit 225, And a linear interpolation (bilinear interpolation) calculation unit 226.

  The block reference luminance setting unit 221 divides the attention area of the input image captured and input by the camera 11 into a plurality of blocks according to the block size designated from the outside, and sums the luminance value of each block for each block. calculate. Here, any block size (for example, 16 × 16 pixels) of 8 × 8 pixels, 16 × 16 pixels, 24 × 24 pixels, and 32 × 32 pixels is designated, and attention is paid to the input image according to the designated size. The area is divided into blocks. When an area corresponding to a fractional pixel smaller than the specified block size occurs in the edge portion of the input image, the area is excluded from the luminance expansion processing target or other luminance expansion processing means is applied.

  The average luminance calculation unit 222 divides the block into blocks by the block reference luminance setting unit 221 and, based on the luminance sum calculated for each block, all blocks subjected to weighting and convolution processing including the block and its surrounding blocks. Is calculated, and the average luminance value (= M) of the image in block units is calculated.

  The power parameter calculation unit 223 is applied in the above-described nonlinear luminance expansion processing method. Based on the average luminance calculated by the average luminance calculation unit 222, an input image (attention area) according to the equation (1) described later. A power parameter for the luminance expansion processing for is calculated.

  The reference table determination unit 224 determines whether the average luminance value is equal to or less than the central luminance value (127) or exceeds the central luminance value (127), and the average luminance value is the central luminance value (127). When the following is true, the table creation unit 231 creates a reference table with a convex curve according to equation (2) described later using the power parameter as a luminance conversion table (reference table) 232 for luminance expansion for the input image, and When the average luminance value exceeds the central luminance (127), the table creation unit 231 uses the above power parameter to convert a reference table with a concave curve according to the following equation (3) into a luminance conversion table 232 for luminance expansion for the input image. Create as.

  The image conversion unit 225 uses the luminance conversion table (reference table) 232 created by the reference table determination unit 225 to perform a luminance conversion process on the input image (attention area) held in the image buffer 21, and Then, the image whose output luminance is converted is written back to the image buffer 21 to create a luminance expanded image on the image buffer 21.

  As shown in FIG. 10, the linear interpolation calculation unit 226, for each block pixel using the luminance conversion table according to the luminance expansion parameter obtained in units of blocks, for each block, the luminance of the block and the surrounding four blocks Bilinear interpolation is performed using the conversion table.

  FIG. 12 shows a processing procedure of the blocking luminance expansion processing unit 20 described above.

  The blocking luminance expansion processing unit 20 performs the process shown in FIG. 12 every time an input image in frame units captured by the camera 11 is held in the image buffer memory 13 via the capture unit 12.

  When the input image (Pa) to be subjected to luminance expansion processing is held in the image buffer memory 13, the input image is resized by the resizing unit 14 and input to the blocking luminance expansion processing unit 20. In the blocking luminance expansion processing unit 20, the block reference luminance setting unit 221 sets the attention area (entire image or set attention area) of the input image captured and input by the camera 11 to a block size (for example, 16) designated from the outside. Is divided into a plurality of blocks (40 × 30 blocks) in accordance with (× 16 pixels), the sum of the luminance values of each block is obtained for each block, and the luminance sum of the block unit is used as the luminance information of the block of interest to calculate the average luminance (Step S21).

  When the average luminance calculation unit 222 receives the luminance information of the block of interest (total luminance in units of blocks) from the block reference luminance setting unit 221, based on this luminance information, as shown in FIG. 8 and FIG. For each block, the total luminance of all blocks subjected to weighting and convolution processing including the block and its surrounding blocks is calculated to calculate the average luminance value of the block unit (target block) image, and the average luminance of the image of the target block The value is sent to the power parameter calculation unit 223 and the reference table determination unit 224 (step S22).

  The power parameter calculation unit 223 calculates a power parameter for luminance expansion processing for the input image (area of interest) according to the equation (1) described below based on the average luminance value calculated by the average luminance calculation unit 222, The parameter is sent to the reference table determination unit 224.

  The reference table determination unit 224 uses the average luminance value input from the average luminance calculation unit 222 and the power parameter input from the power parameter calculation unit 223 as a luminance expansion parameter, and creates a luminance conversion table according to the luminance expansion parameter. Here, it is determined whether the average luminance value is the central luminance value (127) or less or exceeds the central luminance value (127), and the average luminance value is the central luminance value (127) or less. The table creation unit 231 creates a reference table with a convex curve according to the equation (2) described later using the power parameter as a luminance conversion table (reference table) 232 for luminance expansion with respect to the input image, and the average luminance value is When the central luminance (127) is exceeded, the table creation unit 231 creates a reference table with a concave curve according to the later-described equation (3) using the power parameter as the luminance conversion table 232 for luminance expansion for the input image ( Step S23).

  Here, the luminance expansion processing in the nonlinear luminance expansion processing method using the convex curve and the concave curve described above will be described.

  The power parameter calculation unit 223 automatically calculates a power parameter γ for luminance expansion based on the average luminance of the area of interest using the following equation (1).

γ = (α / Bmed ² ) × (Iave−Bmed) ² +1.0 (1)
(Where α is a fixed value, Bmed is the median brightness, Iave is the average brightness of the input image area of interest)
The average luminance is compared with the central luminance of the luminance gradation range of the input image, and the following equation (2) or (3) is automatically switched using the power parameter γ, and the convex curve and the concave curve are Create a brightness conversion table.

I ′ = Bmax − ((Bmax−I) / Bmax) ^γ × Bmax: Iave ≦ Bmed (2) Convex curve
I ′ = I / Bmax ^γ × Bmax: Iave> Bmed (3) Concave curve (where I ′ is the output luminance, I is the input luminance, Bmed is the median luminance, and Bmax is the maximum luminance value) , Iave is the average brightness of the input image attention area)
The convex curve and concave curve luminance conversion table 232 created by the reference table determination unit 224 is sent to the image conversion unit 225 (step S23).

  The image conversion unit 225 uses the luminance conversion table (reference table) 232 created by the reference table determination unit 225 to perform luminance conversion processing on the input image (attention area) held in the image buffer 21. For each pixel of the target block whose luminance has been expanded in units of blocks, the linear interpolation calculation unit 226 uses the luminance conversion table of the target block and the surrounding four blocks as shown in FIG. Bilinear interpolation is performed in accordance with the distance difference, and an image whose luminance has been converted by this bilinear interpolation is written back to the image buffer 21, whereby a luminance expanded image obtained by performing bilinear interpolation between blocks is created on the image buffer 21. (Steps S24 and S25).

  In the above-described embodiment, the VGA (640 × 480 pixels) 0 to 255 gradation camera video is used as the input image to be subjected to the luminance expansion processing. However, the present invention is not limited to this, and an image that handles other image configurations. The luminance expansion processing function according to the embodiment of the present invention can also be applied to the processing device. Further, in the above embodiment, 20 components (block reference luminance setting unit 221, average luminance calculation unit 222, power parameter calculation unit 223, reference table determination unit 224, image conversion unit 225, linear, which realize the luminance expansion processing function. As for the interpolation calculation unit 226 and the like, part or all of the functions can be realized by software processing.

  In the embodiment described above, the sum of the luminance values of each block is obtained for each block, and the sum of the luminance of all blocks subjected to weighting and convolution processing including the target block and its surrounding blocks is calculated to be a block unit (target block). However, the present invention is not limited to this. For example, it is also possible to calculate a luminance average for each block and perform weighting and convolution processing using the average luminance for each block. .

  In the embodiment described above, the block division luminance expansion processing function according to the embodiment of the present invention is applied to a nonlinear luminance expansion processing method using a non-linear curve (convex curve, concave curve). For example, the present invention can be applied to a sigmoid luminance expansion processing method using a sigmoid function curve and other various luminance expansion processing methods.

  By the above-described blocking luminance expansion process, a greatly different portion of brightness and darkness is corrected at the pixel level, and a display function of the entire periphery monitoring image having a dynamic range that can be easily monitored can be realized.

  In the embodiment described above, when there is no cut-out image based on the preset information, the 360 ° all-round input image (all-round image) subjected to the blocking luminance expansion processing is displayed on the display unit (display device) 22 and the preset information is used. When a clipped image (view display) is designated, the display function for displaying the clipped image subjected to the blocking luminance expansion processing unit 20 on the display unit 21 is realized. ) Is designated, it is also possible to simultaneously display the clipped image designated by the preset information together with the all-round image. At this time, the all-round image is displayed on the entire display screen of the display unit 21, and the clipped image is displayed on the all-round image in a balloon format corresponding to the clip position of the all-round image (covering the all-round image). Is also possible. Alternatively, the clipped image can be partially displayed in a predetermined partial display area on the display unit 21.

  2 ... Image captured by a monocular camera (input image), 3 ... Area of interest target image, 4a ... Block processing means, 4b ... Interpolation processing means, 5 ... Luminance expansion output image, 10 ... Mass storage medium, 11 ... Camera ( (Monocular camera (IP camera)), 12 ... capture unit, 13 ... image buffer memory, 14 ... resize unit, 16 ... input device for instructing view creation, 17 ... view parameter storage unit (preset information buffer memory), 18 ... Region-of-interest image creation processing unit, 19 ... cut-out image buffer memory, 20 ... blocking luminance expansion processing unit, 21 ... view creation processing unit, 22 ... display unit (display device), 221 ... block reference luminance setting unit, 222 ... average luminance Calculation unit, 223 ... Power parameter calculation unit, 224 ... Reference table determination unit, 225 ... Image conversion unit, 226 ... Luminance enhancement The output image, 231 ... table creating unit, 232 ... luminance conversion table (see table).

Claims (9)

An omnidirectional monitoring image display processing system capable of displaying a monitoring view according to position designation information designated by a user with respect to a 360 ° all-round monitoring video imaged by a monocular camera,
Monitoring video input means for capturing the 360 ° all-round monitoring video in frame units;
A correction process for cutting out an image of a partial area designated by the position designation information on the monitoring video in frame units captured by the monitoring video input means, and correcting distortion aberration for the cut out image Monitoring image processing means for performing
Blocking luminance expansion processing means for performing blocking luminance expansion processing in units of pixels on the cut-out image with the distortion corrected;
Display output means for displaying the clipped image subjected to the blocking luminance expansion processing as a monitoring image;
An omnidirectional monitoring image display processing system comprising: An omnidirectional monitoring image display processing system capable of displaying a monitoring view according to position designation information and view designation information designated by a user with respect to a 360 ° all-round monitoring video imaged by a monocular camera,
Monitoring video input means for capturing the 360 ° all-round monitoring video in frame units;
Input image storage means for storing the monitoring video in frame units captured by the monitoring video input means as an input image;
A correction process for cutting out an image of a partial area designated by the position designation information on the monitoring video in frame units captured by the monitoring video input means, and correcting distortion aberration for the cut out image Monitoring image processing means for performing
Cutout image storage means for storing the cutout image corrected for distortion as a view designation monitoring image;
Blocking luminance expansion processing means for performing a block-by-pixel blocking luminance expansion process on the view designation monitoring image stored in the cut-out image storage means;
For the view designated monitoring image subjected to the luminance expansion processing, a view screen in a monitoring form according to the view format designated by the view designation information is created from a plurality of view formats prepared in advance. View creation processing means;
Display output means for displaying the view screen created by the view creation processing means;
An omnidirectional monitoring image display processing system comprising: The view creation processing means, according to the view designation information,
A PTZ view format for displaying a clipped image of an arbitrary point designated by the position designation information in accordance with an arbitrary pan, tilt and zoom designation;
A preset view format for sequentially switching and displaying cut-out images of a plurality of points designated by the position designation information every predetermined time;
A multi-point view format that simultaneously displays a plurality of cut-out images designated by the position designation information by dividing an area on the screen;
The omnidirectional monitoring image display processing system according to claim 2, wherein a view screen in a monitoring form according to any one of the view formats is created. The blocking luminance expansion processing means performs a pixel-by-pixel blocking luminance expansion process on either the view designation monitoring image stored in the cut-out image storage means or the 360 ° all-round input image stored in the input image storage means. Carried out,
The view creation processing means, according to the view designation information,
A PTZ view format for displaying a clipped image of an arbitrary point designated by the position designation information in accordance with an arbitrary pan, tilt and zoom designation;
A preset view format for sequentially switching and displaying cut-out images of a plurality of points designated by the position designation information every predetermined time;
A multi-point view format that simultaneously displays a plurality of cut-out images designated by the position designation information by dividing an area on the screen;
A panoramic view format in which the 360 ° full-circle input image that has been subjected to the blocking luminance expansion processing is divided into upper and lower portions and displayed simultaneously;
The omnidirectional monitoring image display processing system according to claim 2, wherein a view screen in a monitoring form according to any one of the view formats is created.   When the view designation is not performed, the blocking luminance expansion processing unit performs the blocking luminance expansion processing on the 360 ° all-round input image stored by the input image storage unit, and monitors the blocking luminance expansion processing. 5. The omnidirectional monitoring image display processing system according to claim 1, wherein the image is sent to the display output means. The blocking luminance expansion processing means includes
For an input image that is subject to luminance expansion processing, the set area of interest is divided into a plurality of blocks, and for each divided block, the luminance values of the surrounding blocks are convolved with the luminance values of the plurality of blocks. Block processing means for calculating a luminance expansion parameter in units of blocks and creating a luminance conversion table in units of blocks based on the luminance expansion parameters;
Interpolation processing for performing bilinear interpolation with reference to the luminance conversion table of the surrounding block for each pixel for each pixel of each divided block based on the luminance conversion table in block units created by the block processing means Means,
Image conversion means for creating a luminance expansion output image for the input image based on the pixels of each block subjected to bilinear interpolation by the interpolation processing means;
The omnidirectional monitoring image display processing system according to claim 1 or 2, further comprising:   The block processing means uses the adjacent upper, lower, left, and right blocks B and the adjacent diagonal blocks C for the target block A to be processed among the divided blocks, and weights A> B> C by weighting A> B> C The omnidirectional monitoring image display processing system according to claim 6, wherein a convolution process is performed. The block processing means includes
Block dividing means for dividing the image of the area of interest into a plurality of blocks according to a block size designated from the outside;
The luminance sum for each block is calculated for each of the blocks divided by the block dividing means, and the weighting and convolution processing including the block and its surrounding blocks is performed for each block using the calculated luminance sum for each block. A luminance sum calculation means for calculating the luminance sum of all blocks;
Using the luminance sum calculated by the luminance sum calculating means, for each block, the luminance average calculating means for calculating the luminance average of the block;
Luminance expansion parameter calculation means for calculating a luminance expansion parameter for each block based on the luminance average of the block calculated by the luminance average calculation means;
Based on the luminance expansion parameter calculated by the luminance expansion parameter calculating unit, a luminance conversion table creating unit that creates a luminance conversion table for luminance expansion for each block;
The omnidirectional monitoring image display processing system according to claim 7, comprising: The monocular camera 11 outputs a 360 ° all-round high-definition monitoring image of an ellipsoid having a number of pixels of 1.3 M pixels or more, in which the long and short sides of the ellipsoid are matched to the width and height of the frame,
The blocking luminance expansion processing unit performs a blocking luminance expansion process on a per-pixel basis on a monitoring video obtained by resizing the pixel configuration of the 360 ° all-round high-definition monitoring video on the elliptical surface according to the pixel configuration of the display output unit. The omnidirectional monitoring image display processing system according to claim 1, wherein the omnidirectional monitoring image display processing system is implemented.

Images