JP2008227562A - Image processor, camera apparatus and camera system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processor, a camera apparatus and a camera system which have improved real time property while suppressing cost increase in terms of a system, can prevent an important scene from being overlooked, have improved visibility and can switch distortion correction images in a time division manner and output them on the basis of a correction parameter. <P>SOLUTION: The camera apparatus 10 is provided with the image processor capable of correcting the distortion of source images imaged by a wide-angle lens using a distortion correction parameter formed in a grid shape. The image processor is provided with processing parts 14-22 for correcting the distortion, also switching the plurality of arbitrarily images in a time division manner and outputting them. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to an image processing apparatus, a camera apparatus, and a camera system that have a function of correcting distortion of an image captured by, for example, a wide-angle lens, can perform electronic pan / tilt zoom, and can be applied to a surveillance camera system or the like.

  A general mechanical pan / tilt / zoom camera is configured to pan, tilt, and zoom by driving a motor, and has a capability of monitoring a wide angle.

  A camera capable of correcting the distortion of an image captured at a wide angle and performing electronic pan / tilt has a function of synthesizing and outputting images of a plurality of areas in one screen of an output image.

  Also known is a system capable of electronic panning, tilting, and zooming by correcting distortion of an image of a camera that outputs a wide-angle captured image as it is (with a wide-angle lens attached to the camera) by a host system at a later stage. .

For example, in this type of system, an image before distortion correction captured by an ITV camera equipped with an ultra-wide-angle lens is transmitted to a host computer, and the host computer captures the image and performs image processing such as distortion correction, clipping, and composition. There has been proposed a system for performing and displaying on a display (see, for example, Patent Document 1).
JP-A-7-93558

By the way, since the mechanical pan / tilt zoom camera performs pan / tilt zoom by driving a motor, the response speed is slow, and it takes time to obtain an image of a truly desired area. There was a possibility of missing the scene.
Similarly, when this is recorded in a video, there is a possibility that important scenes may be missed.

Some cameras capable of correcting the distortion of wide-angle captured images and performing electronic pan / tilt have the function of synthesizing and outputting images of a plurality of areas in one screen of the output image, but the output image size is NTSC. Because of the VGA level, when images of a plurality of areas are combined, the resolution of each image is further lowered and the visibility is poor.
Similarly, when this is recorded in a video or the like, the synthesized image is recorded, so the resolution at the time of reproduction remains low and the visibility is poor.
A plurality of areas may be output separately, but in this case, components for transmitting / receiving or superimposing an output image and a transmission cable are required, which may increase system cost.

  In addition, the system that can correct the distortion of the wide-angle captured image by the host system in the subsequent stage and can perform electronic pan / tilt zoom, the original output image size is about NTSC or VGA. The visibility was poor.

  Thus, although all currently known cameras and systems have the ability to monitor wide angles, they are not truly a substitute for multiple cameras, resulting in the number of cameras installed in the past. Therefore, there is a disadvantage that the system cost increases.

  The present invention can improve the real-time performance while preventing an increase in system cost, prevent an important scene from being overlooked, improve the visibility, and time-divide the distortion-corrected image based on the correction parameters. It is to provide an image processing apparatus, a camera apparatus, and a camera system that can be switched and output.

  A first aspect of the present invention is an image processing apparatus capable of performing distortion correction on an original image captured by a wide-angle lens using a distortion correction parameter formed in a lattice shape. A processing unit that switches the images to time division and outputs them.

  A second aspect of the present invention is a camera device having at least one of an electronic pan, tilt, and zoom function, and includes an imaging device and a wide-angle lens, and forms an object image on the imaging device. And an image processing device capable of correcting the distortion of the original image captured by the wide-angle lens by the imaging element using a distortion correction parameter formed in a lattice shape, and the image processing device performs distortion correction. In addition, a processing unit that outputs a plurality of arbitrary images by switching to time division is included.

  Preferably, the processing unit converts an original image captured through the image sensor at a plurality of arbitrary pan angles, tilt angles, and / or zoom angles at every output frame rate regardless of the frame rate of the image sensor. In addition, it is possible to switch to time division and output.

  Preferably, the processing unit may output an image obtained by correcting distortion based on a plurality of arbitrary distortion correction parameters by switching to time division for each output frame rate regardless of the frame rate of the image sensor. Is possible.

  Preferably, the processing unit has an output frame rate lower than a frame rate of the image sensor.

  Preferably, the processing unit has a function of performing output control of divided images based on setting information.

  Preferably, the processing unit can convert a control code or a control code into an image pattern and insert it into an output image.

Preferably, the processing unit can convert a control code or a control code into an image pattern and insert it into an output image.
The control code includes at least one of a distortion correction parameter type, pan angle, tilt angle, and zoom angle of view.

  A camera system according to a third aspect of the present invention includes a camera device having at least one of an electronic pan, tilt, and zoom function, and a controller that performs predetermined processing on a processed image transferred from the camera device. The image processing apparatus includes an image sensor and a wide-angle lens, and forms an optical system that forms a subject image on the image sensor and an original image captured by the wide-angle lens using the image sensor in a grid pattern. An image processing apparatus capable of performing distortion correction using the distortion correction parameter, and the image processing apparatus performs distortion correction, switches a plurality of arbitrary images to time division, and controls the image in the image. Including a processing unit that converts a code or control code into an image pattern, inserts and outputs the image, and the controller converts the control code or control code from the received image Detecting a pattern of the image, it performs a predetermined process based on the information.

  According to the present invention, it is possible to improve the real-time property while preventing an increase in system cost, prevent an important scene from being overlooked, improve the visibility, and generate a distortion correction image based on the correction parameter. The output can be switched to time division.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration example of a camera apparatus that employs an image processing apparatus according to an embodiment of the present invention.

As shown in FIG. 1, the camera device 10 according to the present embodiment includes an optical system 11, an image sensor (imaging sensor) 12, a camera signal processing unit 13, a first frame memory (FRM0) 14, and a second frame memory (FRM1). ) 15, a write address generation unit 16, a read address conversion unit 17, a switch 18, an output processing unit 19, a microcomputer 20, a distortion correction parameter memory (RAM) unit 21, and a distortion correction parameter switching unit 22.
The frame memories 14 and 15, the write address generation unit 16, the read address conversion unit 17, the switch 18, the output processing unit 97, the microcomputer 20, the distortion correction parameter RAM unit 21, and the distortion correction parameter switching unit 22. A processing unit of the image processing apparatus is configured.

  In the present embodiment, the distortion correction parameter in the lens distortion aberration is simply referred to as a distortion correction parameter.

The camera device 10 according to the present embodiment uses a distortion correction parameter (vector) stretched in a lattice shape based on an image captured by a wide-angle lens having a point-symmetric distortion about the optical axis, and performs distortion correction and electronic pan tilt. The camera is configured to be capable of switching and outputting images of a plurality of arbitrary regions in a time division manner.
The camera device 10 outputs a plurality of images having an arbitrary pan angle, tilt angle, and zoom angle of view by switching to time division for each output frame rate regardless of the frame rate of the imaging sensor 12 as an image sensor. It is possible.
In addition, the camera device 10 can output an image obtained by correcting distortion based on a plurality of arbitrary distortion correction parameters by switching to time division for each output frame rate regardless of the frame rate of the imaging sensor 12. It is.
In addition, the camera device 10 has a configuration in which the frame rate of the imaging sensor 12 is significantly higher than the frame rate of the output, for example, the frame rate of the imaging sensor 12 is 240 fps, the output frame rate is 60 fps, etc. Some configurations are also possible.
Then, the camera device 10 converts (modulates) information such as the type of distortion correction parameter, pan angle, tilt angle, zoom angle of view, etc. into a control code or an image pattern in the output image, It can be inserted.
Note that the camera apparatus 10 has a setting for switching to time division in the camera itself, and has a function of performing output control of divided images based on the setting information.

The camera apparatus 10 according to the present embodiment performs clipping (pan / tilt / zoom), rotation, mirror image processing, composition, and the like while correcting distortion of an image captured at an ultra-wide angle by using distortion correction parameters.
At this time, the microcomputer 20 may store the distortion correction parameter in a memory such as a built-in ROM / RAM in advance, or may obtain it by calculation of the microcomputer 20, or the camera device 10 is connected by a transmission line. It can also be configured to receive by external communication from a host computer (not shown).

  The optical system 11 includes a wide-angle lens 111 formed by, for example, a super-wide-angle lens, and forms a subject image that has passed through the wide-angle lens 111 on the imaging surface of the image sensor 12.

The imaging element 12 is configured by an imaging sensor which is, for example, a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) device.
The imaging sensor (imaging device 12) detects a subject image by the optical system 11 by an optical sensor arranged in a matrix on a semiconductor substrate to generate a signal charge, and the signal charge is transmitted via a vertical signal line or a horizontal signal line. The digital image signal of the subject is output to the camera signal processing unit 13.

The camera signal processing unit 13 performs processing such as color interpolation, white balance, YCbCr conversion processing, compression, and filing, and outputs the results to the first frame memory 14 and the second frame memory 15.
The image output from the camera signal processing unit 13 is output as an image (original image) S13 that has not been subjected to distortion correction.

The first and second frame memories 14 and 15 are switched as a target for writing the original image output from the camera signal processing unit 13 for each frame rate of the imaging sensor 12, and are specified by the write address generation unit 16. Are stored in sequence.
The first and second frame memories 14 and 15 are supplied with real addresses obtained by converting the distortion correction parameters (vectors) switched by the distortion correction parameter switching unit in the read address conversion unit 16, and continuously changing distortion. Distortion correction of the correction target portion of the original image stored by the correction parameter is performed.
The image data read from the first frame memory 14 and the second frame memory 15 becomes an image subjected to distortion correction, clipping, synthesis, and the like, and is selectively output to the output processing unit 19 by the switch 18.

The read address conversion unit 17 switches the frame memory to be read for each frame rate of the imaging sensor and executes (X, Y) indicating the distortion correction parameter vector sequentially supplied by the distortion correction parameter switching unit 22. It is converted into an address and supplied to the first frame memory 14 and the second frame memory 15.
In the frame memories 14 and 15, smooth distortion correction of the correction target portion of the original image is performed using distortion correction parameters that change substantially continuously.

  The switch 18 has an operating contact a connected to the output of the first frame memory 14, an operating contact b connected to the output of the second frame memory 15, and a fixed contact c connected to the input of the output processing unit 19, For example, when reading data from the first frame memory 14 under the control of the microcomputer 20, the fixed contact c is connected to the operating contact a, and when reading data from the second frame memory 15, the fixed contact c is connected to the operating contact b. Is done.

  The output processing unit 19 inserts a control code or an image pattern obtained by converting (modulating) the control code into the image data read from the frame memory 14 and subjected to distortion correction, clipping, synthesis, and the like. Then, gamma processing, mask processing, format conversion, etc. are performed and output to the outside as a signal S19.

The microcomputer 20 stores a distortion correction parameter indicating an original image portion to be corrected for distortion in the RAM 21-1, 21-2, 21-3, 32-4 in the distortion correction parameter storage memory (RAM) unit 21. Further, the memory address MADR for access is output to the distortion correction parameter storage memory (RAM) unit 21.
In the example of FIG. 1, the number of RAMs in the distortion correction parameter storage memory (RAM) unit 21 is four, but this is only an example, and distortion correction parameter switching is performed while rewriting one RAM from the microcomputer. The number of RAMs is arbitrary because they may be transferred to the part.

  The distortion correction parameter switching unit 22 switches four types of distortion correction parameters for each output frame rate in accordance with an instruction from the microcomputer 20. It is also possible not to switch according to instructions from the microcomputer 20.

  Here, the distortion correction parameter, the distortion correction technique, and the like for the wide-angle captured image according to the present embodiment will be described more specifically.

<Summary of distortion correction technology and distortion correction parameters>
First, the correction technique will be briefly described.
FIG. 2 is a diagram illustrating the concept of imaging using a wide-angle lens.
In FIG. 2, a circle indicated by 31 is projected by the super-wide-angle lens 111, and 32 indicates the imaging surface of the image sensor 12.
As shown in FIG. 2, the distortion correction parameter is calculated by projecting the grating showing the distortion aberration of the wide-angle lens 111 onto the imaging surface (plane) 32 to obtain a grating showing the distortion aberration of the wide-angle lens. Then, the vector (X, Y) of each grating is obtained so that the grating showing the distortion aberration of the wide-angle lens 111 matches the square grating of the imaging surface 32 as shown in FIG.

Therefore, as shown in FIG. 4, the distortion correction parameter indicates each lattice point in the output image with a vector (X, Y) indicating each lattice point of the original image before distortion correction.
At this time, the angle of view (zoom magnification) can be changed by the distance f of the normal line OP in FIG. 2, and the area to be cut out can be changed (pan / tilt) by the zenith angle θ and the azimuth angle φ.
Interpolation is performed between lattice points using a vector of surrounding lattice points at the lattice point of interest.
The interpolation algorithm at this time may be a nearest neighbor method, a linear interpolation method (Bilinear), a cubic interpolation method (Bicubic), or an interpolation method based on lens distortion.

  Next, a method that can be adopted in the camera device 10 of FIG. 1 and corrects distortion of images in a plurality of arbitrary regions and outputs the images by switching to time division will be described.

FIG. 5 is a conceptual diagram in the case of using distortion correction parameters corresponding to four regions, and FIG. 6 is a diagram illustrating the timing for correcting distortion in the four regions in FIG.
Here, as an example, as shown in FIG. 5, when four distortion correction parameters <1>, <2>, <3>, <4> are used, distortion correction is performed on the four areas, and output is performed in a time-sharing manner. Will be explained.
6A shows the timing ISTMG of the imaging sensor 12, FIG. 6B shows the original image S13, FIG. 6C shows the write timing FWTMG of the frame memory, and FIG. 6D shows the first frame memory 14. (FRM0) write / read processing FRWR0, FIG. 6 (E) shows the second frame memory 15 (FRM1) write / read processing FRWR1, FIG. 6 (F) shows the frame memory read timing FRTMG, FIG. G) shows the output timing OTMG, FIG. 6 (H) shows the distortion correction parameter switching timing PSWTMG, and FIG. 6 (I) shows the distortion correction image output IMOUT.

The original image S13 of the imaging sensor 12 is written into the first frame memory 14 (FRM0) when the frame memory write timing FWTMG is 0, and when the frame memory write timing FWTMG is 1, the second frame memory 15 (FRM1). ).
The frame memory read timing FRTMG indicates a logic opposite to the frame memory write timing FWTMR, and when the frame memory read timing FRTMG is 0, the first frame memory 14 (FRM0) is a read target, that is, a distortion correction target image. When the frame memory read timing FRRMG is 1, the second frame memory 15 (FRM1) is set as a read target, that is, a distortion correction target image.

  The distortion correction parameter switching unit 22 sequentially switches the distortion correction parameters <1>, <2>, <3>, and <4> for each output timing OTMG, and returns to the original image that is the readout target, that is, the distortion correction target. On the other hand, by correcting distortion, an image of an arbitrary area is generated in a time division manner.

Here, as an example, four regions are time-divisioned in one frame of the imaging sensor 12, but it goes without saying that the number of the regions is not limited as long as distortion correction processing is possible. .
In addition, the pan angle, tilt angle, and zoom angle of view of the four regions are arbitrary, and processing such as rotation, mirror image, and composition, and images without distortion correction may be used, that is, distortion correction parameters are arbitrary. Not too long.

Further, although the output frame rate decreases, as shown in FIGS. 7 and 8A to 8C, for example, six distortion correction parameters <1>, <2>, <3>, <4>, Using <5> and <6>, distortion may be corrected in the six regions, and the plurality of frames of the imaging sensor 12 may be output in a time division manner.
When this is realized with the configuration of FIG. 1, since the number of RAMs in the RAM unit is insufficient, the distortion correction parameters <1>, <2>, <3 are determined by the microcomputer 20 at the frame interval of the imaging sensor 12. >, <4>, <5>, <6> need to be rewritten to RAM.

If the frame rate of the imaging sensor 12 is a high frame rate, the distortion correction parameter may be switched at the frame interval of the imaging sensor 12 and output in a time-sharing manner.
For example, if the frame rate of the imaging sensor 12 is 240 fps and four regions are output in a time-sharing manner, the frame rate of each distortion-corrected image is 60 fps. In this case, the dynamic resolution of the output image is improved.
FIGS. 9A to 9C show an example of timing when the distortion correction parameter is switched and time-division output is performed at the frame interval of the imaging sensor.
Needless to say, the four regions may be switched at intervals of a plurality of frames instead of switching every frame of the output.
At this time, the frame rate of the imaging sensor is arbitrary.

  Next, a setting for switching to time division and a control example by the microcomputer 20 will be described.

In camera device 10, the total number to be switched to time division, the type of distortion correction parameter to be switched to time division, or the coordinates (X, Y, Z) for distortion correction indicating pan angle, tilt angle, zoom angle of view, and The microcomputer 20 controls the number of output frame intervals to be switched as the control table CTBL.
At this time, the control table CTBL may be stored in advance in a memory such as a ROM, or may be received by external communication from the host system.

FIG. 10 is a diagram illustrating an example of a control table according to the present embodiment.
The control table CTBL in FIG. 10 has the total number of distortion correction parameters switched to time division as n and the switching number SWNo. 1 to SWNo. Control information corresponding to each of n is registered.

  The microcomputer 20 refers to the control table CTBL and switches the switching number SWNo. In order from 1 to the higher-order switching number, the distortion correction parameter specified in the setting or the distortion correction parameter based on the coordinates (X, Y, Z) indicating the distortion correction area is changed to the frame rate of the imaging sensor 12, In consideration of the output frame rate, the data is sequentially stored in the RAM of the RAM unit 21 in FIG.

  For example, in the configuration of FIG. 1, in the example of the timing of FIG. 6, the distortion correction parameters <1>, <2>, <3>, <4> are changed to the RAMs 21-1, 21-2, 21 of the RAM unit 21. -3, 21-4.

In the configuration of FIG. 1, in the example of the timing of FIG. 8, first, distortion correction parameters <1>, <2>, <3>, <4> are changed to the RAMs 21-1, 21-21 of the RAM unit 21. 2, 21-3, 21-4.
Next, when the generation of the distortion correction image based on the distortion correction parameter <1> is completed, the distortion correction parameter <5> is stored in the RAM 21-1 of the RAM unit 21, and then the distortion correction parameter <2>. When the generation of the distortion correction image based on the above is completed, the distortion correction parameter <6> may be sequentially stored in the RAM 21-2 of the RAM unit 21.

At this time, it is needless to say that the timing for storing the distortion correction parameter may be performed for each frame of the output or for each frame of the imaging sensor 12.
If there are a plurality of output frame intervals to be switched, the distortion correction parameters may be copied to another RAM, or the distortion correction parameter switching unit may be controlled so that the RAM is not switched by the number of intervals. .
Thereafter, the same processing is repeated until the total number of distortion correction parameters to be switched to time division, and the switching number SWNo. Return to 1 and repeat the above.

  Next, a method for inserting a control code or an image pattern into an output image performed by the output processing unit 19, for example, will be described.

  11A to 11G are diagrams showing examples of inserting control codes or image patterns.

11A to 11G show the types of distortion correction parameters or the coordinates (X, Y, Z) of the distortion correction area indicating the pan angle, tilt angle, and zoom angle of view in the image output in time division. ) And the like are converted into control codes or image patterns and inserted.
The control code or image pattern is inserted by the output processing unit 19 in FIG. 1, and the pixel address of the image whose distortion has been corrected is counted and inserted into a predetermined arbitrary pixel address (H, V).

<Control code>
FIG. 12 is a diagram illustrating a format example of a control code to be inserted when the image output is digital.
The control code CCOD in FIG. 12 includes a switching number, a type of distortion correction parameter, and coordinates (X, Y, Z) of the distortion correction area.
Thus, when the image output is digital, each pixel data may be replaced with a control code.
The control code CCOD to be inserted may be an ASCII code or binary, and the number of pixels, the bit width, the luminance signal, the color difference signal, the RGB signal, etc. for each control code are arbitrary in the pixel to be replaced. Needless to say, they can be combined.
Further, when inserting into the effective image data, an identification code with the pixel data may be added, or it may be inserted into the pixel data being blanked.

<Image pattern>
FIG. 13 is a diagram illustrating an example of converting (modulating) a control code into an image pattern when the image output is analog.
The example of FIG. 13 shows a case where FIG. 12 is converted (modulated) into an image pattern IPTN.
In FIG. 13, AM (amplitude modulation) of the luminance signal is easy to understand. However, if a controller such as a video switcher in the subsequent stage can identify (demodulate) an image pattern, FM (frequency modulation) or PM (pulse modulation) is possible. ) Or QAM, color difference signals, RGB signals or the like may be used, and it is needless to say that a synchronization pattern (preamble) may be added in some cases. It is also possible to superimpose the image pattern information on the area which is the main signal of the image signal.
Further, if a controller such as a video switcher in the subsequent stage can perform block matching or template matching, the image pattern may be a two-dimensional pattern and may be matched to the system.

  Here, the operation of the camera apparatus 10 having the above configuration will be briefly described.

Image data captured by the imaging sensor 12 through the wide-angle lens 111 of the optical system 11 is output from the camera signal processing unit 13 as an original image S13 without distortion correction in an appropriate state (AE, AWB, etc.).
The original image S13 is sequentially stored at the address specified by the write address generation unit 16 while switching the frame memories 14 and 15 to be written for each frame rate of the imaging sensor 12.
Further, the microcomputer 20 stores arbitrary distortion correction parameters in the RAMs 21-1 to 21-4 in the RAM unit 21.
The distortion correction parameter switching unit 22 switches, for example, four types of distortion correction parameters for each output frame rate according to an instruction from the microcomputer 20. It is also possible not to switch by the support of the microcomputer.
Then, the read address conversion unit 17 switches the frame memories 14 and 15 to be read for each frame rate of the imaging sensor 12 and converts (X, Y) indicating the vector of the distortion correction parameter into a real address.
The data read out from the frame memories 14 and 15 becomes a distortion-corrected image, and then the output processing unit 19 inserts a control code or a pattern of an image obtained by converting (modulating) the control code, and undergoes format conversion or the like. Output to the outside.

  FIG. 14 is a system composition showing an example applied to a surveillance camera system employing the camera device of this embodiment.

  This system 100 has a system configuration in which one camera replaces a plurality of cameras, and includes a camera device 110 (corresponding to the camera device in FIG. 1 and the like), a video switcher controller 120, a video device 130, for example, a monitor. The monitor group 140 includes 141 to 143.

As shown in FIG. 14, the camera apparatus 110 inserts control codes or image patterns into images of four areas (areas corresponding to the distortion correction parameters indicated by <1> to <4>) obtained by wide-angle imaging. And output in time division.
The controller 120 such as a video switcher has a function of detecting and identifying the control code CCOD or the image pattern IPTN in the image output from the camera device 110, and in some cases, the output image can be displayed on the monitors 141 to 143. The frame memory can display only an arbitrary image or a plurality of images can be combined and displayed.
When recording video, only an arbitrary image is output to the video device 130, or an image from the camera device 110 is output as it is (in this example, images of four areas divided in time).
Further, when reproducing from a video, the control code CCOD or the image pattern IPTN in the image is identified, and only an arbitrary image is displayed or a plurality of images are combined and displayed.

  In the example of FIG. 14, an example in which the controller 120 such as a video switcher has a function of detecting and identifying the control code CCOD or the image pattern IPRN in the image is shown. Needless to say, it has a function of detecting and identifying an image pattern, and may record only an arbitrary image or reproduce only an arbitrary image.

  As described above, according to the present embodiment, the camera apparatuses 10 and 110 have distortion correction parameters stretched in a lattice shape based on an image captured by a wide-angle lens having a point-symmetric distortion about the optical axis. (Vector) can be used for distortion correction and electronic pan-tilt, and images in multiple arbitrary areas can be switched to time-division output, and multiple arbitrary pan angles, tilt angles, zoom angles of view can be output Regardless of the frame rate of the imaging sensor 12, the output of the imaging sensor 12 is obtained by switching the time-division for each output frame rate, and correcting the distortion based on a plurality of arbitrary distortion correction parameters. Regardless of the frame rate, it is possible to switch to time-sharing for each output frame rate and output the distortion correction parameters in the output image. Information such as type, pan angle, tilt angle, zoom angle of view, etc. can be inserted by converting (modulating) the control code or control code into an image pattern. Since the imaging sensor 12 has a frame rate that is significantly high, for example, the imaging sensor can be configured to have a frame rate of 240 fps and an output frame rate of 60 fps. Obtainable.

Compared to a mechanical pan / tilt / zoom camera, the electronic pan / tilt / zoom camera can instantly switch display areas, improving the real-time performance and avoiding missing important scenes.
Compared to an electronic pan / tilt / zoom camera that synthesizes and outputs images from multiple areas, the image of multiple areas output in a time-sharing manner is distributed to individual monitors by a controller such as a video switcher, thereby reducing the resolution of each image. As a result, visibility is improved.
For example, when recording a video, if all of the images of a plurality of areas output in a time-sharing manner are recorded (as they are), an important scene will not be missed.
In addition, since recording can be performed while maintaining the resolution of each image, the resolution at the time of reproduction is improved and visibility is improved.

Furthermore, if an imaging sensor with a high frame rate is used, the output frame rate is naturally improved, so that the dynamic resolution of each image output in a time-division manner is improved and the visibility is improved.
The camera itself has a setting to switch to time division, and outputs images of multiple areas or images corrected for distortion based on distortion correction parameters in time-sharing in a preset order. There is no need to switch output images or perform control for synchronization, and as a result, the load on the host system can be reduced.
By inserting a control code or a pattern obtained by converting (modulating) a control code into an image, a controller such as a video switcher determines which area each image is captured from, or which distortion correction process from the image itself. It is possible to easily identify whether the image is an image, and display, recording / playback, and the like can be controlled based on the image.
From the above, even if there is only one camera and only one camera output, it is possible to construct a system as if multiple cameras were installed, and the number of cameras can be reduced. Can be greatly reduced.

By the way, in the present embodiment, a technique for correcting distortion using distortion correction parameters (vectors) arranged in a lattice pattern based on an image captured by a wide-angle lens using a wide-angle lens is employed.
When electronic pan / tilt is performed using this technique, it is necessary to prepare a plurality of types of distortion correction parameters for an arbitrary pan / tilt angle and angle of view in advance and store them in a storage device.

  However, in the above-described technology, in order to perform electronic pan-tilt, it is necessary to prepare a plurality of types of distortion correction parameters for arbitrary pan-tilt angles and angles of view in advance, and the capacity of the storage device for distortion correction parameters increases. However, the system cost may increase.

  In addition, when distortion correction parameters for electronic pan / tilt are downloaded from the host system to the camera system for distortion correction and electronic pan / tilt by communication, it is necessary to communicate the distortion correction parameters every time the pan / tilt angle is switched. As the amount of communication increases, the load on the processing of each system may increase.

As a countermeasure, for example, this can be eliminated by using a distortion correction parameter symmetrization technique, the capacity of the storage device for storing the distortion correction parameters can be reduced, and the system cost can be reduced. In addition, the communication amount of the distortion correction parameter can be reduced.
Hereinafter, a configuration example of the camera apparatus using the distortion correction parameter symmetrization technique will be described.

  FIG. 15 is a block diagram illustrating a configuration example of a camera apparatus that employs the distortion correction parameter symmetrization technique according to the embodiment of the present invention.

The camera device 10A of FIG. 15 differs from the camera device 10 of FIG. 1 in that the distortion correction parameter switching unit 22 includes a distortion correction parameter symmetrizing unit 221, and a microcomputer (microcomputer) 20A includes a distortion correction parameter storage memory (RAM). ) The memory address MADR for accessing the part 21 and the symmetrization setting data SSD for symmetrizing the distortion correction parameter are supplied to the distortion correction parameter symmetrizing part.
Hereinafter, the description will focus on points related to the distortion correction parameter symmetrization technique.

The camera apparatus 10A can perform distortion correction and electronic pan tilt using distortion correction parameters (vectors) stretched in a lattice pattern based on an image captured by a wide-angle lens having point-symmetric distortion about the optical axis. Thus, by utilizing the fact that the distortion of the wide-angle lens is symmetrical from the center point of the optical axis, the distortion correction parameters for an arbitrary pan or tilt angle can be shared.
The camera apparatus 10A basically sets a distortion correction parameter (vector) of an arbitrary pan / tilt angle as a first optical axis center line (vertical line) or a second optical axis center line ( By performing mirroring processing in the pan direction (left-right direction) and tilt direction (up-down direction) with the horizontal line as the axis, new distortion correction parameters are generated symmetrically in the left-right and up-down directions from the vertical line or horizontal line.
In the following description, the pan direction is described as the left-right direction and the tilt direction is described as the up-down direction.

The microcomputer 20A sets a distortion correction parameter indicating an original image portion to be corrected for distortion from now on, a memory address MADR for accessing the distortion correction parameter storage memory (RAM) unit 21, and a distortion correction parameter for symmetrization. The symmetrized setting data SSD is supplied to the distortion correction parameter symmetrizing unit 221.
The symmetrization setting data SSD "symmetrizes distortion correction vectors left and right (pan direction)", "symmetrizes vertically (tilt direction)", "symmetrizes left and right up and down (up and down left and right)", and Includes data indicating “not symmetrized”.
As described above, the microcomputer 20A may store the distortion correction parameter in a memory such as a built-in ROM / RAM in advance or may be obtained by calculation of the microcomputer 20A or transmitted by the camera device 10A. It is also possible to configure to receive by external communication from a host computer (not shown) connected by a line.

  The distortion correction parameter symmetrization unit 221 symmetrizes the memory address MADR supplied from the microcomputer 20A by the symmetrization setting data SSD that is also supplied from the microcomputer 20A. The converted address or the non-converted address converted according to (not) is given to the distortion correction parameter (vector) storage memory (RAM) 21, and the distortion correction parameter read out from the distortion correction parameter (vector) storage memory (RAM) 21 is set to a predetermined value. And a symmetrized distortion correction vector is generated and output to the read address conversion unit 17.

<Example of distortion correction parameter symmetrization>
FIG. 16 is a diagram illustrating an example in which pan / tilt is realized by correcting distortion of an image captured by a wide-angle lens in nine regions <1> to <9> and switching them.

In the example of FIG. 16, nine regions <1> to <9> are divided into a 3 × 3 grid (matrix shape), and regions <1>, <2>, <3> are arranged, areas <4>, <5>, <6> are arranged on the second line, and areas <7>, <8>, <9> are arranged on the third line.
Areas <1>, <4>, <7> are arranged in the first column, areas <2>, <5>, <8> are arranged in the second column, and areas are arranged in the third column. <3>, <6>, <9> are arranged.

The pan angle of the region <1>, region <4>, and region <7> in the first column is arbitrarily the same angle from the optical axis central vertical line OCL1 as the first optical axis center line, and the third column The pan angles of the region <3>, the region <6>, and the region <9> are the same as the minus angle of the pan angle of the region <1> from the optical axis central vertical line OCL1.
Similarly, the tilt angles of the region <1>, region <2>, and region <3> in the first row are arbitrarily the same angle from the optical axis central horizontal line OCL2, and the region <7> and region in the third row The tilt angles of <8> and the region <9> are the same as the minus angle of the tilt angle of the region <1> from the optical axis central horizontal line OCL2 as the second optical axis center line.
Region <5> is centered on the optical axis, and the pan angle and tilt angle are both 0 degrees. This region <5> is excluded from the object of symmetrization.
Hereinafter, a symmetrization method will be described based on this.

<Distortion correction parameter left / right (pan direction) coping method (first symmetrization method)>
17A and 17B are conceptual diagrams of left-right symmetrization.
For example, in order to generate the distortion correction parameter of the region <3> from the distortion correction parameter of the region <1>, the optical axis center vertical line OCL1 serving as the first optical axis center line in contact with the optical axis center is used as an axis. The vector (X, Y) at the lattice point is folded symmetrically, and the vector (X) may be multiplied by minus.
This will be described with reference to FIG. 3. The vector (X0, Y0) of the region <1> is replaced with the lattice point of the newly generated vector (X3, Y0) of the region <3>. ).
That is, the transformation of the new region <3> vector (Xn, Ym) = region <1> vector (−1 · X (3−n), Ym) is performed.
Here, n and m are examples in the case of taking a value of 0 to 3 indicating the array number of each lattice point.
Needless to say, this method can generate distortion correction parameters from the region <2> to the region <6> and from the region <7> to the region <9>.

<Distortion correction parameter up / down (tilt direction) coping method (second symmetrization method)>
18A and 18B are conceptual diagrams of symmetrization in the vertical direction.
For example, in order to generate the distortion correction parameter of the region <7> from the distortion correction parameter of the region <1>, the light as the second optical axis center line orthogonal to the first optical axis center line in contact with the optical axis center The vector (X, Y) at each lattice point is folded up and down symmetrically about the axis center horizontal line OCL2, and the vector (Y) may be multiplied by minus (−1).
This will be described with reference to FIG. 3. The vector (X0, Y0) of the region <1> is replaced with a lattice point of the vector (X0, Y3) of the newly generated region <7>, and the vector (Y0) ).
That is, the transformation of the new region <7> vector (Xn, Ym) = region <1> vector (−1 · Xn, Y (3−m)) is performed.
Here, n and m indicate the array element numbers of the respective lattice points, and are examples in the case of taking values of 0 to 3.
Needless to say, distortion correction parameters for the region <2> to the region <8> and the region <3> to the region <9> can be generated by this method.

<Distortion correction parameter up / down / left / right coping method (third symmetrization method)>
For example, in order to generate the distortion correction parameter of the region <9> from the distortion correction parameter of the region <1>, a combination of the distortion correction parameter left-right coping method and the distortion correction parameter vertical coping method described above is combined. Just do it.
This will be described with reference to FIG. 3. The vector (Xn, Ym) of the new area <9> = the vector of the area <1> (−1 · X (3 −n), −1 · Y (3 −m) ).
Here, n and m indicate the array element numbers of the respective lattice points, and are examples in the case of taking values of 0 to 3.

As described above, as described with reference to FIG. 16, when panning / tilting is realized by correcting distortion of an image captured by a wide-angle lens in nine regions <1> to <9> and switching them, By using <Distortion correction parameter left / right coping method>, <Distortion correction parameter up / down coping method>, and <Distortion correction parameter up / down / left coping method>, the result is that region <1>, region < If there are distortion correction parameters for 2>, <4>, and <5>, the remaining regions <3>, <6>, <7>, <8>, and <9> are distorted. It can be seen that the correction parameter is newly generated.
When the above-described first to third symmetrization methods are adopted, the region around the optical axis including the image with high importance, that is, the region <5> in the above example, is not the object of symmetrization. It is also possible to prevent the deterioration of the material.

The distortion correction parameter symmetrization method described above is employed in the distortion correction parameter symmetrization unit 221 of the camera apparatus 10A. However, the configuration may be realized by hardware or software.
In general, since distortion correction parameters are often stored in a memory such as RAM or ROM, the distortion correction parameter symmetrization unit 221 converts the memory address in the same manner as described above. Needless to say, the distortion correction vector (X) output from the memory may be multiplied by minus.

FIG. 19 is a diagram illustrating an example of a hardware configuration of the distortion correction parameter symmetrizing unit according to the present embodiment.
FIG. 20 is a diagram illustrating a flowchart of software that can be employed in the distortion correction parameter symmetrization unit according to the present embodiment.

  8 includes a distortion correction parameter (vector) storage memory (RAM) 2211, a conversion method selection circuit 2212, an address conversion circuit 2213, a selector 2214, and a multiplier 2215.

  The conversion method selection circuit 2212 sends the address conversion method to the address conversion circuit 2213 according to the symmetrization form (left / right, up / down, left / right / up / down, not symmetrized) indicated by the symmetrization setting data SSD supplied by the microcomputer 20A. And instructs the selector 2214 to select 1 or -1.

  The address conversion circuit 2213 converts the address of the memory address MADR supplied by the microcomputer 20A according to the symmetrization form (left / right, up / down, left / right / up / down, not symmetrized) instructed by the conversion method selection circuit 2212, or A converted address or a non-converted address obtained without converting the address is supplied to a distortion correction parameter (vector) storage memory (RAM) 2211.

  The multiplier 2215 multiplies the distortion correction parameter read from the distortion correction parameter (vector) storage memory (RAM) 2211 by 1 or −1 selected by the selector 2214 to generate a symmetric distortion correction vector. And output to the read address conversion unit 17.

In the software processing shown in FIG. 20, the distortion correction vector (Xi, Yj) to be symmetrized is indicated by Xorg (i, j) and Yorg (i, j), and a newly generated distortion correction vector (Xi, Yj) Are denoted by Xnew (i, j) and Ynew (i, j).
In FIG. 20, variables i and j indicate the array number of the lattice.

  The software processing shown in FIG. 20 is configured to repeat the processing from step ST1 to step ST8 until variables (array numbers) i and j become 3.

In step ST3, the type of symmetrization process is determined. If it is determined in step ST3 that the process is symmetrized, the process proceeds to step ST4, where the symmetrized process is performed.
In this case, as described in the above <Distortion correction parameter left-right coping method>, the distortion correction vector Xorg (3 i, j) to be symmetrized is multiplied by minus (−1). The distortion correction vector Yorg (3 i, j) cannot be multiplied by −1 (if it is associated with the configuration of FIG. 19, it is multiplied by 1).

When it is determined in step ST3 that the vertical symmetrization process is performed, the process proceeds to step ST5, where the vertical symmetrization process is performed.
In this case, as described in the above <Distortion Method for Distortion Correction Parameters>, the distortion correction vector Yorg (3 i, j) to be symmetrized is multiplied by minus (−1). The distortion correction vector Xorg (3 i, j) cannot be multiplied by −1 (1 if it is associated with the configuration of FIG. 19).

If it is determined in step ST3 that the left / right symmetrization process is performed, the process proceeds to step ST6, where the vertical symmetrization process is performed.
In this case, as described in the above <Distortion Correction Parameter Left / Right Up / Down Handling Method>, the distortion correction vectors Xorg (3 i, j) and Yorg (3 i, j) to be symmetrized are multiplied by minus (−1). It is done.

  As described above, according to the camera apparatus of FIG. 15, the memory address MADR supplied by the microcomputer 20A is symmetrized (left and right, up and down) indicated by the symmetrization setting data SSD also supplied by the microcomputer 20A. Distortion read out from the distortion correction parameter (vector) storage memory (RAM) by giving the conversion address or non-conversion address converted according to the left / right / up / down, not symmetrized) to the distortion correction parameter (vector) storage memory (RAM) 1 includes the distortion correction parameter symmetrization unit 221 that generates a symmetrized distortion correction vector by giving a predetermined coefficient or sign to the correction parameter and outputs the symmetric distortion correction vector to the read address conversion unit 17. In addition to the effects of the camera device 10, the following effects can be obtained. Can.

1) The capacity of a storage device for storing distortion correction parameters can be reduced, and system cost can be reduced.
For example, the number of distortion correction parameters for pan-tilting the above-mentioned wide-angle captured image in nine regions is changed from nine to four, for example, the number of distortion correction parameters for pan-tilting a wide-angle captured image in 25 regions. Can be reduced from 25 to 5.

2) When the distortion correction parameter is downloaded from the host system by communication, it is not necessary to perform communication if sharing is possible.

3) Since distortion correction parameters can be shared by simple software or hardware processing, the load on processing of each system can be reduced, and in some cases, real-time performance can be improved (high frame rate can be easily increased). Visibility is improved.

4) A wide range of applications such as rotated distortion correction parameters, mirror image processed distortion correction parameters, and composite processed distortion correction parameters as well as distortion correction parameters for an arbitrary pan / tilt are possible. The effects of 2) and 3) can be further expected.

  Note that the symmetrization method shown in the present embodiment is based on distortion correction parameters (strained) formed (latched) in a lattice shape based on an image captured by a wide-angle lens having a point-symmetric distortion about the optical axis. If the system performs distortion correction using (vector), it can be applied not only to distortion correction parameters for an arbitrary pan / tilt, but also to rotated distortion correction parameters, distortion correction parameters subjected to mirror image processing, and distortion correction parameters subjected to synthesis processing. Needless to say.

  In the above, an example has been described in which nine regions <1> to <9> are divided and symmetrized in a 3 × 3 grid (matrix shape), but the present invention further increases the number of regions. Alternatively, it is possible to target a small area.

Note that the method described above in detail can be formed as a program according to the above-described procedure and executed by a computer such as a CPU.
Further, such a program can be configured to be accessed by a recording medium such as a semiconductor memory, a magnetic disk, an optical disk, a floppy (registered trademark) disk, or the like, and to execute the program by a computer in which the recording medium is set.

It is a block diagram which shows the structural example of the camera apparatus which employ | adopted the image processing apparatus which concerns on embodiment of this invention. It is a figure which shows the concept of the imaging using a wide angle lens. It is a figure which shows an example of the lattice arrangement | sequence of a distortion correction vector. It is a figure for demonstrating the example of distortion correction. It is a conceptual diagram in case the distortion correction parameter corresponding to four areas is used. It is a figure which shows the timing which correct | amends distortion of 4 area | regions of FIG. 5, and outputs in a time division. It is a conceptual diagram in the case of using the distortion correction parameter corresponding to six areas. It is a figure which shows the timing which correct | amends distortion of 6 area | regions of FIG. 7, and outputs to a time division. It is a figure which shows an example of the timing in the case of switching a distortion correction parameter by the frame space | interval of an imaging sensor, and time-division output. It is a figure which shows an example of the control table which concerns on this embodiment. It is a figure which shows the example of insertion of a control code or an image pattern. It is a figure which shows the example of a format of the control code inserted when image output is digital. It is a figure which shows the example which converts (modulates) a control code into an image pattern, when an image output is analog. It is a system composition which shows an example applied to the surveillance camera system which employ | adopted the camera apparatus of this embodiment. It is a system composition which shows an example applied to the surveillance camera system which employ | adopted the camera apparatus of this embodiment. It is a conceptual diagram which pan-tilt nine area | regions. It is a conceptual diagram of left-right symmetrization. It is a conceptual diagram of vertical symmetry. It is a figure which shows an example of the hardware structure of the distortion correction parameter symmetrization part which concerns on this embodiment. It is a figure which shows the flowchart of the software which can be employ | adopted as the distortion correction parameter symmetrization part which concerns on this embodiment.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 10 ... Camera apparatus, 11 ... Optical system, 111 ... Wide-angle lens, 12 ... Image sensor (imaging sensor), 13 ... Camera signal processing part, 14 ... 1st frame memory ( FRM0), 15 ... second frame memory (FRM1), 16 ... write address generation unit, 17 ... read address conversion unit, 18 ... switch, 19 ... output processing unit, 20 ... Microcomputer (microcomputer), 21... Distortion correction parameter (vector) storage memory (RAM), 22... Distortion correction parameter switching unit, 221.

Claims (17)

An image processing apparatus capable of correcting distortion using a distortion correction parameter formed in a lattice shape on an original image captured by a wide-angle lens,
An image processing apparatus having a processing unit that performs distortion correction and outputs a plurality of arbitrary images by switching to time division. The processing unit
The original image captured through the image sensor at multiple arbitrary pan angles, tilt angles, and / or zoom angles is output in a time-sharing manner for each output frame rate regardless of the image sensor frame rate. The image processing apparatus according to claim 1. The processing unit
The image according to claim 1, wherein an image that has been subjected to distortion correction based on a plurality of arbitrary distortion correction parameters can be switched to time division for each output frame rate regardless of the frame rate of the image sensor. Processing equipment. The processing unit
The image processing apparatus according to claim 2, wherein an output frame rate is lower than a frame rate of the image sensor. The processing unit
The image processing apparatus according to claim 2, wherein the image processing apparatus has a function of performing output control of divided images based on setting information. The processing unit
The image processing apparatus according to claim 2, wherein the control code or the control code can be converted into an image pattern and inserted into the output image. The processing unit
The image processing apparatus according to claim 2, wherein the control code or the control code can be converted into an image pattern and inserted into the output image. The image processing apparatus according to claim 7, wherein the control code includes at least one of a distortion correction parameter type, a pan angle, a tilt angle, and a zoom angle of view. A camera device having at least one of an electronic pan, tilt, and zoom function,
An image sensor;
An optical system including a wide-angle lens and forming a subject image on the imaging device;
An image processing apparatus capable of correcting distortion using a distortion correction parameter formed in a lattice shape on an original image captured by a wide-angle lens by the imaging element,
The image processing apparatus includes:
A camera device including a processing unit that performs distortion correction and outputs a plurality of arbitrary images by switching to time division. The processing unit
The original image captured through the image sensor at multiple arbitrary pan angles, tilt angles, and / or zoom angles is output in a time-sharing manner for each output frame rate regardless of the image sensor frame rate. The camera device according to claim 9. The processing unit
10. The camera according to claim 9, wherein an image obtained by correcting distortion based on a plurality of arbitrary distortion correction parameters can be output by switching to time division for each output frame rate regardless of the frame rate of the image sensor. apparatus. The processing unit
The camera device according to claim 10, wherein an output frame rate is lower than a frame rate of the image sensor. The processing unit
The camera device according to claim 10, wherein the camera device has a function of performing output control of divided images based on setting information. The processing unit
The camera device according to claim 10 or 11, wherein a control code or a control code can be converted into an image pattern and inserted into an output image. The processing unit
The camera device according to claim 10 or 11, wherein a control code or a control code can be converted into an image pattern and inserted into an output image. The camera device according to claim 15, wherein the control code includes at least one of a distortion correction parameter type, a pan angle, a tilt angle, and a zoom angle of view. A camera device having at least one of electronic pan, tilt and zoom functions;
A controller that performs predetermined processing on the processed image transferred from the camera device,
The image processing apparatus includes:
An image sensor;
An optical system including a wide-angle lens and forming a subject image on the imaging device;
An image processing apparatus capable of correcting distortion using a distortion correction parameter formed in a lattice shape on an original image captured by a wide-angle lens by the imaging element,
The image processing apparatus includes:
In addition to performing distortion correction, the image processing apparatus includes a processing unit that switches a plurality of arbitrary images to time division, converts a control code or a control code into an image pattern, inserts and outputs the image, and the controller includes:
A camera system that detects a control code or a pattern of an image obtained by converting a control code from a received image and performs predetermined processing based on the information.

Images