JP7038935B2 - Image processing device, image processing method, and image processing program - Google Patents

Description

The present disclosure relates to an image processing apparatus, an image processing method, and an image processing program.

There is a technique of converting a viewpoint of a plurality of captured images (that is, camera images) taken by a plurality of cameras to generate a plurality of bird's-eye view images, and combining these to generate a single bird's-eye view composite image. Normally, in the viewpoint conversion, an object located higher than the ground is converted into a bird's-eye view image in a stretched state (that is, a state projected on the ground as a projection surface) so as to fall on the ground. Therefore, the object in the bird's-eye view image is distorted. Further, when the bird's-eye view composite image is generated, the images are taken so as to partially overlap the adjacent camera images. Generally, in order to seamlessly connect images in an area where adjacent camera images overlap (hereinafter, also referred to as "superimposed area"), an alpha (α) value, which is a transparency set for each pixel, is used in the superimposed area. Image is created. However, when constructing a system for creating a bird's-eye view composite image, it is necessary to manually set the projection range of each camera image and the blend area which is a superposed area of each camera image, and these operations are troublesome and cumbersome. , Was a cause of increasing the introduction cost of the system.

Therefore, in Patent Document 1, when a tall subject exists in the overlapping region of the two camera images and the correlation between the feature amounts of the subjects in the two camera images is high, the blending ratio of the subjects is changed. We are proposing a technology to generate a bird's-eye view composite image in which the image in the superimposed region is difficult to see as a double image.

JP-A-2015-192198

However, in the technique described in Patent Document 1, when the subject projected on the projection surface looks distorted according to the distance from the camera to the subject, the visibility of the bird's-eye view composite image is lowered.

The present disclosure has been made to solve the above problems, and an object of the present disclosure is to generate a highly visible composite image.

The image processing device according to the present disclosure is a device that generates a composite image based on a plurality of captured images taken by a plurality of image pickup devices, and is based on the camera parameters of the plurality of image pickup devices and the plurality of image pickup devices. A permissible threshold indicating the permissible value of the elongation rate of the subject in a plurality of projected images obtained by projecting the photographed subject onto a predetermined projection surface, and adjacent projected images among the plurality of projected images. The projection range of the plurality of projected images and the plurality of projections are performed by using the camera parameter input unit for acquiring the difference threshold indicating the allowable value of the difference in the elongation rate, the camera parameter, the allowable threshold, and the difference threshold. The plurality of image pickup devices corresponding to each pixel of the composite image in the blend region and each pixel of the composite image in the blend region by determining a blend region in which adjacent projected images in the image overlap each other. The composite image is created by performing a composite process of connecting the adjacent projected images using the reference table and the reference table creation unit that creates a reference table showing the relationship between the corresponding pixels, which are the respective pixels of the above. It has a synthesis processing unit and.

The image processing method according to the present disclosure is a method executed by an image processing device that generates a composite image based on a plurality of captured images taken by a plurality of image pickup devices, and is a method executed by a camera parameter of the plurality of image pickup devices. , A permissible threshold indicating the permissible value of the elongation rate of the subject in the plurality of projected images obtained by projecting the subject photographed by the plurality of image pickup devices onto a predetermined projection surface, and the plurality of projected images. The step of acquiring the difference threshold indicating the permissible value of the difference of the elongation rate in the adjacent projected images in the image, and the projection range of the plurality of projected images using the camera parameter, the permissible threshold, and the difference threshold. The blend region, which is a portion where adjacent projected images overlap in the plurality of projected images, is determined, and the pixels of the composite image in the blend region and the pixels of the composite image in the blend region correspond to each other. The composite image is created by performing a step of creating a reference table showing the relationship between the corresponding pixels, which are pixels of each of the plurality of image pickup devices, and a composite process of connecting the adjacent projected images using the reference table. And have a step to do.

According to the image processing apparatus, the image processing method, and the image processing program according to the present disclosure, it is possible to generate a composite image with high visibility.

It is a figure which shows the hardware configuration of the image processing apparatus which concerns on embodiment. It is a functional block diagram which shows the structure of the image processing apparatus which concerns on embodiment. It is a figure which shows the composite image and a plurality of projection images which are created. It is a figure which shows the corresponding pixel of the projection image corresponding to the pixel of the created composite image. It is a figure which shows the elongation rate of a subject at the position where a subject exists. It is a flowchart which shows the creation process of the reference table which is executed by the reference table creation part of the image processing apparatus which concerns on embodiment. It is a flowchart which shows the creation process of the composite image which is executed by the image acquisition part of the image processing apparatus which concerns on embodiment.

Hereinafter, the image processing apparatus, the image processing method, and the image processing program according to the embodiment will be described with reference to the drawings. The following embodiments are merely examples, and it is possible to appropriately combine the embodiments and change the embodiments as appropriate.

<Configuration of image processing device 10>
FIG. 1 is a diagram showing a hardware configuration of the image processing apparatus 10 according to the embodiment. The image processing device 10 generates a composite image based on a plurality of captured images (also referred to as "camera images") taken by a plurality of image pickup devices (generally, a plurality of cameras). FIG. 1 shows an example in which a plurality of cameras include two cameras Cam1 and Cam2. However, the number of cameras may be three or more. Further, the image processing device 10 and the plurality of cameras Cam1, Cam2, ... Consists of an image composition system. Further, the image composition system may have a display device 20 such as a liquid crystal display for displaying a composite image.

As shown in FIG. 1, the image processing device 10 includes a processor 11 as an information processing unit that performs various arithmetic processes and various hardware control processes, a memory 12 that is a main storage device, and an HDD (hard disk drive). ) Or a storage device 13 which is an auxiliary storage device such as an SSD (solid state drive). Further, the image processing device 10 receives a plurality of camera images I1, I2, ... Transmitted from each of the plurality of cameras Cam1, Cam2, ..., Converts them into image data, and captures them inside the image processing device 10. It has an input interface 14 and an output interface 15 that outputs a composite image (that is, composite image data) generated by the image processing device 10 to the display device 20.

The image processing device 10 is composed of, for example, a processing circuit. The image processing device 10 may be configured by a computer including a processor 11 that executes an image processing program that is a software program. Further, the image processing program is installed in the image processing apparatus 10 from a computer-readable information recording medium for recording information or by downloading via a network.

The plurality of cameras Cam1, Cam2, ... Are image pickup devices having a function of capturing an image. The plurality of cameras Cam1, Cam2, ... Is an image pickup device having an image pickup device such as a CCD (Chaged-Coupled Devices) or CMOS (Complementary Metal-Oxide-Semiconductor) and a lens. The plurality of cameras Cam1, Cam2, ... Are, for example, surveillance cameras. At this time, the image composition system shown in FIG. 1 is a monitoring system.

It is desirable that the plurality of cameras Cam1, Cam2, ... Are the same type of equipment. The plurality of cameras Cam1, Cam2, ... Are connected to the input interface 14 of the image processing device 10 by wire, but may be connected wirelessly. Further, the connection method between the plurality of cameras Cam1 and Cam2 and the input interface 14 is not limited to a specific method. The connection method is, for example, a connection method using an IP (Internet Protocol) network or a connection method using a coaxial cable. It is desirable that the input interface 14 has a function of simultaneously receiving a plurality of camera images I1, I2, ... That is, a plurality of image signals transmitted from the plurality of cameras Cam1, Cam2, ....

The storage device 13 or the memory 12 stores an image processing program which is a software program for causing the image processing device 10 to execute the image processing method according to the present embodiment. By executing this image processing program, the processor 11 can execute the image processing method according to the embodiment.

The image data captured in the image processing device 10 is stored in the memory 12. The image data captured in the image processing device 10 is combined after being subjected to viewpoint conversion (that is, coordinate conversion), which is a process of converting an image having a viewpoint above the image pickup target area into a bird's-eye view image. The composite image, which is an image synthesized by the image processing device 10, is transmitted to the display device 20 via the output interface 15. Further, the composite image obtained by synthesizing the bird's-eye view images is also referred to as a "bird's-eye view composite image". Further, when the image processing device 10 is used in the monitoring system, the image pickup target area is also referred to as a “monitoring target area”.

FIG. 2 is a functional block diagram showing the configuration of the image processing device 10 according to the present embodiment.
FIG. 3 is a diagram showing a composite image created and a plurality of camera images taken by a plurality of cameras (for example, Cam1 to Cam4).

As shown in FIG. 2, the image processing device 10 includes an image acquisition unit 101, a camera parameter input unit 102, a reference table creation unit 103, a composition processing unit 104, and a display control unit 105. ..

The image acquisition unit 101 receives a plurality of camera images I1, I2, ... Which are captured images taken by the plurality of cameras Cam1, Cam2, ....

The camera parameter input unit 102 projects a plurality of projected images obtained by projecting a subject photographed by a plurality of cameras Cam1, Cam2, ... Camera parameters, a plurality of cameras Cam1, Cam2, ... On a predetermined projection surface. The permissible threshold _kth indicating the permissible value of the elongation rate k of the subject in (for example, P1, P2, ... Shown in FIG. 3), and the adjacent projected images among the plurality of projected images (for example, shown in FIG. 3). The difference threshold Sub _th indicating the permissible value of the difference Sub of the values k ₁ and k ₂ of the elongation rate k in P1 and P2) is acquired. The camera parameters include, for example, internal parameters and external parameters of each camera Cam1, Cam2, ..., Lens distortion correction data of each camera Cam1, Cam2, ..., And the like. The elongation rate k of the subject of each camera Cam1 and Cam2 at each point in the image imaging target region (that is, in the projected image) is also referred to as "k value". The camera parameter input unit 102 may _acquire the camera parameter, the allowable threshold value kth of the elongation rate k, and the difference threshold value _Subth from the storage device 13 shown in FIG. 1 or from the outside of the image processing device 10. The elongation rate k will be described in detail with reference to FIG. 5 described later.

The reference table creation unit 103 projects a plurality of projected images (P1, P2, ... In FIG. 3) using the camera parameters, the allowable threshold value _kth , and the difference threshold value _Subth acquired by the camera parameter input unit 102. A range and a superposed region (also referred to as a "blended region") in which adjacent projected images (P1 and P2 in FIG. 3) overlap each other are determined, and each pixel of the composite image in the superposed region is determined. , Create a reference table showing the relationship with the corresponding pixels, which are the respective pixels of the plurality of image pickup devices Cam1, Cam2, ... Corresponding to each pixel of the composite image in the superimposed region. In other words, the reference table creating unit 103 creates a reference table for each camera Cam1 and Cam2 based on the camera parameters acquired by the camera parameter input unit 102 and the projection surface such as the ground. That is, the reference table creation unit 103 creates the number of reference tables equal to the number of cameras. The created reference table is stored in the memory 12, for example. Each reference table is composed of α values of each of the pixels having W _syn in the horizontal direction and H _syn in the vertical direction. The pixels of horizontal W _syn and vertical H _syn correspond to the pixels of the composite image.

The compositing processing unit 104 creates a composite image of an arbitrary viewpoint, that is, an arbitrary viewpoint image, using a reference table for each of the cameras Cam1 and Cam2. When the arbitrary viewpoint is above the subject, the projected image is a bird's-eye view image, and the arbitrary viewpoint image is a bird's-eye view composite image.

The display control unit 105 transmits an image signal based on the arbitrary viewpoint image created by the synthesis processing unit 104 to the display device 20. The display device 20 displays an image based on the received image signal.

FIG. 4 is a diagram showing the corresponding pixels of the camera image corresponding to the pixels of the created composite image. The reference table created by the reference table creation unit 103 corresponds to the corresponding pixels of each camera Cam1 and _Cam2 corresponding to the horizontal W- _syn pixels (x- _syn , y- _syn ) of the created composite image. It has x _{cam1, y cam1), (x cam2, y cam2 ) and an} α value of the corresponding pixel (that is, a weighted coefficient used for the blending process). In other words, the reference table includes the corresponding pixels (x _{cam1, y cam1) of the camera Cam1 and their α values, and the corresponding pixels (x cam2 , y cam2 ) of} the camera Cam2 for each pixel (x _syn , y _syn ). It has an α value.

When the cameras Cam1 and Cam2 are fixed surveillance cameras installed on the outer wall of a building or the like, the reference table may be created in advance and stored in a storage device. Further, when the cameras Cam1 and Cam2 are surveillance cameras installed on a moving object such as an unmanned aerial vehicle, the reference table may be created for each frame of the image taken by each of the cameras Cam1 and Cam2.

When the reference table creation unit 103 does not have the corresponding pixels (x cam1, y _{cam1) and (x cam2, y cam2 ) (} that is, the corresponding two pixels) of the cameras Cam1 and Cam2 corresponding to the pixels on the composite _image . Sets a value indicating that the corresponding pixel does not exist in the reference table. The reference table creation unit 103 determines the correspondence between the pixels of each camera Cam1 and Cam2 and each pixel of the composite image from the coordinates on the defined projection plane (that is, the predetermined projection plane) to the coordinates before the projection conversion process and the viewpoint. It is calculated by back-calculating the coordinates before the conversion process and the coordinates before the lens distortion correction process. By using the reference table, the compositing processing unit 104 simply receives the pixel values of the corresponding pixels of the cameras Cam1 and Cam2 corresponding to each pixel of the compositing image, and the camera images I1 and I2 of the plurality of cameras Cam1 and Cam2. It is possible to create a composite image based on.

FIG. 5 is a diagram showing the elongation rate k of the subject of each of the cameras Cam1 and Cam2 at the position of the subject (x, y, z). The camera parameter input unit 102 acquires the elongation rate k of the subject of each camera Cam1 and Cam2 at each point of the shooting target area on the projection surface.

(That is, each pixel)
In FIG. 5, the subject is a person. The elongation rate k of the camera Cam1 at the point (x, y, z) is k _{(x, y, z) Cam1} , and the elongation rate k of the camera Cam2 at the point (x, y, z) is k _{(x, y} ). _{, Z) Cam2} is represented by the following equations (1) and (2).

Here, Length (AC) indicates the distance between points A and C, that is, the length of the subject in the projected image on the projection plane. Further, the Length (BD) indicates the distance between the points B and the point D, that is, the length of the subject in the projected image on the projection plane. Further, H indicates the height of the actual subject. Note that k _{(x, y, z) Cam1} is also referred to as k1, and _{k ( x, y, z) Cam2} is also referred to as k2. The elongation rate k is calculated for each pixel in the projected image for each camera. Further, the elongation rate k becomes larger as the distance from the center of the image, which is the center of the image captured by the camera, increases, and the distortion of the image also increases.

By selecting a camera having a small elongation rate k from among the plurality of cameras, it is possible to reduce the elongation of the subject on the composite image, and as a result, it is possible to suppress the deterioration of the image quality.

The reference table creating unit 103 automatically uses the elongation rate k calculated for each pixel of each camera Cam1 and Cam2 in the equations (1) and (2) to automatically set the projection range of the cameras Cam1 and Cam2 and the camera Cam1. The α value in the superimposed region of Cam2 is determined.

<Operation of image processing device 10>
Next, the operation of the image processing device 10 according to the present embodiment will be described. Here, the operation when the cameras are two cameras Cam1 and Cam2 will be described. The processing performed by the image processing apparatus 10 includes acquisition of camera parameters and elongation rate k, creation of a reference table, and composition of camera images.

The camera parameter input unit 102 is an internal parameter (that is, a camera-specific parameter) including information such as the focal distance and image center of each camera Cam1 and Cam2, and an external parameter (that is, a camera) including position / orientation information of each camera Cam1 and Cam2. Position / orientation parameter), a distortion correction table that corrects lens distortion in camera images taken by each camera Cam1 and Cam2, and a subject of each camera Cam1 and Cam2 at each position (x, y, z) of a subject (for example, a person). Difference between the allowable thresholds _kth of the elongation rates k of k ₁ and k ₂ and the elongation rate k ₁ of the subject in the pixels of the camera Cam 1 and the elongation k of the subject k 2 in the pixels of the camera Cam ₂ Sub. Various parameters such as a difference threshold _Subth indicating an allowable value of (= | k ₁ − k ₂ |) are acquired. Here, an example is given in which the camera parameter input unit 102 reads various parameters from a setting file or the like on the memory 12 (FIG. 1), but various parameter inputs are not limited to reading from the setting file. It may be provided by an external device capable of communicating.

FIG. 6 is a flowchart showing a reference table creation process executed by the reference table creation unit 103 of the image processing apparatus 10. First, the reference table creation unit 103 is a reference table showing the α values of the corresponding pixels (x _cam , y _cam ) of the cameras Cam1 and Cam2 corresponding to each pixel (x _syn , y _syn ) of the created composite image. Is created (step S10). For example, when the plurality of cameras are two cameras Cam1 and Cam2, the reference table creation unit 103 uses the corresponding pixels (x _cam1 , y _cam1 ) of the camera Cam1 for each pixel (x _syn , y _syn ) of the composite image. The α value of, and the α value of the corresponding pixel (x _cam2 , y _cam2 ) of the camera Cam2 are determined.

The reference table creating unit 103 repeats the following processing (steps S11 to S14) for each pixel (x _{sin, y syn )} of the composite image. First, the reference table creation unit 103 photographs the corresponding pixels used as the pixels of the composite image by using the reference table showing the α values of the corresponding pixels of the cameras Cam1 and Cam2 corresponding to the pixels of the composite image1. Select (that is, extract) more than one camera (step S11).

Next, when the camera extracted in step S11 is one camera (YES in step S12), the reference table creating unit 103 describes the subject in each pixel of the projected image of the extracted one camera. It is determined whether or not the elongation rate k is larger than the allowable threshold value _kth , and if the elongation rate k of the subject is larger than the allowable threshold value _kth (when k> _kth ), the α value is set to 0 and the corresponding camera. When the elongation rate k of the subject in the pixel is equal to or less than the allowable threshold value _kth (when k ≦ _kth ), the α value is set to 1.

When the reference table creating unit 103 has two or more cameras extracted in step S11 (NO in step S12), the reference table creating unit 103 includes _a camera having the lowest value k1 of the elongation rate k of the subject and a subject. A camera having the second lowest elongation rate k ₂ is extracted (step S13). The camera having the lowest value k ₁ of the subject elongation rate k is referred to as "camera A1", and the camera having the second lowest value k ₂ of the subject elongation rate k is referred to as "camera A2".

The reference table creating unit 103 is in the case where the difference Sub of the values k ₁ and k ₂ of the elongation rate k for each pixel of the subjects of the two cameras extracted in step S13 is larger than the predetermined difference threshold Sub _th (Sub>. (When Sub _th ), the α value is set to 0, and when the difference Sub of the elongation rates k ₁ and k ₂ is equal to or less than the difference threshold Sub _th (when Sub ≦ Sub _th ), the α value is set to 1. Through these processes, the reference table creation unit 103 determines the blend region, which is the superimposed region in the composite image.

Next, the reference table creation unit 103 applies an alpha blending process to the determined blend region to create a reference table for α values (step S14). In the alpha blending process, when synthesizing two images, the pixel value of the composite image is calculated by the weighted average of the pixel values of the two original images.

FIG. 7 is a flowchart showing a composite image creation process executed by the image acquisition unit 101 of the image processing device 10.

The image acquisition unit 101 decodes the camera image and acquires RAW image data. The RAW image data is image data in which the image is not compressed. For example, from cameras Cam1 and Cam2 to H.I. When the image compressed and encoded in the 264 format is streamed, the image acquisition unit 101 may use the H.A. By performing decoding corresponding to 264, RGBA 32-bit RAW image data is acquired. RGBA is image data having α value information in addition to the three colors of Red, Green, and Blue. Although RGBA is taken as an example here, the format of the image data is not limited to RGBA.

The compositing processing unit 104 performs distortion correction and bird's-eye view conversion on the camera image (step S20), performs projection projection using the α value of the reference table created by the reference table creating unit 103, and synthesizes the bird's-eye view as a composite image. Create an image (step S21). The display control unit 105 transmits the composite image created by the composite processing unit 104 to the display device 20 (step S22).

<Effect of image processing device 10>
As described above, the parameters set by the image processing apparatus and the image processing method of the present embodiment can be set by setting the allowable threshold value _kth and the difference threshold value _Subth of the elongation rate regardless of the number of cameras. It is possible to automatically create a seamless composite image.

Further, by determining the projection area and the superimposition area (that is, the blend area) according to the elongation rate k of the subject on the composite image, the visibility of the observer is ensured and it is necessary when installing the image composition system. Work can be reduced.

10 image processing device, 20 display device, 101 image acquisition unit, 102 camera parameter input unit, 103 reference table creation unit, 104 composition processing unit, 105 display control unit, Cam1, Cam2 camera, I1, I2 camera image.

Claims (7)

An image processing device that generates a composite image based on a plurality of captured images taken by a plurality of image pickup devices.
The camera parameters of the plurality of image pickup devices, and the permissible threshold value indicating the permissible value of the elongation rate of the subject in the plurality of projected images obtained by projecting the subject photographed by the plurality of image pickup devices onto a predetermined projection surface. , And a camera parameter input unit that acquires a difference threshold indicating the allowable value of the difference in the elongation rate in the adjacent projected images among the plurality of projected images.
Using the camera parameters, the permissible threshold, and the difference threshold, the projection range of the plurality of projected images and the blended region, which is a portion of the plurality of projected images where adjacent projected images overlap, are determined. Creation of a reference table for creating a reference table showing the relationship between each pixel of the composite image in the blend region and the corresponding pixel which is each pixel of the plurality of image pickup devices corresponding to each pixel of the composite image in the blend region. Department and
A compositing processing unit that creates the compositing image by performing a compositing process of connecting the adjacent projection images using the reference table, and a compositing processing unit.
Image processing device with. The image processing device according to claim 1, wherein the camera parameter includes an internal parameter and an external parameter of each of the plurality of image pickup devices. The image processing apparatus according to claim 1 or 2, wherein the elongation rate is a value obtained by dividing the length of the subject in each of the plurality of projected images by the height of the actual subject. The reference table creation unit
From the plurality of image pickup devices, an image pickup device that has captured the captured image projected as pixels of the composite image is selected.
When the selected captured image is an captured image captured by one imaging device, it is determined whether or not the elongation rate of the subject in each pixel of the projected image is larger than the allowable threshold value. ,
The image according to any one of claims 1 to 3, wherein the α value is 0 when the elongation rate is larger than the allowable threshold value, and the α value is 1 when the elongation rate is equal to or less than the allowable threshold value. Processing equipment. The reference table creation unit
From the plurality of image pickup devices, an image pickup device that has captured the captured image projected as pixels of the composite image is selected.
When the selected captured images are two or more captured images, the image pickup device having the lowest value among the elongation rates of the subject in each pixel of the projected image and the elongation rate of the subject. Extracts the imager with the second lowest value,
It is determined whether or not the difference between the lowest value and the second lowest value is larger than the difference threshold value.
The image processing apparatus according to any one of claims 1 to 4, wherein the α value is 0 when the difference is larger than the difference threshold value, and the α value is 1 when the difference is equal to or less than the difference threshold value. .. An image processing method executed by an image processing device that generates a composite image based on a plurality of captured images taken by a plurality of image pickup devices.
A permissible threshold indicating the camera parameters of the plurality of image pickup devices and the permissible value of the elongation rate of the subject in the plurality of projected images obtained by projecting the subject photographed by the plurality of image pickup devices onto a predetermined projection surface. , And the step of acquiring a difference threshold indicating the permissible value of the difference in the elongation rate in the adjacent projection images among the plurality of projection images.
Using the camera parameters, the permissible threshold, and the difference threshold, the projection range of the plurality of projected images and the blended region, which is a portion of the plurality of projected images where adjacent projected images overlap, are determined. A step of creating a reference table showing a relationship between each pixel of the composite image in the blend region and a corresponding pixel which is a pixel of each of the plurality of image pickup devices corresponding to each pixel of the composite image in the blend region.
A step of creating the composite image by performing a composite process of connecting the adjacent projection images using the reference table, and
Image processing method having. An image processing program executed by a computer that generates a composite image based on a plurality of captured images taken by a plurality of imaging devices.
A permissible threshold indicating the camera parameters of the plurality of image pickup devices and the permissible value of the elongation rate of the subject in the plurality of projected images obtained by projecting the subject photographed by the plurality of image pickup devices onto a predetermined projection surface. , And the step of acquiring a difference threshold indicating the permissible value of the difference in the elongation rate in the adjacent projection images among the plurality of projection images.
Using the camera parameters, the permissible threshold, and the difference threshold, the projection range of the plurality of projected images and the blended region, which is a portion of the plurality of projected images where adjacent projected images overlap, are determined. A step of creating a reference table showing a relationship between each pixel of the composite image in the blend region and a corresponding pixel which is a pixel of each of the plurality of image pickup devices corresponding to each pixel of the composite image in the blend region.
A step of creating the composite image by performing a composite process of connecting the adjacent projection images using the reference table, and
An image processing program that lets your computer run.

Images