JPWO2018105097A1 - Image synthesizing device, image synthesizing method, and image synthesizing program - Google Patents

Abstract

Non-uniform luminance due to a difference in luminance between captured images at joints of composite images is reduced, and a composite image that is easy to recognize is provided. The image synthesizing device 100 includes an image processing unit 200, a storage device 210, an input interface 220, an output interface 230, and a reference area setting unit 240. The image processing unit 200 includes a distortion correction unit 110, a luminance adjustment unit 120, and an image combining unit 130. In the image combining device 100, the reference area setting unit 240 sets the reference area to a part of the plurality of photographed images, and the luminance adjustment unit 120 performs the plurality of photographing so that the luminance of the reference area does not exceed a predetermined threshold. Adjust the brightness between images.

Description

  The present invention relates to an image combining apparatus, an image combining method, and an image combining program for combining a plurality of captured images acquired by capturing by a plurality of capturing devices to generate one continuous image.

  BACKGROUND Conventionally, there is a technique of generating a plurality of photographed images by a plurality of cameras attached around an object (for example, a vehicle) and connecting (combining) the generated plurality of photographed images to generate a panoramic image. At this time, when the brightness (illuminance) of the subject changes depending on the shooting time and the shooting location, the brightness adjustment of the shot image uses the AGC (Automatic Gain Control) function of the camera to set the brightness of each shot image of the camera. It is common to adjust the

  However, if the camera orientation or mounting direction is different, the luminance of each photographed image will vary, so if only a plurality of photographed images are synthesized as they are, the seams between photographed images will be due to the luminance difference between the photographed images. There is a problem that the continuity of the composite image is impaired and the appearance as a single picture becomes unnatural. Therefore, Patent Document 1 selects an image having the highest average brightness in the screen from among a plurality of photographed images as a reference image, and adjusts the brightness of the entire composite image to match the brightness of the reference image. It is disclosed.

JP, 2011-160019, A

  However, in the technique of Patent Document 1, since the image with the highest average luminance among the plurality of photographed images is used as a reference, the luminance adjustment is performed so as to increase the average luminance of the other photographed images. At this time, a low-brightness portion (dark portion) and a high-brightness portion (bright portion) are mixed in the photographed image to be subjected to brightness adjustment, and if the brightness adjustment of the entire screen is performed, the brightness of the original bright portion is adjusted. Is high, the brightness of the bright part becomes higher than that of the original image, and there is a problem that the brightness becomes saturated (so-called overexposure).

  In addition, in the case where the composite image is used for monitoring a moving object such as a vehicle, and it is known in advance that the moving object appears in the bright part, the state in which the brightness of the photographed image after the brightness adjustment is saturated. If this is the case, there is a problem that it becomes difficult for the observer to recognize the moving object (it takes time to recognize).

  The image combining apparatus, the image combining method, and the image combining program according to the present invention have been made to solve the above problems, and the uneven brightness due to the difference in brightness between the photographed images at the joint of the combined image. And provide a composite image that is easy to recognize.

  The image combining apparatus according to an aspect of the present invention combines the plurality of captured images so that a part of the plurality of captured images obtained by the plurality of capturing apparatuses capture one another and generates one continuous image. A reference area setting unit configured to set a part of at least one of the plurality of photographed images as a reference area, and a first photographed image as the photographed image in which the reference area is set. The average luminance of the reference area is adjusted such that the average luminance of the reference area is less than the first threshold when the average luminance of the reference area in the image is equal to or greater than a predetermined first threshold. A second adjustment for adjusting the average luminance of a second photographed image which is the photographed image adjacent to the first photographed image based on the adjustment of 1 and the average luminance of the reference area after the first adjustment And the above A luminance adjustment unit that performs the second adjustment based on the average luminance of the first photographed image for which the first adjustment has not been performed, when the average luminance of the reference area is less than the first threshold; And an image combining unit configured to combine the plurality of photographed images whose brightness has been adjusted by the brightness adjusting unit to generate one continuous image.

  In an image combining method according to another aspect of the present invention, a plurality of captured images are combined such that a part of a plurality of captured images captured by a plurality of capturing devices is superimposed and one continuous image is displayed. In the image combining method for generating, a reference area setting step of setting a part of at least one photographed image of the plurality of photographed images as a reference area, and the photographed image in which the reference area is set. When the average luminance of the reference area in one captured image is equal to or greater than a predetermined first threshold, the average luminance of the reference area is set so that the average luminance of the reference area is less than the first threshold. Adjusting an average luminance of a second photographed image which is the photographed image adjacent to the first photographed image based on the first adjustment to be adjusted and the average luminance of the reference area after the first adjustment; Adjustment of 2 And if the average luminance of the reference area is less than the first threshold, perform the second adjustment based on the average luminance of the first captured image for which the first adjustment has not been performed. And an image combining step of combining the plurality of photographed images whose brightness has been adjusted by the brightness adjusting step to generate one continuous image.

  According to the present invention, it is possible to reduce luminance unevenness due to a difference in luminance between photographed images at joints of synthetic images, and to provide a synthetic image that is easy to recognize.

FIG. 1 is a block diagram showing a schematic configuration of an image combining device according to Embodiment 1 of the present invention. FIG. 7 is a diagram showing the arrangement of a first imaging device, a second imaging device, a third imaging device, and a fourth imaging device, and an imaging range in Embodiment 1. FIG. 7 is a diagram showing a reference area in a video (image) captured by the first imaging device, the second imaging device, the third imaging device, and the fourth imaging device in the first embodiment. 7 is a flowchart showing an operation of a luminance adjustment unit in Embodiment 1. It is a flowchart which shows the detail of operation | movement of the process A in FIG. FIG. 6 is a diagram for describing an operation of an image combining unit in Embodiment 1. FIG. 17 is a diagram showing a reference area and a surrounding area in a video (image) captured by the first imaging device, the second imaging device, the third imaging device, and the fourth imaging device in the second embodiment. FIG. 13 is a flowchart showing an operation of a luminance adjustment unit in Embodiment 2. FIG. It is a flowchart which shows the detail of operation | movement of the process A in FIG. It is a figure which shows the form of the synthesized image in a modification.

  Hereinafter, an image combining device according to an embodiment of the present invention will be described. In the embodiment, an example in which four imaging devices are used will be described, but the number of imaging devices is not limited to four. In addition, the invention of the image combining device described below can be understood as the invention of an image combining method or an image combining program.

<< 1 >> Embodiment 1
<< 1-1 >> Configuration of First Embodiment FIG. 1 is a block diagram showing a schematic configuration of an image combining device 100 according to the first embodiment of the present invention. In the first embodiment, the image combining device 100 is, for example, a computer. As shown in FIG. 1, the image combining device 100 includes an image processing unit 200, a storage device 210, an input interface 220, an output interface 230, and a reference area setting unit 240. The image processing unit 200, the storage device 210, the input interface 220, and the output interface 230 are hardware of the image combining device 100. As shown in FIG. 1, the input interface 220 is connected to the first imaging device 101a, the second imaging device 101b, the third imaging device 101c, and the fourth imaging device 101d.

  As shown in FIG. 1, the image processing unit 200 includes a distortion correction unit 110, a luminance adjustment unit 120, and an image combining unit 130. The image processing unit 200 is a processor that executes instructions described in software in the image combining device 100.

  The image processing unit 200 is connected to other hardware via a signal line and controls the other hardware. The image processing unit 200 is an integrated circuit (IC) that performs processing. The image processing unit 200 is, for example, a CPU (Central Processing Unit).

  In the following description, the functions of the distortion correction unit 110, the brightness adjustment unit 120, and the image combining unit 130 in the image processing unit 200 are collectively referred to as the function of the image processing unit 200. The function of the image processing unit 200 of the image combining device 100 can be realized by software.

  The image synthesizing device 100 may include only one image processing unit 200, or may include a plurality of image processing units 200. The plurality of image processing units 200 may cooperatively execute a program for realizing the function of the image processing unit 200.

  As shown in FIG. 1, the storage device 210 includes an auxiliary storage device 211 and a memory 212. The auxiliary storage device 211 includes a storage unit 150. The storage unit 150 stores a distortion correction table 160, reference area information 170, a coordinate conversion table 180, and camera installation information 190. The storage device 210 includes an auxiliary storage device 211 and a memory 212.

  The auxiliary storage device 211 is, for example, a read only memory (ROM), a flash memory, or a hard disk drive (HDD). The memory 212 is, for example, a random access memory (RAM). The storage unit 150 is realized by the auxiliary storage device 211. The storage unit 150 may be realized by the auxiliary storage device 211 and the memory 212.

  In the auxiliary storage device 211, a program for realizing the function of the image processing unit 200 is stored. The program is loaded into the memory 212, read into the image processing unit 200, and executed by the image processing unit 200. The auxiliary storage device 211 also stores an operating system (OS). At least a part of the OS is loaded into the memory 212, and the image processing unit 200 executes a program that implements the functions of the image processing unit 200 while executing the OS.

  Information, data, signal values, and variable values indicating the results of processing by the function of the image processing unit 200 are stored in the auxiliary storage device 211, the memory 212, or a register or cache memory in the image processing unit 200. In FIG. 1, an arrow connecting each unit to the storage unit 150 indicates that each unit stores the result of processing in the storage unit 150 or that each unit reads information from the storage unit 150. Further, arrows connecting the respective parts represent the flow of control. Further, arrows between the memory 212 and each part indicating that each piece of image information is exchanged between the parts via the memory 212 will be omitted.

  The program for realizing the function of the image processing unit 200 may be stored in a portable recording medium such as a magnetic disk, a flexible disk, an optical disk, a compact disk, a Blu-ray (registered trademark) disk, or a DVD (Digital Versatile Disc).

  A program for realizing the function of the image processing unit 200 is also referred to as an image processing program. An image processing program product is a storage medium and a storage device in which the image processing program is recorded, and a computer readable program is loaded regardless of the format of the image processing program.

  The input interface 220 is a port connected to the first imaging device 101a, the second imaging device 101b, the third imaging device 101c, and the fourth imaging device 101d. The input interface 220 is an image input interface that takes in an image captured by each imaging device into the image combining device 100. Each captured image captured by the input interface 220 is stored in the memory 212. Further, the input interface 220 may be a port connected to an input device such as a mouse, a keyboard, or a touch panel.

  Specifically, the input interface 220 is a port connected to a LAN (Local Area Network). The input interface 220 may be a USB (Universal Serial Bus) terminal. In addition, the input interface 220 is an image synthesis device 100 for video signals such as serial digital interface (SDI), high definition multimedia interface (HDMI) (registered trademark), video graphics array (VGA), digital video interface (DVI), and components. It may be a capture board that can be captured.

  The output interface 230 is a port to which a cable of a display device such as a display is connected. The output interface 230 is, for example, a USB terminal or an HDMI (registered trademark) (High Definition Multimedia Interface) terminal. The display is, for example, an LCD (Liquid Crystal Display).

  The reference area setting unit 240 has a function of setting a reference area in each captured image. The reference area is an area in which the observer (supervisor) of the composite image focuses on the captured image with priority. This reference area can be set by the observer (user) to an arbitrary area in an arbitrary image of each imaging device. As shown in FIG. 1, the reference area set by the reference area setting unit 240 is stored in the reference area information 170 of the storage unit 150.

<1-2> Operation of First Embodiment Next, the operation of the image combining device 100 according to the first embodiment of the present invention will be described. FIG. 2 is a diagram showing the arrangement of the first imaging device 101a, the second imaging device 101b, the third imaging device 101c, and the fourth imaging device 101d and the imaging range 300 in the first embodiment. In the following description, the first imaging device 101a, the second imaging device 101b, the third imaging device 101c, and the fourth imaging device 101d are also referred to as imaging devices 101a, 101b, 101c, and 101d.

  As shown in FIG. 2, in the first embodiment, the wide imaging range 300 is imaged by four imaging devices 101a, 101b, 101c, and 101d. In the example of FIG. 2, the imaging range 300 is a road, the imaging devices 101a, 101b, 101c, and 101d are monitoring cameras that monitor the road condition, and the monitoring target (subject) 301 is a vehicle traveling on the road.

  The imaging devices 101a, 101b, 101c, and 101d are camera devices including an imaging element such as a CCD (Charged-Coupled Devices) or a CMOS (Complementary Metal-Oxide-Semiconductor) and a lens.

  The imaging devices 101a, 101b, 101c, and 101d include the same type of imaging device and the same type of lens. The imaging devices 101a, 101b, 101c, and 101d are, for example, fixed cameras such as surveillance cameras. Further, it is desirable that the photographing devices 101a, 101b, 101c, and 101d have an AGC (Automatic Gain Control) function.

  Next, an operation of combining the images captured by the imaging devices 101a, 101b, 101c, and 101d will be described. First, the operation of the distortion correction unit 110 in the image processing unit 200 will be described. The images captured by the imaging devices 101a, 101b, 101c, and 101d are distorted due to lens characteristics of the imaging device and the like. In particular, when a wide-angle lens is used, distortion of a captured image becomes large.

  In order to correct this distortion, the images captured by the imaging devices 101a, 101b, 101c, and 101d are taken into the image combining device 100 by the input interface 220, temporarily stored in the memory 212, and then corrected for distortion. It is sent to 110.

  The distortion correction unit 110 performs processing for correcting distortion of a captured image. In distortion correction processing, correction data for correcting lens distortion in an imaging device to be used is created in advance as a distortion correction table 160 and held in the storage unit 150. The distortion correction unit 110 corrects distortion of a captured image with reference to the correction data of the distortion correction table 160 held in the storage unit 150.

  The distortion correction table 160 can be provided individually for each type of imaging device. Even when the type of imaging apparatus is different, distortion of a captured image can be corrected by using a distortion correction table that is matched to the imaging apparatus. The photographed image subjected to distortion correction by the distortion correction unit 110 is sent to the luminance adjustment unit 120. The luminance adjustment unit 120 adjusts the luminance of the photographed image after the distortion correction is performed.

  Next, the operation of the luminance adjustment unit 120 will be described using FIGS. 3 to 5. As shown in FIG. 2, in an image in which camera images (images) output from four cameras installed in different directions are projectively transformed and connected to one projection plane 360, the first imaging device 101 a, the second If the variation in brightness among photographed images (images) of the photographing device 101b, the third photographing device 101c, and the fourth photographing device 101d is large, the continuity of the combined image is lost. Therefore, the brightness adjustment unit 120 adjusts the brightness between the photographed images.

  FIG. 3 shows the brightness adjustment of the composite image in the video (image) captured by the first imaging device 101a, the second imaging device 101b, the third imaging device 101c, and the fourth imaging device 101d in the first embodiment. It is a figure shown about the reference field (brightness reference field) referred when performing.

  In FIG. 3, an image 400 a is an image of which an image captured by the first imaging device 101 a has been subjected to distortion correction by the distortion correction unit 110, and an image 400 b is an image captured by the second imaging device 101 b by the distortion correction unit 110. The image 400c is an image subjected to distortion correction, the image 400c is an image subjected to distortion correction by the distortion correction unit 110, and the image 400d is each image taken by the fourth imaging device 101d. It is an image that has been subjected to distortion correction by the distortion correction unit 110.

  Hereinafter, the image 400a, the image 400b, the image 400c, and the image 400d are collectively referred to as distortion-corrected images 400a, 400b, 400c, and 400d.

  As in the example shown in FIG. 2, when the image synthesizing device 100 according to the first embodiment is used as a monitoring application, a subject (for example, a vehicle) to be monitored appears in a captured video (image) (frame in) Or it is important to capture the moment when it disappears (frames out) from within the captured image (image).

  Therefore, in the first embodiment, when the image combining apparatus 100 pre-specifies an area where a subject frame-in or frame-out in a captured image as a reference area, when the subject is frame-in or frame-out, the inside of the reference area The brightness adjustment of the entire composite image is performed so that the brightness of the image is appropriate.

  Specifically, the reference area A1 is set at the right end of the road in the image 400a, and the reference area A2 is set at the left end of the road in the image 400d. Note that the reference area refers to a partial area in which an object is framed in or out in an image (in particular, a moving image), and does not indicate the entire screen.

  The priority when referring to the reference areas A1 and A2 is A1 and A2 in this order, and the first reference area is referred to as the reference area A1 and the second reference area is referred to as the reference area A2. Note that one or more reference areas are provided in the composite image. For example, a plurality of reference areas may be provided in each photographed image forming a composite image, or a photographed image in which the reference area is not provided may exist.

  Further, FIG. 3 shows overlapping regions (overlapping portions) when combining the images 400a, 400b, 400c, and 400d. 452a is a superimposed area of the image 400a and the image 400b in the image 400a, 451b is a superimposed area of the image 400a and the image 400b in the image 400b, 452b is a superimposed of the image 400b and the image 400c in the image 400b An area 451c is a superimposed area of the image 400b and the image 400c in the image 400c, 452c is a superimposed area of the image 400c and the image 400d in the image 400c, and 451d is the image 400c and the image in the image 400d It is an overlapping area with 400d. Details of the process of combining the images 400a, 400b, 400c, and 400d will be described later.

  Next, the brightness adjustment processing of the brightness adjustment unit 120 according to the first embodiment will be described using FIGS. 4 and 5. FIG. 4 is a flowchart showing the operation of the luminance adjustment unit 120 in the first embodiment. First, the brightness adjustment unit 120 reads the reference area A1 which is the first reference area from the reference area information 170, and stores the image 400a including the reference area A1 in the memory 212 as a reference image (step ST101). The reference area information 170 stores information related to the reference area (for example, an imaging device including the reference area, the position of the reference area in the screen, and the threshold of the luminance of the reference area).

  Next, the luminance adjusting unit 120 calculates the average luminance Ia1 of the reference area A1 which is the first reference area (step ST102), and the threshold X relating to the luminance of the reference area held in advance in the reference area information 170 (first And the average luminance Ia1 of the reference area A1 (step ST103).

  When the average luminance Ia1 of the reference area A1 is equal to or higher than the threshold X (NO in step ST103), the luminance adjustment unit 120 proceeds to step ST104 and performs the reference until the average luminance Ia1 of the reference area A1 becomes lower than the threshold X The screen brightness level of the image 400a is lowered (first adjustment). If the average luminance Ia1 of the reference area A1 is less than the threshold X (YES in step ST103), the process proceeds to the next step (step ST105).

  The threshold value X is set to a value such that the average luminance in the reference area is not overexposed (the luminance is not too high). For example, in 8-bit data in which 255 is the brightest maximum luminance, the value is set to a value in the range of approximately 200 to 250. The threshold value X is not limited to the above value, and it is preferable that the threshold value X be appropriately adjusted according to the installation environment of the imaging device, the type of subject, and the like. The threshold value X may be set appropriately by the observer (user) by means not shown.

  In the process of step ST105, the brightness adjustment unit 120 adjusts the screen brightness of the image 400b adjacent to the reference image 400a (second adjustment). The screen luminance level of the reference image 400a or the adjacent image 400b so that the luminance levels of the superimposed portions 452a and 451b of the reference image 400a and the adjacent image 400b become the same within the range where the average luminance Ia1 of the reference area A1 does not exceed the threshold X Adjust the

  Here, in general, the photographed image is a screen that is easy for the observer to see when the screen brightness is high. Therefore, when the average luminance Ia1 of the reference area A1 does not satisfy the threshold X, the adjustment is performed so that the screen luminance of the reference image 400a becomes high in the range where the average luminance Ia1 of the reference area A1 does not exceed the threshold X The screen brightness of the adjacent image 400b is adjusted to be the same level as the screen brightness of the reference image 400a.

  Alternatively, the luminance of the adjacent image 400b may be adjusted without changing the screen luminance of the reference image 400a (the screen luminance of the reference image 400a remains unchanged). This is because the reference image 400a is the most important image for the observer, and it may be natural to see a composite image generated according to the screen brightness of the reference image 400a.

  With regard to the method of adjusting the screen brightness of the reference image and the adjacent image, the observer (user) can select in advance whether to adjust without changing the screen brightness of the reference image or to adjust the screen brightness of the reference image to be high. The reference area information 170 holds the option of the brightness adjustment method to be selected, and in step ST105, the option of the brightness adjustment method is read, and the brightness adjustment is performed by the read method. It is also good.

  Here, as a method of adjusting the luminance levels of the superimposed portions 452a and 451b of the reference image 400a and the adjacent image 400b, the average luminance of the superimposed portion 452a and the average luminance of the superimposed portion 451b may be the same level. Also, since the corresponding points of the overlapping portion 452a and the overlapping portion 451b are known, the feature points of the overlapping portion 452a and the overlapping portion 451b are extracted in advance from several points to several tens points, and the average luminance of the feature points is the same. The brightness may be adjusted to be the level.

  Next, in the process of step ST106, the luminance adjusting unit 120 determines whether a reference area exists in the adjacent image 400b whose luminance has been adjusted. In the first embodiment, since the reference area does not exist in the adjacent image 400b, the process proceeds to the next step ST107.

  Next, in the process of step ST107, the luminance adjustment unit 120 determines whether or not the luminance adjustment process has been performed on all photographed screens (photographed images). In the present embodiment, since the brightness adjustment processing of the image 400c and the image 400d is not performed yet, the process proceeds to step ST108.

  Next, in the process of step ST108, the luminance adjusting unit 120 sets the adjacent image 400b adjusted in luminance in step ST105 as a temporary reference image, and the luminance level of the overlapping portion 452b with the image 400c adjacent to the temporary reference image 400b. Are calculated and stored in the memory 212.

  Next, in the process of step ST109, the luminance adjusting unit 120 is configured such that the luminance level of the superimposed portion 451c of the image 400c adjacent to the temporary reference image 400b is the same as the luminance level of the superimposed portion 452b stored in the memory 212. , Adjust the screen brightness level of the adjacent image 400c. When the process of step ST109 ends, the process proceeds to step ST106.

  Here, the method of adjusting the screen luminance level may be performed so that the average luminance of the superimposed portion 452b and the average luminance of the superimposed portion 451c become the same level as the reference image 400a and the adjacent image 400b. Further, since the corresponding points of the overlapping portion 452b and the overlapping portion 451c are known, the feature points of the overlapping portion 452b and the overlapping portion 451c are extracted in advance from several points to several tens points, and the average luminance of the feature points is the same. The brightness may be adjusted to be the level.

  Next, in step ST106, the luminance adjustment unit 120 determines whether the reference area is included in the image 400c whose screen luminance level has been adjusted in the immediately preceding process. In the present embodiment, since the reference area does not exist in the adjacent image 400c, the process proceeds to the next step ST107, and the brightness adjustment unit 120 determines whether or not the brightness adjustment processing has been performed on all photographed screens. . In the present embodiment, since the brightness adjustment process of the image 400d is not performed yet, the process proceeds to step ST108.

  In the process of step ST108, the brightness adjustment unit 120 sets the adjacent image 400c, which has been subjected to the brightness adjustment immediately before, as a temporary reference image, calculates the brightness level of the overlapping portion 452c of the temporary reference image 400c with the adjacent image 400d, and stores the memory. It stores in 212 (step ST108).

  In the process of step ST109, the brightness adjustment unit 120 sets the adjacent image so that the brightness level of the superimposed portion 451d of the image 400d adjacent to the temporary reference image 400c is the same as the brightness level of the superimposed portion 452c stored in the memory 212. Adjust the screen brightness level of 400d. The adjustment method of the screen luminance level here is the same as that of the reference image 400a and the adjacent image 400b.

  Next, in step ST106, the luminance adjustment unit 120 determines whether the reference area is included in the image 400d of which the screen luminance level has been adjusted in the immediately preceding process. Since the adjacent image 400d includes the second reference area A2, the processing shifts to processing A.

  FIG. 5 is a flowchart showing the details of the operation of process A in FIG. As shown in FIG. 5, in processing A, the brightness adjustment unit 120 sets an image 400 d including the second reference area as a temporary reference image, calculates the average brightness I an of the reference area A 2, and stores it in the memory 212. (Step ST201).

  Next, the luminance adjustment unit 120 compares the threshold X held in advance in the reference area information 170 with the average luminance Ian of the reference area A2 (step ST202). When the average luminance Ian of the reference area A2 is less than the threshold X (YES in step ST202), the luminance adjustment unit 120 proceeds to processing B without performing any processing, and performs luminance adjustment processing for all photographed screens. It is determined whether or not (step ST107 in FIG. 4).

  In step ST202, when the average luminance Ian of the reference area A2 is equal to or higher than the threshold X (NO in step ST202), the luminance adjustment unit 120 proceeds to the process of step ST203, and the average luminance Ian of the reference area A2 is the threshold X The screen brightness level of the provisional reference image 400d is lowered until the value becomes less than.

  In step ST204, the luminance adjusting unit 120 stores the rate (decreasing rate) at which the screen luminance level of the temporary reference image 400d is lowered in the memory 212, and removes the temporary reference image 400d in accordance with the lowering rate stored in the memory 212. The screen brightness levels of all the images 400c, 400b, and 400a are lowered. After the process of step ST204 ends, the process proceeds to process B, and the brightness adjustment unit 120 determines whether the brightness adjustment process has been performed on all the photographed screens (step ST107).

  In the first embodiment, the brightness adjustment processing for all the images is completed by the end of the brightness adjustment processing for the image 400d, and the images 400a, 400b, 400c, and 400d, which are the four images whose brightness has been adjusted. Are stored in the memory 212. The image combining unit 130 generates a combined image from the stored four brightness-adjusted images. The generated composite image is an image that is easy to be recognized with no whiteout in the reference areas A1 and A2 and with little variation in luminance throughout the composite image.

  Next, the operation of the image combining unit 130 will be described using FIG. FIG. 6 is a diagram for explaining the operation of the image combining unit 130 in the first embodiment. The image subjected to the brightness adjustment processing by the brightness adjustment unit 120 is temporarily stored in the memory 212 and sent to the image combining unit 130. The processing operation of the image combining unit 130 is the same as that of the image sent to the image combining unit 130 even if the brightness adjustment processing has not been performed by the brightness adjusting unit 120.

  The image combining unit 130 combines the plurality of photographed images subjected to the distortion correction and the brightness adjustment as one image. The combining process is realized by pasting a plurality of images subjected to distortion correction and brightness adjustment to the projection plane 360 defined by the image combining unit 130. Note that the projection plane 360 is not a plane provided in the real space, but is a virtual projection plane provided in the memory 212, and each image is pasted at a defined coordinate position on the memory space defined as the projection plane 360. wear.

  First, as preprocessing for attaching a plurality of photographed images subjected to distortion correction and luminance adjustment to the projection surface 360, distortion correction and luminance adjustment are performed using the coordinate conversion table 180 held in the storage unit 150. Projective transformation is performed on a plurality of photographed images.

  The first captured image 400a, the second captured image 400b, the third captured image 400c, and the fourth captured image 400d subjected to distortion correction and luminance adjustment are projectively converted into the first captured image 410a and the second captured image 410b. , The third captured image 410c, and the fourth captured image 410d. Hereinafter, the first captured image 410a, the second captured image 410b, the third captured image 410c, and the fourth captured image 410d after projective transformation are also referred to as images 410a, 410b, 410c, and 410d.

  Next, the image combining unit 130 pastes the images 410 a, 410 b, 410 c, and 410 d after projective transformation on the projection plane 360.

  The attachment positions of the images 410a, 410b, 410c, and 410d after projective transformation on the projection surface 360 are the installation of the first imaging device 101a, the second imaging device 101b, the third imaging device 101c, and the fourth imaging device 101d. It is determined in advance based on position information (for example, camera posture, installation coordinates), and is stored in the camera installation information 190 of the storage unit 150.

  As shown in FIG. 6, the image combining unit 130 sets the first captured image 410 a after projective conversion to the first image projection area 361, and the second captured image 410 b after projective conversion to the second image projection area 362, The third photographed image 410 c after projective transformation is pasted to the third image projection area 363, and the fourth photographed image 410 d after projective transformation is pasted to the fourth projection area 364.

  As shown in FIG. 6, when the images 410a, 410b, 410c, and 410d after projective transformation are attached to the projection surface 360, the attachment positions are determined so that parts of adjacent images overlap each other. There is.

  A superimposed area 370 in FIG. 6 is a superimposed area of the first photographed image 410a after projective transformation and the second photographed image 410b after projective transformation, and the superimposed area 380 is a projected area of the second photographed image 410b after projective transformation. The overlapping area 390 is a superimposed area of the third captured image 410c after projective conversion and the fourth captured image 410d after projective conversion.

  The process until projective transformation of the distortion-corrected images 400a, 400b, 400c, and 400d is performed by a formula is performed, so that the pixel position on the projection plane 360 and the image after distortion correction are It is possible to take a response (know). That is, it is possible to know in advance at which position on the projection plane 360 each pixel of the distortion-corrected images 400 a, 400 b, 400 c, and 400 d is projected.

  As described above, since the correspondence between the pixel position on the projection plane 360 and the image after distortion correction can be known in advance, the distortion corrected images 400a, 400b, 400c, and 400d in FIG. The correspondence with the areas 370, 380, and 390 on which the respective images overlap on the projection plane 360 can be known.

  Specifically, the overlapping area 370 in FIG. 6 corresponds to the area 452a and the area 451b in FIG. 3, and the overlapping area 380 in FIG. 6 corresponds to the area 452b and the area 451c in FIG. The overlapping area 390 corresponds to the area 452 c and the area 451 d in FIG.

  The composite image 350 is generated by attaching the images 410 a, 410 b, 410 c, and 410 d after projective transformation to the projection plane 360. The composite image 350 generated as described above is output to a display device such as a display via the output interface 230.

<1-3> Effects of First Embodiment As described above, according to the image combining device 100 according to the first embodiment, when combining a plurality of captured images, the area where the subject appears in the shooting screen, or An area where the subject disappears from the photographing screen is set as a reference area, and the luminances of a plurality of photographed images are adjusted so that the luminance of the reference area does not exceed a predetermined threshold X. As a result, a composite image with little variation in luminance can be obtained, and in a region where the subject to be observed by the observer appears in the shooting screen or in a region where the subject disappears from the shooting screen, an easy-to-see image with no overexposure due to excessive brightness You can get it. In particular, in an application for monitoring whether or not an object appears (or disappears) in a monitoring area, it is possible to provide a composite image that is easy for the observer to view. In particular, it is useful when using a composite image as a visual aid of the observer.

<< 2 >> Second Embodiment
<< 2-1 >> Configuration of Second Embodiment In the second embodiment, points different from the first embodiment will be mainly described. In the first embodiment, it is assumed that the average luminance of the reference area does not change sharply with respect to at least the peripheral area of the reference area. On the other hand, in the second embodiment, the operation of the luminance adjustment unit 120 of the image combining device 100 when the average luminance of the reference area changes sharply with respect to the luminance of the peripheral area of the reference area will be described.

  The operation of the luminance adjustment unit 120 according to the second embodiment will be described with reference to FIGS. 7, 8 and 9. The configuration of the image combining device 100 is the same as the configuration described in the first embodiment, and thus the description thereof is omitted.

  FIG. 7 shows the reference area and the surrounding area in the video (image) captured by the first imaging device 101a, the second imaging device 101b, the third imaging device 101c, and the fourth imaging device 101d in the second embodiment. FIG. In the second embodiment, when the screen brightness of the video (image) captured by the first imaging device 101a, the second imaging device 101b, the third imaging device 101c, and the fourth imaging device 101d is low, for example, outdoor imaging at night Is assumed.

  In FIG. 7, B1 is a peripheral area of the reference area A1, and B2 is a peripheral area of the reference area A2. The peripheral areas B1 and B2 are areas adjacent to the reference areas A1 and A2. Note that the reference areas A1 and A2 are not included in the peripheral areas B1 and B2, respectively. FIG. 7 shows a state in which the vehicle 301 whose light is turned on is shown in the reference area A1.

  The reference areas A1 and A2 and the peripheral areas B1 and B2 are close to each other, and usually, a large difference in luminance does not occur. However, in the example shown in FIG. 7, since the image in which the light of the vehicle 301 is turned on appears in the reference area A1, the luminance of the reference area A1 is high and the luminance of the peripheral area B1 is low. A large difference occurs between the brightness of A1 and the brightness of the peripheral area B1.

  Next, the operation of the luminance adjustment unit 120 in the second embodiment will be described. FIG. 8 and FIG. 9 are flowcharts explaining the luminance adjustment operation of the luminance adjustment unit 120 in the second embodiment. In the flowcharts of FIG. 8 and FIG. 9, the processing that is the feature of the second embodiment is step ST300, step ST301, step ST302, and step ST303. The other processing steps are the same as in the first embodiment, and thus the description thereof is omitted.

  When the average luminance Ia1 of the first reference area becomes equal to or greater than the threshold X in step ST103, the average of the peripheral area B1 is determined from the information (the position in the screen, pixel information) of the peripheral area B1 stored in the reference area information 170. The luminance Ib1 is calculated (step ST300), and then the luminance difference between the average luminance Ia1 of the first reference area and the average luminance Ib1 of the peripheral area B1 is determined, and the luminance difference and the threshold stored in advance in the reference area information 170 It compares with Y (the second threshold) (step ST301). The threshold value Y may be set to a value such that the difference in brightness between the reference area brightness and the surrounding brightness is unnatural (appearing and subjective).

  For example, in the case of FIG. 7, since the outdoor state at night is assumed, the difference between the brightness of the reference area and the brightness of the peripheral area is 100 or more in 8-bit data having 255 as the brightest maximum brightness. Set The threshold value Y is not limited to the above value, and it is preferable that the threshold value Y be appropriately adjusted according to the installation environment of the imaging device, the type of subject, and the like. The threshold value Y may be appropriately set by the observer (user) by means not shown.

  If the luminance difference is equal to or larger than the threshold Y (NO in ST301), it is determined that a high luminance object appears in the reference area, and the process proceeds to step ST105 without performing the luminance adjustment processing. If the luminance difference is less than the threshold Y (YES in ST301), the process of step ST104 is performed as in the first embodiment.

  When the luminance difference is equal to or larger than the threshold Y, a high luminance subject appears in the reference area. In the case of FIG. 7, the luminance of the reference area A1 is high because the vehicle 301 whose light is turned on appears in the reference area A1, but the entire screen is not actually bright. If the process of step ST104 is executed in this state, the luminance of the entire reference image is lowered, and as a result, the luminance of the entire composite image is lowered, which makes the image difficult to see.

  Therefore, in the second embodiment, when the luminance difference is equal to or more than the threshold value Y, it is determined that the high luminance subject appears in the reference area, and the reference image is not adjusted to thereby perform the reference image. The composite image can be generated without reducing the brightness of the image.

  The processes of step ST303 and step ST304 in the flowchart of FIG. 9 are also basically the same as the processes of step ST300 and step ST301. When the difference luminance between the average luminance Ian of the second reference area A2 and the peripheral area luminance Ibn of the second reference area is equal to or greater than the threshold Y, the screen luminance including the reference area is not adjusted.

<2-2> Effects of Second Embodiment As described above, according to the image combining device 100 according to the second embodiment, when combining a plurality of captured images, the area where the subject appears in the shooting screen, or An area where the subject disappears from the shooting screen is set as a reference area, and a peripheral area which is an area around the reference area is set. If the difference between the luminance of the reference area and the luminance of the peripheral area is large, the reference area is selected. Since the luminance adjustment of the screen including is not performed, it is possible to provide an easily viewable composite image without wasting the luminance of the entire composite image.

<< 3 >> Modifications Although the average luminance of the reference area and the average luminance of the peripheral area are compared in the second embodiment, the average luminance of the reference area at the past time of the reference area is used instead of the luminance of the peripheral area. Is stored in the memory 212, and the average luminance of the reference area at the past time and the average luminance of the reference area at the current time are compared, and if a large luminance difference occurs between them, It may be determined that a high-brightness subject appears on the display. In this case, the memory 212 preferably has a time acquisition unit that acquires time information.

  That is, when the high-brightness subject appears, the screen brightness including the reference area is not adjusted. In this case, the average luminance of the reference area at the past time is the average luminance of the reference area when the subject is not shown in the reference area. The timing for acquiring the average luminance of the reference area at the past time may be appropriately set according to the moving speed of the subject and the like.

  Further, in the above embodiment, the case where the reference area A1, which is the first reference area, is in the image 400a positioned at the end of the composite image 350 has been described. In this case, the brightness adjustment order is processed in the order of the images 400b, 400c, and 400d adjacent to the image 400a including the first reference area.

  On the other hand, when the reference area A1 which is the first reference area is included in the image other than the end of the composite image, for example, when the image 400b includes the reference area A1, the image 400b includes the reference area A1. There are two images, an image 400c and an image 400a. Here, if the order of the brightness adjustment is performed from the side closer to the image 400d including the next reference area A2, the brightness adjustment processing can be efficiently performed. Specifically, the brightness adjustment order may be performed in the order of the image 400c, the image 400d, and the image 400a.

  Further, in the above embodiment, the case of generating a horizontally long composite image (so-called panoramic image) has been described, but the form of the composite image of the image combining device 100 of the present invention is not limited to only a horizontally long image. FIG. 10 is a view showing the form of a composite image in the modification. For example, as shown in FIG. 10, the present invention can be applied to the case of combining an image vertically and horizontally.

  In the example illustrated in FIG. 10, the luminance adjustment is performed so that the luminance level of the overlapping portion 440 with the image 400b, the image 400c, and the image 400d adjacent to the image 400a is the same. In the example of FIG. 10, after performing the brightness adjustment of the image 400a including the first reference area A1 and the image 400c including the second reference area A2, the brightness of the image 400b and the image 400d, which are the remaining adjacent images, may be adjusted. Good.

  Further, the composite image may be a bird's-eye-view image as viewed from above. In the case of converting into the overhead image, in the image combining device 100, the processing of the image combining unit 130 may be provided with the overhead conversion processing function.

  In the above embodiment, the reference area is set in advance, but subject position detection means is provided separately from the image combining apparatus 100, and the reference area is set based on the subject position information obtained from the subject detection means. You may do it. As the subject position detection means, a method of providing a subject with a GPS transmitter, installing an object detection sensor in the roadside zone, or the like can be considered.

  In the above embodiment, the shooting range is outdoor, and the subject is a vehicle. However, the usage example of the image combining apparatus of the present invention is not limited to this, and the subject is framed in a certain area within the shooting range. It is applicable to the use which carries out a flameout. For example, it may be applied to monitoring of a flight scene (airport surface, runway surface).

  In this case, a sliding surface corresponds to the road 300 in FIG. 2, and an airplane corresponds to the vehicle 301 in FIG. Also, the reference area is the airspace above the runway end. In the case of the top of the flight, it is possible to know in advance the direction in which the plane lands on the runway from the information of the airport surveillance radar etc., and in accordance with the direction in which the plane lands It becomes possible to set.

DESCRIPTION OF SYMBOLS 100 Image synthesizing device, 101a 1st imaging device, 101b 2nd imaging device, 101c 3rd imaging device, 101d 4th imaging device, 110 distortion correction part, 120 brightness adjustment part, 130 image synthesizing part, 150 storage part, 160 distortion Correction table, 170 reference area information, 180 coordinate conversion table, 190 camera installation information, 200 image processing section, 210 storage unit, 211 auxiliary storage unit, 212 memory, 220 input interface, 230 output interface, 240 reference area setting unit, 300 Shooting range, 301 subject, 350 composite image, 360 projection plane, 361 first image projection area, 362 second image projection area, 363 third image projection area, 364 fourth image projection area, 370 first projection after projection conversion Superimposed area of the image and the second captured image after projective transformation 380 Superimposed area of second photographed image after projective transformation and third photographed image after projective transformation 390 Superimposed area of third photographed image after projective transformation and fourth photographed image after projective transformation 400a After distortion correction First photographed image of 400b Second corrected image after distortion correction 400c Third photographed image after distortion correction 400d Fourth photographed image after distortion correction 410a First photographed image after projective conversion 410b After projective conversion Second photographed image, 410c third photographed image after projective transformation, 410d fourth photographed image after projective transformation, 440 overlapping area of first photographed image, second photographed image, third photographed image and fourth photographed image, 452a An image portion captured by the first imaging device of the superimposed region of the first captured image and the second captured image, 451b An image captured by the second imaging device of the superimposed region of the first captured image and the second captured image Part, 452b An image portion captured by the second imaging device of the superimposed area of the second captured image and the third captured image, 451c An image portion captured by the third imaging device of the superimposed area of the second captured image and the third captured image , 452c An image portion captured by the third imaging device of the superimposed region of the third captured image and the fourth captured image, 451d captured by the fourth imaging device of the superimposed region of the third captured image and the fourth captured image Image part, A1 first reference area, A2 second reference area, B1 peripheral area of first reference area, B2 peripheral area of second reference area.

Claims (9)

In an image combining device that combines a plurality of captured images so as to overlap a part of a plurality of captured images captured by a plurality of capturing devices, and generates one continuous image,
A reference area setting unit configured to set a part of at least one photographed image of the plurality of photographed images as a reference area;
When the average luminance of the reference area in the first photographed image, which is the photographed image in which the reference area is set, is equal to or greater than a predetermined first threshold, the average luminance of the reference area is the first The first adjustment of adjusting the average luminance of the reference area to be less than the threshold, and the photographed image adjacent to the first photographed image based on the average luminance after the first adjustment of the reference area. Perform a second adjustment to adjust the average brightness of a second captured image;
When the average luminance of the reference area is less than the first threshold, the second adjustment is performed based on the average luminance of the first photographed image for which the first adjustment is not performed. Department,
An image combining unit configured to combine the plurality of photographed images whose brightness has been adjusted by the brightness adjusting unit to generate the one continuous image. When the difference between the average luminance in the reference area and the average luminance in the peripheral area adjacent to the reference area is equal to or greater than a predetermined second threshold, the luminance adjustment unit may adjust the first adjustment. The image synthesizing device according to claim 1, wherein A time acquisition unit that acquires time information;
And a storage unit storing average brightness of the reference area at a past time.
When the difference between the average luminance of the reference area at the current time and the average luminance of the reference area at the past time is equal to or greater than a predetermined second threshold, the luminance adjustment unit may The image synthesizing device according to claim 1, wherein the adjustment of is not performed. Among the plurality of photographed images, the first photographed image for which the reference area is set is a photographed image arbitrarily selected by the user of the image combining device,
The image synthesizing device according to any one of claims 1 to 3, wherein the reference area is an area set by the user at an arbitrary position of the first photographed image. The reference area setting unit sets two or more of the reference areas in different areas of one of the plurality of photographed images, or sets two or more of the plurality of photographed images. The image synthesizing device according to any one of claims 1 to 4, wherein two or more reference areas are set. The image combining apparatus according to claim 5, wherein the brightness adjustment unit prioritizes the two or more reference areas, and performs the first and second adjustments in accordance with the priorities. The image synthesizing device according to any one of claims 1 to 6, further comprising a distortion correction unit configured to correct distortion of the plurality of photographed images. An image combining method for combining the plurality of captured images so as to generate a single continuous image so that a part of the plurality of captured images captured by the plurality of capturing devices is superimposed;
A reference area setting step of setting a part of at least one photographed image of the plurality of photographed images as a reference area;
When the average luminance of the reference area in the first photographed image, which is the photographed image in which the reference area is set, is equal to or greater than a predetermined first threshold, the average luminance of the reference area is the first The first adjustment of adjusting the average luminance of the reference area to be less than the threshold, and the photographed image adjacent to the first photographed image based on the average luminance after the first adjustment of the reference area. Perform a second adjustment to adjust the average brightness of a second captured image;
When the average luminance of the reference area is less than the first threshold, the second adjustment is performed based on the average luminance of the first photographed image for which the first adjustment is not performed. Step and
An image combining step of combining the plurality of photographed images whose brightness has been adjusted in the brightness adjusting step to generate the one continuous image. On the computer
An image combining program for combining the plurality of captured images so as to generate one continuous image so that a part of the plurality of captured images captured by the plurality of capturing devices is superimposed,
Reference area setting processing of setting a part of at least one photographed image of the plurality of photographed images as a reference area;
When the average luminance of the reference area in the first photographed image, which is the photographed image in which the reference area is set, is equal to or greater than a predetermined first threshold, the average luminance of the reference area is the first The first adjustment of adjusting the average luminance of the reference area to be less than the threshold, and the photographed image adjacent to the first photographed image based on the average luminance after the first adjustment of the reference area. Perform a second adjustment to adjust the average brightness of a second captured image;
When the average luminance of the reference area is less than the first threshold, the second adjustment is performed based on the average luminance of the first photographed image for which the first adjustment is not performed. Processing and
An image combining program for executing an image combining process of combining the plurality of photographed images whose brightness has been adjusted by the brightness adjustment process to generate one continuous image.

Images