JP2018186514A - Image processing device, imaging system having the same and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To enable proper display of a captured image as a reference while boundaries of areas of respective images to be combined in a composed panoramic composite image are suppressed from being unnaturally conspicuous.SOLUTION: An image processing device 3 includes a reference camera determination unit 41 that determines one of the plurality of cameras CAM 1 -CAM 7 as a reference camera, an imaging condition acquisition unit 42 that acquires imaging conditions relating to exposure and white balance set in the reference camera, an imaging condition setting unit 43 for transmitting setting commands concerning exposure and white balance to other cameras excluding the reference camera based on the imaging conditions of the reference camera, and an image composing unit 43 that performs image composing processing on a plurality of captured images acquired from a plurality of cameras CAM1 to CAM7 and outputs a panoramic composite image.SELECTED DRAWING: Figure 3

Description

  The present invention relates to an image processing apparatus that performs image composition processing on a plurality of captured images and outputs a composite image, an imaging system including the image processing apparatus, and an image processing method.

  Conventionally, in an imaging system that generates a single panoramic composite image by combining images captured by a plurality of cameras, a wide-angle image that cannot be obtained by a single camera can be obtained in a seamless state by performing so-called stitching processing. Can be generated. In such stitching processing, two adjacent cameras are arranged so that the respective imaging areas partially overlap, and the image areas at the boundary portions corresponding to the overlapping portions of the imaging areas of the two cameras are overlapped. In addition, the image is synthesized by appropriately trimming.

  On the other hand, if there is a subject whose distance from the camera is significantly different in the overlapping area of the imaging areas of the two cameras, that is, if there is a subject that is a distant view and a subject that is a close-up view, between the captured images of the two cameras In this case, the positional relationship between the image of the subject in the distant view and the image of the subject in the close shadow is shifted, so-called parallax occurs, and the close-up image appears twice in the composite image, or one of the close-up images. The trouble that a part disappears occurs. Thus, in stitching, parallax correction is performed to suppress image defects caused by parallax.

  As for such parallax correction, conventionally, the positional relationship between the images of the subjects appearing in the captured images of the two cameras is obtained by block matching based on edges and feature amounts, and the image is deformed based on this information. A technique for performing parallax correction is known (see Patent Document 1).

  Further, a bent stitching boundary is set with respect to the two images so as to avoid an image region in which a near-field image such as a person appears, that is, an image region having a high cost function, and 2 along the stitching boundary. A technique for synthesizing images after performing trimming to cut out two images is known (see Patent Document 2).

2010-50842 Japanese Patent No. 5225313

  By the way, in the above conventional imaging system, since the imaging environment (for example, forward light and backlight) of each camera is different from each other, imaging conditions regarding exposure, white balance, etc. are individually set so as to be appropriate for each camera. It is common.

  However, in the conventional techniques disclosed in Patent Documents 1 and 2, when the stitching processing of captured images (composited images) by a plurality of cameras in which imaging conditions are individually set as described above is executed as it is, There is a problem that the boundary of the area of each captured image in the composite image becomes unnaturally conspicuous due to a difference in brightness and hue between images.

  On the other hand, it is conceivable to set the imaging conditions for exposure, white balance, etc. uniformly for all cameras, but the reference captured image (for example, a highly important captured image that may be noticed by the user) ), The exposure and white balance are not properly set, and the imaging target is not properly displayed (for example, overexposure or blackout occurs).

  The present invention has been devised to solve such problems of the prior art, and the main purpose of the present invention is to appropriately display a region of a captured image as a reference in a panorama composite image while appropriately displaying the region. An object of the present invention is to provide an image processing apparatus that suppresses unnaturally conspicuous boundaries between regions of each captured image, an imaging system including the image processing apparatus, and an image processing method.

  An image processing apparatus of the present invention is an image processing apparatus that is communicably connected to a plurality of cameras, outputs a panorama composite image by performing an image composition process on a plurality of captured images acquired from the plurality of cameras. A reference camera determination unit that determines one of the plurality of cameras as a reference camera, an imaging condition acquisition unit that acquires an imaging condition related to exposure and white balance set in the reference camera, and a plurality of cameras Among them, an imaging condition setting unit that sends a setting command related to exposure and white balance based on the imaging conditions of the reference camera to other cameras other than the reference camera, and a plurality of captured images acquired from the plurality of cameras And an image composition unit for performing image composition processing and outputting the panorama composite image.

  The image processing method of the present invention outputs a panorama composite image by performing an image composition process on a plurality of captured images acquired from the plurality of cameras in an image processing apparatus connected to be communicable with the plurality of cameras. A reference camera determination step of determining one of the plurality of cameras as a reference camera, and an imaging condition acquisition step of acquiring an imaging condition related to exposure and white balance set in the reference camera An imaging condition setting step of sending an imaging condition setting command relating to exposure and white balance to other cameras other than the reference camera among the plurality of cameras based on the imaging conditions of the reference camera; The panoramic composite image is obtained by performing image composition processing on a plurality of captured images acquired from the camera And having an image combining step of outputting.

  ADVANTAGE OF THE INVENTION According to this invention, it becomes possible to suppress that the boundary of the area | region of each synthesized captured image stands out unnaturally, displaying the area | region of the captured image used as a reference | standard appropriately in a panoramic composite image.

1 is an overall configuration diagram showing an imaging system according to an embodiment Explanatory drawing which shows the synthesized image displayed with the display apparatus in FIG. Functional block diagram showing an imaging system according to an embodiment Explanatory drawing which shows typically the state of the image when not implementing parallax correction, and when parallax correction is implemented Explanatory drawing which shows the actual state of the image when parallax correction is not performed and when parallax correction is performed Explanatory drawing which shows the determination method of the reference camera by a reference camera determination part Explanatory drawing which shows the 1st modification of the determination method of the reference camera by a reference camera determination part. Explanatory drawing which shows the 2nd modification of the determination method of the reference camera by a reference camera determination part. FIG. 3 is a flowchart showing a processing flow of the image processing apparatus according to the embodiment. Explanatory drawing which shows the processing content of step ST107 in FIG.

  A first invention made to solve the above problems is connected to a plurality of cameras so as to be communicable, and performs an image synthesis process on a plurality of captured images acquired from the plurality of cameras to output a panorama synthesized image. A reference camera determination unit that determines one of the plurality of cameras as a reference camera, and an imaging condition acquisition unit that acquires imaging conditions related to exposure and white balance set in the reference camera An imaging condition setting unit that sends a setting command related to exposure and white balance based on the imaging conditions of the reference camera to other cameras other than the reference camera among the plurality of cameras, and the plurality of cameras An image composition unit that performs image composition processing on the plurality of acquired captured images and outputs the panorama composite image And features.

  According to the image processing apparatus of the first aspect, one of the plurality of cameras is determined as the reference camera, and the imaging conditions of the other cameras are set based on the imaging conditions regarding the exposure and white balance of the reference camera. Therefore, the boundary of each synthesized image area is unnatural while appropriately displaying the area of the captured image serving as a reference (that is, suppressing overexposure or blackout in the area of the captured image by the reference camera). It is possible to suppress the conspicuousness (that is, to suppress the difference in brightness and hue in a plurality of captured image areas).

  In a second aspect based on the first aspect, the reference camera determining unit obtains luminance information of captured images respectively captured by the plurality of cameras when the reference camera is determined. A camera that captures a captured image having a median luminance value among the luminance values is determined as the reference camera.

  According to the image processing apparatus of the second invention, one of a plurality of cameras is used as a reference camera for capturing a standard captured image by a simple method based on a statistical comparison of the brightness of the captured image. Can be determined appropriately.

  In a third aspect based on the first aspect, the reference camera determination unit acquires information on the luminance of captured images respectively captured by the plurality of cameras when the reference camera is determined. A camera that captures a captured image having a brightness closest to an average value of the brightness is determined as the reference camera.

  According to the image processing apparatus of the third aspect of the present invention, one of a plurality of cameras is used as a reference camera for capturing a standard captured image by a simple method based on a statistical comparison of the brightness of the captured image. Can be determined appropriately.

  In a fourth aspect based on the first aspect, the reference camera determination unit determines one camera selected by the user from the plurality of cameras as the reference camera.

  According to the image processing apparatus of the fourth aspect of the present invention, one of a plurality of cameras can be appropriately determined as a reference camera having high importance for the user by a simple method based on the user's selection.

  In a fifth aspect based on the first aspect, the reference camera determining unit obtains an image recognition result of a predetermined imaging target in the captured images respectively captured by the plurality of cameras when determining the reference camera. The reference camera is determined based on the image recognition result.

  According to the image processing apparatus of the fifth invention, one of a plurality of cameras includes a desired imaging target by a simple method based on an image recognition result of a predetermined imaging target that can be set in advance by the user. It can be appropriately determined as a reference camera with high importance.

  In addition, in a sixth aspect based on any one of the first to fifth aspects, the imaging condition setting unit is configured to execute the image composition processing on the plurality of captured images by the image composition unit. When the difference between at least one of the luminance and the color difference in the boundary region between the two adjacent captured images exceeds a preset threshold value, the difference exceeding the threshold value is reduced with respect to the other camera. The imaging condition in the setting command is corrected.

  According to the image processing apparatus of the sixth aspect of the invention, the individual difference of the image sensor in the other camera is set even though the imaging condition of the other camera is set based on the imaging condition regarding the exposure and white balance of the reference camera. Even when the difference between at least one of the luminance and the color difference in the boundary region between two captured images exceeds a threshold due to (output variation) or the like (that is, the boundary region may be unnaturally conspicuous), the two captured images By correcting the imaging condition of at least one of the cameras that capture the image, it is possible to more reliably suppress unnaturally conspicuous boundaries between the areas of the combined captured images.

  A seventh invention is an imaging system including the image processing device according to any one of the first to sixth inventions and the plurality of cameras.

  According to an eighth aspect of the present invention, in the image processing apparatus that is communicably connected to a plurality of cameras, an image synthesis process is performed on the plurality of captured images acquired from the plurality of cameras to output a panorama synthesized image. A processing method, a reference camera determination step for determining one of the plurality of cameras as a reference camera, an imaging condition acquisition step for acquiring an imaging condition related to exposure and white balance set in the reference camera, An imaging condition setting step for sending imaging condition setting commands relating to exposure and white balance to other cameras other than the reference camera among the plurality of cameras based on the imaging conditions of the reference camera; and the plurality of cameras The image synthesis process is performed on the plurality of captured images acquired from the image and the panorama synthesized image is output. And having a image synthesis step.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is an overall configuration diagram illustrating an imaging system 1 according to an embodiment of the present invention, and FIG. 2 illustrates a composite image generated by the image processing device 3 illustrated in FIG. 1 and displayed on the display device 4. It is explanatory drawing.

  As shown in FIG. 1, the imaging system 1 includes a camera unit 2 having first to seventh cameras CAM1 to CAM7 that generate captured images (here, moving images) for synthesis, and image synthesis processing of these captured images. By performing the above, the image processing device 3 that sequentially generates the composite image and the display device 4 that displays the composite image output from the image processing device 3 are provided.

  The cameras CAM1 to CAM7 have a wide angle of view (for example, 120 °) and have a known configuration for recording captured images as digital data. The first to sixth cameras CAM1 to CAM6 are arranged at equal intervals in the circumferential direction so that the optical axes are substantially horizontal, and the seventh camera CAM7 has an optical axis facing upward in a substantially vertical direction. It is arranged to be. The cameras CAM1-CAM7 are arranged so that the imaging areas are partially overlapped between two adjacent cameras CAM1-CAM7. It should be noted that the number and arrangement of cameras that generate captured images for synthesis in the imaging system 1 are not limited to those shown in FIG. 1 and can be variously changed. For example, the upward camera CAM 7 may be omitted.

  The image processing apparatus 3 includes a PC (Personal Computer) capable of wireless or wired communication with the cameras CAM1 to CAM7. The display device 4 includes a known image display device such as a liquid crystal monitor connected to the image processing device 3.

  Although not shown in detail, the image processing device 3 functions as a CPU (Central Processing Unit) that centrally executes control of the cameras CAM1-CAM7 and the display device 4 based on a predetermined control program, a work area of the CPU, and the like. It has a known hardware configuration including a RAM (Random Access Memory), a ROM (Read Only Memory) that stores a control program executed by the CPU and data, a speaker, an input device, an HDD (Hard Disk Drive), and the like. . Further, for at least a part of various functions executed in the image processing apparatus 3 (image composition processing, imaging condition setting, etc., which will be described in detail later), the CPU has a dedicated control program (image composition processing program, imaging condition setting program). This can be realized by executing a program). Note that the image processing device 3 is not limited to a PC, and other information processing devices (such as a server) that perform the same function may be used. Further, at least a part of the functions of the image processing apparatus 3 may be replaced by processing using other known hardware.

  As shown in FIG. 2, the image processing device 3 synthesizes captured images (here, seven images) captured by the first to seventh cameras CAM1 to CAM7 to generate one panorama composite image. (Stitching) is performed for each frame, a composite image for each frame is output to the display device 4, and the composite image (camera image) is displayed on the display device 4 in real time. In the image processing device 3, instead of displaying the composite image on the display device 4, or simultaneously, it is also possible to store the composite image sequentially in a known storage device (not shown) such as an HDD.

  FIG. 3 is a functional block diagram showing the imaging system 1 according to the embodiment of the present invention. Here, only the part for processing the captured images of the two cameras CAM1 and CAM2 is shown, but the other cameras CAM3-CAM7 shown in FIG. 1 have the same configuration. In the present embodiment, the parallax correction amount calculation unit 31 is provided for each combination of two adjacent cameras CAM1-CAM7, and the panoramic image generation unit 32 and the parallax correction unit 33 are provided for each camera CAM1-CAM7.

  As shown in FIG. 3, each of the cameras CAM1 and CAM2 has the same configuration, and receives a light from the lens unit 11 including an optical system such as a lens and photoelectrically converts the image by receiving light. The operation of the image sensor unit 12 that outputs a signal (digital signal), the signal processing unit 13 that performs various signal processing on the image signal from the image sensor unit 12, and the operation of the image sensor unit 12 and the signal processing unit 13 And a camera control unit 14 that performs overall control.

  The image sensor unit 12 has a known configuration such as a CMOS image sensor, a CDS (Correlated Double Sampling), an AFE (Analog Front End), and a drive circuit. Further, the image sensor unit 12 changes a shutter value (shutter speed) related to the electronic shutter, and changes a sensor gain value applied to an AGC (Auto Gain Control) circuit (that is, an analog circuit before AD conversion) in the AFE. Thus, it is possible to adjust the exposure (brightness).

  The signal processing unit 13 performs known signal processing (digital gain adjustment, white balance correction, contour correction, γ correction, YC conversion, color difference correction, noise reduction, etc.) on the RAW image based on the image signal output from the image sensor unit 12. Etc.) and a captured image based on the luminance and color difference signals is output. The signal processing unit 13 can adjust exposure (brightness) by changing a digital gain value related to digital gain adjustment, and can also adjust white balance (hue) by changing a white balance gain value related to white balance correction. Can be adjusted.

  In the auto shooting mode executed by the cameras CAM1 to CAM7, the camera control unit 14 performs imaging with appropriate exposure and white balance, and the shutter value and sensor gain value of the image sensor unit 12 and the signal processing unit 13 Controls the digital gain value and white balance gain value.

  As will be described in detail later, when the image processing apparatus 3 determines that its own camera is the reference camera, the camera control unit 14 causes the image processing apparatus 3 to perform imaging conditions relating to exposure and white balance (here, the shutter value). , Sensor gain value, digital gain value, and white balance gain value). On the other hand, the camera control unit 14 in another camera that is not the reference camera acquires an imaging condition setting command related to exposure and white balance in the reference camera from the image processing device 3, and the imaging condition related to exposure and white balance based on the setting command. (Here, setting of shutter value and sensor gain value in the image sensor unit 12, setting of digital gain value and white balance gain value in the signal processing unit 13).

  The image processing apparatus 3 includes a stitching unit 21 that synthesizes the captured images so as to be pasted together, and an imaging control unit 22 that controls imaging of the cameras CAM1-CAM7.

  The stitching unit 21 includes a parallax correction amount calculation unit 31, a panoramic image generation unit 32, a parallax correction unit 33, and an image composition unit 34.

  The parallax correction amount calculation unit 31 performs a process of calculating a parallax correction amount that defines the degree of deformation of the image at the time of parallax correction for each frame. Specifically, parallelization (projection onto a cylinder), processing area cutout, and parallax calculation are performed. In the parallax calculation process, parallax (deviation amount) is calculated by block matching between two captured images. That is, the difference between the two captured images is calculated while shifting the two captured images little by little, and the parallax is obtained from the positional relationship in which the difference is the smallest.

  The panorama image generation unit 32 performs panoramaization (projection onto a sphere) on the captured images output from the two cameras CAM1 and CAM2 to generate two panorama images.

  The parallax correction unit 33 performs the parallax correction on the two panoramic images generated by the panoramic image generation unit 32 based on the parallax correction amount output from the parallax correction amount calculation unit 31 to obtain two parallax correction images. Processing to generate is performed.

  In the image composition unit 34, an image composition process is performed on the two parallax correction images generated by the parallax correction unit 33 to generate one composite image. The panorama composite image generated by the image composition unit 34 is output to the display device 4, and the panorama composite image is displayed on the display device 4.

  The imaging control unit 22 includes a reference camera determination unit 41, an imaging condition acquisition unit 42, and an imaging condition setting unit 43.

  Although details will be described later, the reference camera determination unit 41 determines one of the plurality of cameras CAM1 to CAM7 as a reference camera. Note that the reference camera is preferably a camera that can capture a highly important captured image (or an imaging target such as a person, an object, or a part thereof) that is noticed by the user or is likely to be the user. In addition, the imaging condition acquisition unit 42 captures imaging conditions related to exposure and white balance set by the reference camera determined by the reference camera determination unit 41 (here, shutter value, sensor gain value, digital gain value, and white balance gain). Value). Further, the imaging condition setting unit 43 sends an imaging condition setting command related to exposure and white balance to other cameras except the reference camera based on the imaging conditions related to exposure and white balance set by the reference camera.

  Next, parallax correction performed by the image processing apparatus 3 (stitching unit 21) will be described. FIG. 4 is an explanatory diagram schematically illustrating a state of an image when parallax correction is not performed and when parallax correction is performed, and FIG. 5 is an image when parallax correction is not performed and when parallax correction is performed. It is explanatory drawing which shows the actual state of. Here, for convenience of explanation, an example in which images of two adjacent cameras CAM1 and CAM2 are processed is shown.

  FIG. 4A shows an imaging situation by two adjacent cameras CAM1 and CAM2. In the example shown in FIG. 4A, a person and a mountain as a background are simultaneously imaged by two cameras CAM1 and CAM2.

  4B-1 and 4B-2 illustrate images captured by the two cameras CAM1 and CAM2. As shown in FIGS. 4B-1 and 4B-2, the images captured by the two cameras CAM1 and CAM2 are images of boundaries corresponding to overlapping portions of the imaging areas of the two cameras CAM1 and CAM2. A distant view image representing a mountain and a close-up image representing a person appear in the region. Here, the distance from the cameras CAM1 and CAM2 is greatly different between a mountain that is a distant view and a person that is a close view, and the positional relationship between the far view image and the close view image is shifted between the two captured images.

  FIGS. 4C-1 and 4C-2 illustrate composite images obtained by performing simple composition on the basis of a distant view image with respect to images captured by the two cameras CAM1 and CAM2. As described above, since the positional relationship between the distant view image and the close-up image is shifted in the two captured images, when the two captured images are simply combined on the basis of the distant view image, FIG. ), As shown in (C-2), there is a problem in that the near-shadow image appears twice or a part of the near-shadow image disappears in the composite image.

  Here, the shift of the near-shadow image when the distant view image is used as a reference represents the parallax between the captured images of the two cameras CAM1 and CAM2. The positional relationship between the distant view images that respectively appear in the two cameras CAM1 and CAM2 can be obtained in advance in the absence of a close shadow, and image synthesis is performed based on this information. Accordingly, there is no parallax in the captured images of the two cameras CAM1 and CAM2 in the absence of the near-shadow image, and when the near-shadow image appears, parallax occurs between the captured images of the two cameras CAM1 and CAM2.

  In this manner, in the state where the distant view image and the close-up image appear in the image area at the boundary portion of the picked-up images of the two cameras CAM1 and CAM2, the close-up image appears in the composite image due to the parallax generated between the two picked-up images. Such as appearing double images. Therefore, parallax correction is necessary to eliminate this problem, and there are typically two types of correction methods for this parallax correction. FIG. 4C-3 shows the case of the first correction method, and FIG. 4C-4 shows the case of the second correction method.

  As shown in FIG. 4 (C-3), the first correction method has two methods so that the positional relationship between the distant view image and the close-up image matches between the images captured by the two cameras CAM1 and CAM2. The image synthesis is performed on the captured images of the cameras CAM1 and CAM2 after performing parallax correction for deforming the image so that an image region in which a close-in image appears mainly is shifted in the horizontal direction.

  As shown in FIG. 4C-4, in the second correction method, a stitching boundary that is bent so as to avoid an image region in which a near-shadow image appears is set, and two cameras CAM1 are set along the stitching boundary. The image is synthesized after performing trimming to cut out the captured image of CAM2.

  FIG. 5A-1 shows a composite image obtained by simple synthesis without performing parallax correction, and FIG. 5A-2 shows a composite image subjected to parallax correction. FIGS. 5B-1 and 5B-2 show enlarged views of the periphery of the foreground image in the composite image shown in FIGS. 5A-1 and 5A-2, respectively.

  As shown in FIGS. 5 (A-1) and (B-1), in simple composition without parallax correction, a part of the near-shadow image disappears or the near-shadow image appears twice in the composite image. However, when parallax correction is performed as shown in FIGS. 5A-2 and 5B-2, a problem such as simple composition is improved and an appropriate image is generated. be able to.

  By performing parallax correction in this way, an appropriate image can be generated. In the present embodiment, the first correction method, that is, a distant view image and a close shadow image between the captured images of the two cameras CAM1 and CAM2 are used. In order to match the positional relationship with the images, parallax correction is performed on the captured images of the two cameras CAM1 and CAM2 so that the image is deformed so that an image region in which a near-shadow image mainly appears is shifted in the horizontal direction.

  FIG. 6 is an explanatory diagram showing a reference camera determination method by the reference camera determination unit 41 shown in FIG. 3, and FIGS. 7 and 8 show first and second modifications of the reference camera determination method, respectively. It is explanatory drawing.

  The reference camera determination unit 41 acquires captured images captured by the cameras CAM1 to CAM7 when determining the reference camera, and as illustrated in FIG. 6, brightness information (here, luminance of all pixels) in each captured image is acquired. Average). Then, the reference camera determination unit 41 determines the camera (here, the camera CAM4) that has captured a captured image having a median brightness among the plurality of brightnesses as the reference camera.

  Note that the reference camera determination unit 41 uses, as a reference camera, a camera that captures a captured image having the brightness closest to the average value of a plurality of brightnesses, instead of the camera that captures the captured image having the median brightness described above. You may decide. In addition, the calculation of the luminance in the captured image does not necessarily have to target all pixels, and a part of the captured image (for example, the range of the image selected by the user or the image region occupied by the imaging target set by the user) The luminance may be calculated from the above.

  As a first modification of the reference camera determination method, the reference camera determination unit 41 executes a known image recognition process for a photographing target preset by the user, and determines a reference camera based on the image recognition result. Is also possible. FIG. 7 shows an example in which a human face 50 is set by the user as an imaging target. The reference camera determination unit 41 captures a captured image from which the human face 50 is detected (here, the camera CAM3). ) As a reference camera.

  When a human face is detected from a plurality of captured images, the reference camera determination unit 41 calculates the area of each face area, and uses the camera that captured the captured image that maximizes the area of the face area as a reference. It can be determined as a camera. Alternatively, the reference camera determination unit 41 can calculate the number of faces in each captured image and determine the camera that has captured the captured image with the maximum number of included faces as the reference camera. As another method, the reference camera determination unit 41 detects a predetermined operation (for example, an operation of shaking hands) of an imaging target preset by the user, and determines a reference camera based on the detection result of the operation. You can also.

  Furthermore, as a second modification of the reference camera determination method, the reference camera determination unit 41 executes a known image recognition process for a predetermined marker set in advance by the user, and selects a reference camera based on the image recognition result. It is also possible to decide. FIG. 8 shows an example in which a preset two-dimensional code 60 is set by the user as a marker, and the reference camera determining unit 41 captures a captured image from which the two-dimensional code 60 is detected (here, , Camera CAM3) is determined as the reference camera. In this case, the user can designate a reference camera that captures a highly important captured image by placing the panel on which the two-dimensional code 60 is displayed in the imaging area of a desired camera. The marker used is not limited to a two-dimensional code, and various changes such as a panel with a specific color scheme can be made.

  The reference camera is not limited to the above-described example, and the reference camera can be determined based on a user operation. For example, as shown in FIG. 1, each camera CAM1-CAM7 is provided with a reference camera determination button 65, and the reference camera determination unit 41 determines a camera in which an operation of the reference camera determination button 65 is detected as a reference camera. It is good. Alternatively, in the image processing apparatus 3, it is also possible to determine the camera selected by the user via the input device (keyboard, touch panel, etc.) (or the camera that captured the captured image selected by the user on the display screen) as the reference camera. It is.

  FIG. 9 is a flowchart showing a processing flow of the image processing apparatus 3 according to the embodiment, and FIG. 10 is an explanatory diagram showing processing contents of step ST107 in FIG.

  As shown in FIG. 9, in the image processing device 3, processing for setting the imaging conditions of the cameras CAM1-CAM7 is executed. In this imaging condition setting process, as described above, first, the reference camera determination unit 41 determines one of the plurality of cameras CAM1-CAM7 as the reference camera (ST101).

  Subsequently, the imaging condition acquisition unit 42 acquires imaging conditions related to exposure and white balance set by the reference camera (ST102). Therefore, the imaging condition setting unit 43 sends an imaging condition setting command for other cameras (other than the reference camera) based on the imaging conditions regarding the exposure and white balance of the reference camera acquired by the imaging condition acquisition unit 42. The camera control unit 14 of the other camera that has acquired the setting command (here, including the shutter value, sensor gain value, digital gain value, and white balance gain value in the reference camera) Imaging conditions are set (ST103). Here, in the other camera, the shutter value, sensor gain value, digital gain value, and white balance gain value in the reference camera are set as they are as the imaging conditions.

  Next, in the image processing device 3, an image synthesis process for synthesizing a plurality of captured images is executed. First, the image composition unit 34 in the image composition unit 34 selects two adjacent captured images that have not been stitched (ST104), and the luminance (average value) Y and color difference (average) in the boundary region between the two images. Value) R and B are compared (difference value is calculated) (ST105). In this case, the luminance Y and the color differences R and B may be acquired only from pixels in a relatively flat area in the boundary area.

  Therefore, the difference value of the luminance Y in step ST105 is less than or equal to a preset luminance threshold value (for example, less than four gradations in 256 gradations), and the difference values of the color differences R and B in step ST105 are respectively If it is less than or equal to a preset color difference threshold (for example, less than 4 of 256 gradations) (ST106: Yes), it is unlikely that the boundary between two image areas will be unnaturally conspicuous. The image compositing unit 34 performs stitching processing of these two adjacent captured images (ST108).

  On the other hand, when the difference value of luminance Y exceeds a preset luminance threshold value or the difference value of color differences R and B exceeds a preset luminance threshold value (No) in step ST106. The image composition unit 34 captures at least one of the two adjacent captured images by the imaging condition setting unit 43 so that the difference value of the luminance Y and the difference values of the color differences R and B are both equal to or less than the respective threshold values. The imaging condition setting unit 43 sends a setting command (imaging condition correction command) in which at least a part of the imaging conditions is changed to the camera (ST107).

  More specifically, for example, as shown in FIG. 10A, in step ST103, the shutter value S, the sensor gain value Gs, the digital gain value Gd, the white balance gain values Gr, Gb (same values as the reference camera). As a result, brightness Y = 120, color difference R = 150, color difference B = 100 in the boundary area 71 of the captured image (3), and brightness Y = 132, color difference R in the boundary area 72 of the captured image (4). = 180, color difference B = 90 is assumed. Such variations in luminance Y and color differences R and B between the captured images (3) and (4) are caused by individual differences (performance variations) of CMOS image sensors in the image sensor unit 12 of each camera. It is.

  In this case, in the determination in step ST106, the difference value of the brightness Y between the captured image (3) and the captured image (4) (here, the absolute value of the difference = 12) exceeds a preset threshold value = 4. ing. Further, the difference values of the color differences R and B between the captured image (3) and the captured image (4) (here, the absolute value of the difference of the color difference R = 30 and the absolute value of the difference of the color difference B = 10) are set in advance. Threshold value = 4 (here, the color differences R and B are the same value).

  Therefore, in step ST107, the image composition unit 34 digitally controls the camera CAM3 by the imaging condition setting unit 43 so that the difference value of the luminance Y and the difference values of the color differences R and B are both equal to or less than the respective threshold values. A setting command in which the gain value and the white balance gain value are changed (here, Gd × 1.1, Gr × 1.2, and Gb × 0.9) is transmitted.

  Accordingly, as shown in FIG. 10B, in the boundary area 71 of the captured image (3), the same luminance Y = 132, color difference R = 180, and color difference B = the boundary area 72 of the captured image (4), respectively. 90, and the brightness and hue in the boundary region between two adjacent captured images are leveled. Thereafter, stitching processing of those captured images is executed (ST108).

  Although an example in which the digital gain value and the white balance gain value are changed (corrected) has been described here, the present invention is not limited to this, and correction for changing other imaging conditions is also possible. Further, the determination in step ST105 is sequentially executed for all adjacent captured images, and step ST107 is executed as necessary. The determination order of step ST106 (that is, the selection order of step ST104) is executed in ascending order of the numbers of the cameras CAM1 to CAM7. However, the order is not limited to this. You can also

  In step ST107, the camera to which the setting command is sent by the imaging condition setting unit 43 may be a camera that captures a captured image at a position further away from the captured image of the reference camera. In step ST107, a setting command in which the digital gain value and the white balance gain value are changed can be sent to both cameras with reference to the luminance Y and the color differences R and B of the reference camera.

  Steps ST104 to ST108 are repeatedly executed until the stitching of all captured images is finally completed (ST109). Note that the processing of steps ST101-ST1-7 and ST109 does not have to be executed for all captured images (frames). Normally, only step ST108 is executed, and at predetermined time intervals or every several frames ( For example, steps ST101-ST1-7 and ST109 may be executed every 10 frames in a 60 fps camera video.

  As mentioned above, although this invention was demonstrated based on specific embodiment, these embodiment is an illustration to the last, Comprising: This invention is not limited by these embodiment. In addition, all the components of the image processing apparatus, the imaging system including the image processing apparatus, and the image processing method shown in the above embodiment are not necessarily essential, and may be appropriately selected as long as they do not depart from the scope of the present invention. Is possible.

  For example, in the above embodiment, an image processing apparatus (including a PC) provided separately from a camera unit including a plurality of cameras performs imaging condition setting and image composition processing, and outputs a composite image. It can also be configured as a camera unit (imaging device) incorporating an image processing device. In some cases, it is possible to add a function similar to that of the above-described image processing apparatus to at least one of the plurality of cameras. The imaging condition setting function (imaging control unit function) in the above-described image processing apparatus may be configured to be executed by another apparatus (imaging condition setting apparatus) different from the image processing apparatus.

  An image processing apparatus, an imaging system including the image processing apparatus, and an image processing method according to the present invention appropriately display an area of a captured image serving as a reference in a panoramic composite image, and a boundary between areas of the combined captured images This is useful as an image processing apparatus capable of suppressing unnatural standing and performing image composition processing on a plurality of captured images to output a panorama composite image, an imaging system including the image processing apparatus, and an image processing method.

DESCRIPTION OF SYMBOLS 1 Imaging system 2 Camera unit 3 Image processing apparatus 4 Display apparatus 12 Image sensor part 13 Signal processing part 14 Camera control part 21 Stitching part 22 Imaging control part 34 Image composition part 41 Reference camera determination part 42 Imaging condition acquisition part 43 Imaging condition Setting unit 65 Reference camera decision buttons 71, 72 Boundary area CAM1-CAM7 Camera

Claims (10)

An image processing apparatus that is communicably connected to a plurality of cameras, performs image composition processing on a plurality of captured images acquired from the plurality of cameras, and outputs a panorama composite image,
A reference camera determination unit that determines one of the plurality of cameras as a reference camera;
An imaging condition acquisition unit that acquires imaging conditions related to exposure and white balance set in the reference camera;
An imaging condition setting unit that sends out an imaging condition setting command related to exposure and white balance based on the imaging conditions of the reference camera to other cameras other than the reference camera among the plurality of cameras;
After sending the setting command, images are captured based on the imaging conditions set in each of the plurality of cameras, and the panorama composite image is output by performing image composition processing on the plurality of captured images acquired from the plurality of cameras. An image synthesizing unit. The imaging condition setting unit, when the image synthesizing process is performed on the plurality of captured images by the image synthesizing unit, out of the plurality of captured images, brightness in a boundary region between two adjacent captured images, and A process for determining whether or not at least one of the color differences exceeds a preset threshold is performed for each adjacent captured image. When the threshold is exceeded in the determination process, two adjacent captured images are captured. For one setting change target camera among the cameras, a setting change command for the imaging condition of the setting change target camera is sent so as to reduce the difference exceeding the threshold value,
The image composition unit, after the determination process is executed for all the adjacent captured images by the imaging condition setting unit, if there is an adjacent captured image determined to exceed the threshold in the determination process, 2. The panorama composite image is output by performing an image composition process on the plurality of captured images including the captured images of the setting change target camera imaged based on a setting change command. The image processing apparatus described.   The image processing apparatus according to claim 1, wherein the imaging condition setting unit executes the determination process in ascending order of the plurality of cameras.   The image processing apparatus according to claim 1, wherein the imaging condition setting unit executes the determination process in order from a camera closer to the reference camera in the circumferential arrangement of the plurality of cameras.   The reference camera determination unit acquires information on the luminance of the captured images captured by the plurality of cameras when determining the reference camera, and images a captured image having a median luminance among the plurality of luminances. The image processing apparatus according to claim 1, wherein a camera is determined as the reference camera.   The reference camera determination unit obtains information on the luminance of the captured images captured by the plurality of cameras when determining the reference camera, and captures a captured image having a brightness closest to the average value of the plurality of luminances. The image processing apparatus according to claim 1, wherein a camera is determined as the reference camera.   The image processing apparatus according to claim 1, wherein the reference camera determination unit determines one camera selected by a user from the plurality of cameras as the reference camera.   The reference camera determining unit obtains an image recognition result of a predetermined imaging target in captured images respectively captured by the plurality of cameras when determining the reference camera, and determines the reference camera based on the image recognition result. The image processing apparatus according to claim 1.   An imaging system comprising the image processing apparatus according to claim 1 and the plurality of cameras. In an image processing apparatus communicably connected to a plurality of cameras, an image processing method for outputting a panorama composite image by performing an image composition process on a plurality of captured images acquired from the plurality of cameras,
A reference camera determining step of determining one of the plurality of cameras as a reference camera;
An imaging condition acquisition step of acquiring imaging conditions related to exposure and white balance set in the reference camera;
An imaging condition setting step for sending an imaging condition setting command related to exposure and white balance to the other cameras other than the reference camera among the plurality of cameras based on the imaging conditions of the reference camera;
After sending the setting command, images are captured based on the imaging conditions set in each of the plurality of cameras, and the panorama composite image is output by performing image composition processing on the plurality of captured images acquired from the plurality of cameras. An image synthesizing step.