JP2018142963A - Image processing system, image processing device, and image processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate a proper composite image even in a state in which an image of short-distance view and an image of long-distance view appear in an image region at a border part corresponding to an overlap part of imaging areas of a plurality of cameras.SOLUTION: The present invention relates to an image processing system comprising: a plurality of cameras arranged such that imaging areas of adjacent cameras overlap with each other; and an image processing device which performs image composition processing on a plurality of picked-up images acquired by the plurality of cameras and outputs a panoramic composite image to a display part. The image processing device composes an image of picked-up images of the adjacent cameras based upon images of long-distance view appearing in an image region at a border part corresponding to overlap parts of image pickup areas, and also composes an image of the picked-up images of the adjacent cameras by deforming an image of the image region where an image of short-distance view appears when the image of short-distance view and an image of long-distance view appear in the image region at the border part.SELECTED DRAWING: Figure 3

Description

  The present invention performs image composition processing on a plurality of captured images acquired from a plurality of cameras arranged so that the imaging areas partially overlap between adjacent cameras, and outputs a panorama composite image to a display unit The present invention relates to a system, an image processing apparatus, and an image processing method.

  A wide-angle image that cannot be obtained with a single camera can be generated seamlessly by performing a so-called stitching process that combines images captured by multiple cameras to generate a single composite image (panoramic image). can do. In such stitching processing, two adjacent cameras are arranged so that the respective imaging areas partially overlap, and the image areas at the boundary portions corresponding to the overlapping portions of the imaging areas of the two cameras are overlapped. In addition, the image is synthesized by appropriately trimming.

  On the other hand, if there is a subject whose distance from the camera is significantly different in the overlapping area of the imaging areas of the two cameras, that is, if there is a subject that is a distant view and a subject that is a close-up view, between the captured images of the two cameras In this case, the positional relationship between the image of the subject in the distant view and the image of the subject in the close shadow is shifted, so-called parallax occurs, and the close-up image appears twice in the composite image, or one of the close-up images. The trouble that a part disappears occurs. Thus, in stitching, parallax correction is performed to suppress image defects caused by parallax.

  As for such parallax correction, conventionally, the positional relationship between the images of the subjects appearing in the captured images of the two cameras is obtained by block matching based on edges and feature amounts, and the image is deformed based on this information. A technique for performing parallax correction is known (see Patent Document 1). In particular, in this technique, a stitching point that defines the degree of deformation of an image at the time of parallax correction is changed for each frame, and an appropriate composite image is generated for each frame.

  Further, a bent stitching boundary is set with respect to the two images so as to avoid an image region in which a near-field image such as a person appears, that is, an image region having a high cost function, and 2 along the stitching boundary. A technique for synthesizing images after performing trimming to cut out two images is known (see Patent Document 2). In particular, in this technology, the stitching boundary is optimized in consideration of the continuity of the frame, and this enables the setting of a stable stitching boundary in terms of time. Therefore, it is possible to avoid an unnatural movement in the image of the subject appearing in the image.

JP 2010-50842 A Japanese Patent No. 5225313

  The present invention has been devised to solve such problems of the prior art, and its main purpose is to arrange a plurality of cameras so that the imaging areas partially overlap between adjacent cameras. When a plurality of captured images acquired from a plurality of cameras are combined to output a combined image, a near-shadow image and a distant view image are displayed in the boundary image region corresponding to the overlapping portion of the imaging area. An object of the present invention is to provide an image processing system, an image processing apparatus, and an image processing method configured to generate an appropriate composite image even when it appears.

  An image processing system according to the present invention performs panorama processing by performing image synthesis processing on a plurality of cameras arranged so that imaging areas partially overlap between adjacent cameras and a plurality of captured images acquired from the plurality of cameras. An image processing system comprising: an image processing device that outputs a composite image to a display unit, wherein the image processing device is based on an image of a distant view that appears in an image area at a boundary corresponding to an overlapping portion of the imaging area. In the state where the captured images of the adjacent cameras are combined and a near-shadow image and the distant view image appear in the image area of the boundary portion, the near-shadow image is obtained for each of the captured images of the adjacent cameras. The image in the image area where the image appears is deformed to compose the image.

  In addition, the image processing apparatus of the present invention displays a panoramic composite image by performing image composition processing on a plurality of captured images acquired from a plurality of cameras arranged so that the imaging areas partially overlap between adjacent cameras. An image processing apparatus that outputs to the image processing unit, and combines the captured images of the adjacent cameras based on a distant view image that appears in the image area of the boundary corresponding to the overlapping portion of the imaging area, and the image of the boundary In a state where a near-shadow image and a distant view image appear in a region, an image in the image region where the near-shadow image appears is deformed and combined with each of the captured images of the adjacent cameras.

  Further, the image processing method of the present invention displays a panorama composite image by performing image composition processing on a plurality of captured images acquired from a plurality of cameras arranged so that the imaging areas partially overlap between adjacent cameras. An image processing method for outputting to an image capturing section, wherein the captured image of the adjacent camera is synthesized with reference to a distant view image appearing in an image area of the boundary corresponding to the overlapping portion of the image capturing area, and the image of the boundary In a state where a near-shadow image and a distant view image appear in a region, an image in the image region where the near-shadow image appears is deformed and combined with each of the captured images of the adjacent cameras.

  According to the present invention, a plurality of cameras are arranged so that imaging areas partially overlap between adjacent cameras, and a composite image is output by performing image composition on a plurality of captured images acquired from the plurality of cameras. In this case, an appropriate composite image can be generated even in a state in which a near-shadow image and a distant view image appear in the image area at the boundary portion corresponding to the overlapping portion of the imaging area.

1 is an overall configuration diagram showing an image processing system according to a first embodiment. Explanatory drawing which shows the synthesized image produced | generated by the image processing apparatus 3, and displayed on the display apparatus 4 Explanatory drawing which shows typically the state of the image when not implementing parallax correction, and when parallax correction is implemented Explanatory drawing which shows the actual state of the image when parallax correction is not performed and when parallax correction is performed Functional block diagram showing a schematic configuration of the image processing apparatus 3 Explanatory drawing explaining the outline | summary of the process performed by the image processing apparatus 3 Explanatory drawing which shows the camera unit 21 which concerns on 2nd Embodiment. Functional block diagram showing a schematic configuration of an image processing apparatus 23 according to the second embodiment. Overall configuration diagram showing an image processing system according to a third embodiment Functional block diagram showing a schematic configuration of the PC 31 shown in FIG. Explanatory drawing which shows the screen displayed on the display apparatus 4 shown in FIG.

  1st invention made | formed in order to solve the said subject is with respect to the some captured image acquired from the several camera arrange | positioned so that an imaging area may partially overlap between adjacent cameras, and the said several camera An image processing system that performs an image synthesis process and outputs a panorama composite image to a display unit, wherein the image processing apparatus applies an image area to a boundary portion corresponding to an overlapping portion of the imaging area. The captured image of the adjacent camera is synthesized based on the image of the distant view that appears, and in the state where the near-shadow image and the distant view image appear in the image area of the boundary portion, On the other hand, the image in the image area in which the near-shadow image appears is deformed to compose the image.

  According to this, when a plurality of cameras are arranged so that imaging areas partially overlap between adjacent cameras, and a composite image is output by performing image composition on a plurality of captured images acquired from the plurality of cameras. An appropriate composite image can be generated even in the state in which a near-shadow image and a distant view image appear in the image area at the boundary corresponding to the overlapping portion of the imaging area.

  According to a second aspect of the present invention, in the state in which the near-shadow image and the distant view image appear in the image area of the boundary portion, the image processing apparatus applies the near-shadow to each of the captured images of the adjacent cameras. The image in the image area where the image appears is deformed so as to be shifted in the horizontal direction, and the image is synthesized.

  According to a third aspect of the present invention, in the state in which the near-shadow image and the distant view image appear in the boundary image area, the image processing apparatus has a close-shadow image and the distant view image in the boundary image area. So that the images in the image areas where the near-shadow images appear are deformed with respect to each of the images taken by the adjacent cameras so as to match the positional relationship.

  According to a fourth aspect of the present invention, the image processing device synthesizes the captured images of the adjacent cameras based on the positional relationship of the distant view image obtained in a state where there is no near-shadow image in the boundary image area. In the state where the near-shadow image and the distant view image appear in the image area of the boundary portion, the image pickup by the adjacent camera is performed so as to correct the positional relationship between the distant view image and the close-view image. For each of the images, an image in an image region where the close-shadow image appears is deformed and image composition is performed.

  According to a fifth aspect of the present invention, the image processing apparatus obtains a positional relationship between the distant view images in the absence of the near-shadow image and combines the captured images of the adjacent cameras.

  According to a sixth aspect of the present invention, when the image processing device performs the image composition for each frame and displays the image on the display unit as a moving image, and the near-shadow image changes in the image area of the boundary, the adjacent When the near-shadow image moves across the overlapping part of each imaging area of the camera or when the near-shadow image disappears from the state in which the near-shadow image appears in the boundary image area, only the far-field image When the state changes to the state, the image in the image area at the boundary is deformed step by step.

  In addition, according to a seventh aspect of the present invention, image synthesis processing is performed on a plurality of captured images acquired from a plurality of cameras arranged so that imaging areas partially overlap between adjacent cameras, and a panoramic composite image is displayed on the display unit. An image processing device for outputting, wherein a captured image of the adjacent camera is synthesized with reference to an image of a distant view that appears in an image area of a boundary corresponding to an overlapping portion of the imaging area, and the image area of the boundary is In a state in which a near-shadow image and a distant view image appear, an image region image in which the near-shadow image appears is deformed with respect to each of the captured images of the adjacent cameras to perform image composition.

  Further, according to an eighth aspect of the present invention, image synthesis processing is performed on a plurality of captured images acquired from a plurality of cameras arranged so that imaging areas are partially overlapped between adjacent cameras, and a panoramic composite image is displayed on the display unit. An image processing method for outputting, wherein a captured image of the adjacent camera is synthesized based on an image of a distant view that appears in an image area of a boundary corresponding to an overlapping portion of the imaging area, and the image area of the boundary is In a state in which a near-shadow image and a distant view image appear, an image region image in which the near-shadow image appears is deformed with respect to each of the captured images of the adjacent cameras to perform image composition.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

(First embodiment)
FIG. 1 is an overall configuration diagram showing an image processing system according to the first embodiment. FIG. 2 is an explanatory diagram showing a composite image generated by the image processing device 3 shown in FIG. 1 and displayed on the display device 4.

  As shown in FIG. 1, the image processing system includes a camera unit 2 having first to seventh cameras 1 a to 1 g, an image processing device 3, and a display device 4.

  The cameras 1a to 1g have a wide angle of view (for example, 120 °), and the first to sixth cameras 1a to 1f are spaced equidistantly in the circumferential direction so that the optical axes are substantially radial in the horizontal direction. The seventh camera 1g is arranged so that the optical axis is substantially upward in the vertical direction. The cameras 1a to 1g are arranged such that two adjacent cameras 1a to 1g partially overlap imaging areas.

  As shown in FIG. 2, the image processing apparatus 3 performs a process (stitching) for synthesizing the captured images captured by the first to seventh cameras 1 a to 1 g to generate one composite image (panoramic image). This is performed for each frame, and a composite image for each frame is output to the display device 4, and the composite image is displayed as a moving image on the display device 4.

  Next, parallax correction performed by the image processing apparatus 3 illustrated in FIG. 1 will be described. FIG. 3 is an explanatory diagram schematically illustrating the state of an image when parallax correction is not performed and when parallax correction is performed. FIG. 4 is an explanatory diagram illustrating an actual state of an image when parallax correction is not performed and when parallax correction is performed. Here, for convenience of explanation, an example in which images of two adjacent cameras 1a and 1b are processed is shown.

  FIG. 3A shows an imaging state by two adjacent cameras 1a and 1b. In the example shown in FIG. 3A, a person and a mountain as a background are simultaneously imaged by two cameras 1a and 1b.

  3 (B-1) and (B-2) show images captured by the two cameras 1a and 1b. As shown in FIGS. 3B-1 and 3B-2, images taken by the two cameras 1a, 1b are images of boundaries corresponding to overlapping portions of the imaging areas of the two cameras 1a, 1b. A distant view image representing a mountain and a close-up image representing a person appear in the region. Here, the distance from the cameras 1a and 1b differs greatly between a mountain that is a distant view and a person that is a close view, and in the two captured images, the positional relationship between the distant view image and the close view image is shifted.

  FIGS. 3C-1 and 3C-2 show synthesized images obtained by performing simple synthesis on the images taken by the two cameras 1a, 1b with reference to a distant view image. As described above, since the positional relationship between the distant view image and the close-up image is shifted in the two captured images, when the two captured images are simply combined on the basis of the distant view image, FIG. ), (C-2), there arises a problem that double images of near shadows appear in the composite image or a part of the close shadow images disappears.

  Here, the shift of the near-shadow image when the distant view image is used as a reference represents the parallax between the captured images of the two cameras 1a and 1b. The positional relationship between the images of the distant views that appear in the two cameras 1a and 1b can be obtained in advance in the absence of a close shadow, and image synthesis is performed based on this information. Accordingly, there is no parallax in the captured images of the two cameras 1a and 1b in a state where there is no near-shadow image, and when the near-shadow image appears, parallax occurs between the captured images of the two cameras 1a and 1b.

  In this way, in a state where a distant view image and a close-up image appear in the image area at the boundary between the picked-up images of the two cameras 1a and 1b, the close-up image appears in the composite image due to the parallax generated between the two picked-up images. Such as appearing double images. Therefore, parallax correction is necessary to eliminate this problem, and there are typically two types of correction methods for this parallax correction. FIG. 3C-3 shows the case of using the first correction method, and FIG. 3C-4 shows the case of using the second correction method.

  As shown in FIG. 3 (C-3), the first correction method uses two images so that the positional relationship between the distant view image and the close-up image matches between the captured images of the two cameras 1a and 1b. The image synthesis is performed on the captured images of the cameras 1a and 1b after performing parallax correction that deforms the image so that an image region in which a near-shadow image mainly appears is shifted in the horizontal direction.

  As shown in FIG. 3 (C-4), the second correction method sets a stitching boundary that is bent so as to avoid an image region in which a near-shadow image appears, and two cameras 1a along the stitching boundary. , 1b, the image is synthesized after trimming to cut out the captured image.

  FIG. 4A-1 shows a composite image by simple synthesis without performing parallax correction, and FIG. 4A-2 shows a composite image with parallax correction performed. FIGS. 4B-1 and 4B-2 show enlarged peripheral portions of the foreground image in the composite image shown in FIGS. 4A-1 and 4A-2, respectively.

  As shown in FIGS. 4 (A-1) and (B-1), in simple composition without parallax correction, a part of the near-shadow image disappears or the near-shadow image appears twice in the composite image. However, when parallax correction is performed as shown in FIGS. 4A-2 and 4B-2, a problem such as simple composition is improved and an appropriate image is generated. be able to.

  By performing parallax correction in this way, an appropriate image can be generated. In the present embodiment, the first correction method, that is, the image of the distant view and the near-shadow between the captured images of the two cameras 1a and 1b. In order to match the positional relationship with the image, parallax correction is performed on the captured images of the two cameras 1a and 1b so that the image is deformed so that an image region in which a near-shadow image mainly appears is shifted in the horizontal direction.

  Here, unlike a distant view image such as a mountain as a background, a near-shadow image such as a person may change greatly in a short time, and due to this change in the near-shadow image, the parallax between two captured images varies. When the state becomes significantly different in a short time, the amount of parallax correction that defines the degree of image deformation at the time of parallax correction changes abruptly, causing significant image changes between frames, and when displaying a composite image as a moving image Unnatural movement occurs in the image of the subject appearing in the image, giving the viewer a sense of incongruity.

  For example, when a person in the foreground moves across an overlapping portion of the imaging areas of the two cameras 1a and 1b, a near-shadow image appears in the image area at the boundary in the captured images of the two cameras 1a and 1b. The state changes from a close-up state to a state in which a close-up image disappears and only a distant view image appears. The distance of suddenly changes. Thus, when the depth of the image area at the boundary portion changes abruptly, the parallax correction amount also changes accordingly, and the image deformation at the time of parallax correction also becomes abrupt.

  In particular, for the person who sees the image, the near-shadow image is important, so if you focus on the near-shadow image and perform processing that prioritizes the near-shadow image so that there is no problem with the near-shadow image, In the state where the image appears, the distant view image is slightly different from the actual shape. For this reason, as in the case where a person crosses the overlapping part of the imaging areas of the two cameras 1a and 1b, the state in which the close-in image appears is changed to the state in which the close-in image disappears and only the distant view image appears. When the parallax correction amount changes abruptly, a large change appears in the distant view image, for example, a mountain image that does not change changes, giving the viewer a sense of incongruity.

  Therefore, in the present embodiment, the parallax correction by the first correction method, that is, the image is deformed so that the image region in which the near-shadow image mainly appears is shifted in the horizontal direction with respect to the captured images of the two cameras 1a and 1b. When performing the parallax correction to be performed, a process of applying a filter to the parallax correction amount obtained from the parallax between the captured images of the two cameras 1a and 1b to limit the temporal change amount with respect to the parallax correction amount. Make it happen. This suppresses rapid image deformation during parallax correction, makes image deformation due to parallax correction inconspicuous, and prevents the viewer from feeling uncomfortable when displaying the composite image as a moving image.

  Next, processing performed by the image processing apparatus 3 shown in FIG. 1 will be described. FIG. 5 is a functional block diagram illustrating a schematic configuration of the image processing apparatus 3. FIG. 6 is an explanatory diagram for explaining an overview of processing performed in the image processing apparatus 3.

  The image processing apparatus 3 includes a parallax correction amount calculation unit 11, a filter 12, a parallax correction amount accumulation unit 13, a panoramic image generation unit 14, a parallax correction unit 15, and an image composition unit 16.

  Here, two cameras 1a. Although only the part that processes the captured image 1b is shown, as shown in FIG. 1, in the present embodiment, seven cameras 1a to 1g are provided, and the parallax correction amount calculation unit 11 and the filter 12 are The panoramic image generation unit 14 and the parallax correction unit 15 are provided for each of the cameras 1a to 1g.

  The parallax correction amount calculation unit 11 performs a process of calculating a parallax correction amount that defines the degree of deformation of the image at the time of parallax correction for each frame. Specifically, as shown in FIG. 6, parallel processing (projection onto a cylinder), processing region cutout, and parallax calculation are performed. In the parallax calculation process, parallax (deviation amount) is calculated by block matching between two captured images. That is, the difference between the two captured images is calculated while shifting the two captured images little by little, and the parallax is obtained from the positional relationship in which the difference is the smallest. Here, when a near-shadow image appears in the captured image, since the near-shadow image usually occupies a large area, the near-shadow image greatly affects the block matching result, and the image obtained by block matching The shift is a shift of a near-shadow image.

  The filter 12 performs a process of limiting the temporal change amount with respect to the parallax correction amount output from the parallax correction amount calculation unit 11. In the present embodiment, the parallax correction amount output from the filter 12 is accumulated in the parallax correction amount accumulation unit 13, and the parallax correction amount of the current (current) frame is accumulated in the parallax correction amount accumulation unit 13 in the filter 12. A process of limiting the amount of change from the parallax correction amount of the immediately preceding (previous) frame to be within a predetermined allowable range is performed.

Specifically, the parallax correction amount P _t−1 ′ of the immediately previous frame is acquired from the parallax correction amount accumulation unit 13, and this is used as the parallax correction amount P _t of the current frame input from the parallax correction amount calculation unit 11. The amount of change (ΔP = P _t −P _t−1 ′) is obtained. Then, the amount of change ΔP is compared with limit values (minimum value and maximum value) −A, A that define the allowable range of the amount of change. If the amount of change ΔP is within the allowable range, the parallax correction amount When the parallax correction amount P _t input from the calculation unit 11 is output as it is and the change amount ΔP is out of the allowable range, the parallax correction amount P _t−1 ′ of the immediately preceding frame acquired from the parallax correction amount accumulation unit 13 is output. The limit value ± A is added to and output. The processing performed by the filter 12 can be _expressed by the following formula, and the parallax correction amount P _t ′ output from the filter 12 can be obtained by this formula.
P _t ′ = min (max (P _t −P _t−1 ′, −A), A) + P _t−1 ′

In this embodiment, the parallax correction amount is limited by the difference (change amount) between the parallax correction amount of the current frame and the parallax correction amount of the previous frame. The parallax correction amount may be limited by the ratio (change rate) of the parallax correction amounts of the frames. In this case, the processing performed by the filter 12 is expressed by the following formula. Here, B _min and B _max are limit values (minimum value and maximum value) that define the allowable range of the change rate.
P _t ′ = min (max (P _t / P _t−1 ′, B _min ), B _max ) × P _t−1 ′

  In addition, the parallax correction amount of the current frame may be calculated as an average value of the parallax correction amounts for a plurality of previous or previous frames.

  In the panorama image generation unit 14, panoramaization (projection onto a sphere) is performed on the captured images output from the two cameras 1 a and 1 b to generate two panorama images.

  The parallax correction unit 15 performs a process of performing parallax correction on the two panoramic images generated by the panoramic image generation unit 14 based on the parallax correction amount output from the filter 12 to generate two parallax correction images. Done.

  The image synthesis unit 16 performs image synthesis processing on the two parallax correction images generated by the parallax correction unit 15 to generate one synthesized image. The composite image generated by the image composition unit 16 is output to the display device 4, and the composite image is displayed on the display device 4.

  Each unit of the image processing apparatus 3 illustrated in FIG. 5 is realized by causing the CPU of the information processing apparatus to execute an image processing program. This program is introduced in advance into the information processing apparatus so as to be configured as a dedicated image processing apparatus, and is recorded in an appropriate program recording medium as an application program that runs on a general-purpose OS in a PC as the information processing apparatus. It may also be provided to the user via a network.

  As described above, in this embodiment, since the temporal change amount is limited with respect to the parallax correction amount acquired by the parallax correction amount calculation unit 11 in the filter 12, the parallax between the two captured images is greatly different. Even when a change occurs, the parallax correction amount does not change rapidly but changes stepwise. Thereby, the image deformation performed by the parallax correction unit 15 based on the parallax correction amount is suppressed to be small, for example, only one pixel is moved in one frame, and the image deformation is performed step by step. For this reason, when the composite image is displayed as a moving image, it is possible to avoid giving a sense of discomfort to the viewer due to an unnatural movement in the image of the subject appearing in the image.

  In particular, in the present embodiment, the parallax correction amount accumulation unit 13 accumulates the parallax correction amount output from the filter 12, and the filter 12 accumulates the parallax correction amount of the current frame in the parallax correction amount accumulation unit 13. The amount of change from the amount of parallax correction of the previous frame is limited so that it falls within a predetermined allowable range, and information on future frames is no longer necessary. Real-time processing that outputs in real time can be realized.

(Second Embodiment)
Next, a second embodiment will be described. FIG. 7 is an explanatory diagram showing the camera unit 21 according to the second embodiment. FIG. 8 is a functional block diagram illustrating a schematic configuration of the image processing apparatus 23 according to the second embodiment. Note that points not particularly mentioned here are the same as in the above embodiment.

  In the first embodiment, the parallax correction amount is calculated from images captured by the cameras 1a to 1g. However, in the second embodiment, as shown in FIG. 7, the camera unit 21 includes a depth sensor (distance detection device). 22a to 22f are provided, and the parallax correction amount is calculated based on the detection results of the depth sensors 22a to 22f.

  The depth sensors 22a to 22f are distance image sensors that detect the depth of an image, that is, a distance to a subject corresponding to a foreground image appearing in the image, using a TOF (Time-Of-Flight) method or the like. The depth sensors 22a to 22f are arranged between the cameras 1a to 1f so that the depth of the image area at the boundary corresponding to the overlapping portion of the imaging areas of the two adjacent cameras 1a to 1f can be detected. ing.

  As shown in FIG. 8, the image processing apparatus 23 includes a parallax correction amount calculation unit 24 as in the first embodiment. The detection result of the depth sensor 22 a is input to the parallax correction amount calculation unit 24. Then, the parallax correction amount calculation unit 24 performs a process of calculating the parallax correction amount based on the detection result of the depth sensor 22a. Here, there is a correlation between the distance to the subject and the parallax, and based on this correlation, the parallax correction amount can be calculated from the detection result of the depth sensor 22a.

(Third embodiment)
Next, a third embodiment will be described. FIG. 9 is an overall configuration diagram showing an image processing system according to the third embodiment. FIG. 10 is a functional block diagram showing a schematic configuration of the PC 31 shown in FIG. FIG. 11 is an explanatory diagram showing a screen displayed on the display device 4 shown in FIG. Note that points not particularly mentioned here are the same as in the above embodiment.

  As shown in FIG. 9, the image processing system according to the third embodiment includes a camera unit 2, a PC (information processing device) 31, a display device 4, and a mouse (operation input device) 32. Yes.

  As shown in FIG. 10, the PC 31 includes an image storage unit 41, a first image processing unit 42, a second image processing unit 43, an image abnormality detection unit 44, a correction area setting unit 45, and parallax correction. An amount setting unit 46 and an input / output control unit 47 are provided.

  The image storage unit 41 is provided for each of the cameras 1a and 1b and stores images captured by the cameras 1a and 1b.

  In the 1st image processing part 42 and the 2nd image processing part 43, the process which synthesize | combines the captured image of the cameras 1a and 1b and produces | generates a synthesized image is performed. The first image processing unit 42 adopts the first correction method for parallax correction, as in the first embodiment, and is mainly used for the near-shadow on the captured images of the two cameras 1a and 1b. Processing for synthesizing an image is performed after performing parallax correction for deforming the image so that the image region in which the image appears is shifted in the horizontal direction. The first image processing unit 42 has the same configuration as that of the image processing apparatus 3 (see FIG. 5) of the first embodiment. The second image processing unit 43 employs the second correction method for parallax correction, sets a stitching boundary that is bent so as to avoid an image region in which a near-shadow image appears, and sets the stitching boundary as the stitching boundary. A process of cutting out and synthesizing two images along is performed.

  The image abnormality detection unit 44 detects an abnormal image region in the composite image, that is, an image region in which a malfunction has occurred in which a close shadow image appears twice or a part of the close shadow image disappears. Is done. In the correction area setting unit 45, processing for setting a correction area to be subjected to parallax correction performed in the first image processing unit 42 is performed in accordance with a user input operation using the mouse 32. In the parallax correction amount setting unit 46, processing for setting the parallax correction amount used in the first image processing unit 42 is performed in accordance with a user input operation using the mouse 32. In the input / output control unit 47, the composite image generated by the first image processing unit 42 and the second image processing unit 43 is output and displayed on the display device 4, and a user input operation using the mouse 32 is performed. Processing for acquiring the corresponding operation input information is performed.

  In the present embodiment, there are an image abnormality warning mode, a parallax correction amount adjustment mode, and a stitching boundary designation mode. Each mode will be described below.

  First, the image abnormality warning mode will be described. In the image abnormality warning mode, an abnormal image area in the composite image is detected, and a warning regarding the abnormal image area is output to the display device 4. The user can recognize the abnormal image area by seeing this warning, and subsequently perform an operation related to parallax correction as shown in FIGS.

  At this time, in the PC 31, the first image processing unit 42 or the second image processing unit 43 performs simple synthesis without performing parallax correction on the captured image, and the image abnormality detection unit 44 performs abnormality in the composite image. In the input / output control unit 47, a warning image representing the abnormal image region is superimposed on the composite image on the screen of the display device 4 based on the detection result of the image abnormality detection unit 44. The displayed process is performed.

  Specifically, as illustrated in FIG. 11A, the composite image 51 output from the first image processing unit 42 or the second image processing unit 43 is displayed on the screen of the display device 4. On the composite image displayed on the screen of the display device 4, a warning image 52 representing an abnormal image region is displayed. The warning image 52 is displayed so as to surround an abnormal image area, that is, an image area in which a near-shadow image appears twice or a part of the near-shadow image disappears, Furthermore, coloring such as red is given and a transparent paint is given. Thereby, the user can easily identify an abnormal image region.

  In this way, in the image abnormality warning mode, a warning regarding an abnormal image region is performed, so that the user can easily recognize an abnormal image region in the composite image, that is, an image region to be subjected to parallax correction.

  Note that the image abnormality detection unit 44 may perform block matching between the two images output from the parallax correction unit 15 and detect an image region in which the difference is large as an abnormal image region in the composite image. .

  Next, the parallax correction amount adjustment mode will be described. In this parallax correction amount adjustment mode, parallax correction by the first correction method performed by the first image processing unit 42, that is, an image region in which a close-in image mainly appears is shifted in the horizontal direction with respect to the captured image. The user adjusts the parallax correction amount used for the parallax correction in the process of synthesizing the image after performing the parallax correction for deforming the image to the user. In this parallax correction amount adjustment mode, the user using the mouse 32 is adjusted. The parallax correction amount is set according to the input operation, and a composite image based on the parallax correction amount is displayed on the display device 4.

  At this time, in the PC 31, the input / output control unit 47 acquires input information corresponding to the operation of the mouse 32, and the correction region setting unit 45 sets the correction region based on the operation input information acquired by the input / output control unit 47. The setting process is performed, and the parallax correction amount setting unit 46 performs the process of setting the parallax correction amount based on the operation input information acquired by the input / output control unit 47, so that the parallax correction of the first image processing unit 42 is performed. In the unit 15, the parallax correction processing is performed based on the correction region and the parallax correction amount set by the correction region setting unit 45 and the parallax correction amount setting unit 46, respectively.

  Specifically, as illustrated in FIG. 11B, the composite image output from the first image processing unit 42 is displayed on the screen of the display device 4, and the composite image displayed on the screen of the display device 4. The user designates an abnormal image area with the mouse pointer 53. Then, the first image processing unit 42 performs parallax correction on the image area around the position specified by the mouse pointer 53. At this time, when the user operates the mouse wheel 33, the parallax correction amount is adjusted, and a composite image in which the parallax correction is performed with the parallax correction amount according to the user operation is displayed on the screen of the display device 4.

  Therefore, in the parallax correction amount adjustment mode, the user can adjust the parallax correction amount so that an appropriate composite image can be obtained while viewing the state of the composite image displayed on the screen of the display device 4.

  In this embodiment, the parallax correction amount is adjusted by the mouse wheel 33. However, in the information processing apparatus having a touch panel display such as a tablet terminal, the parallax correction amount is adjusted by a touch operation such as pinch-in and pinch-out. It is good to be able to adjust. Even when the mouse 32 is used, the parallax correction amount may be adjusted by another operation such as a drag operation.

  Next, the stitching boundary designation mode will be described. In this stitching boundary designation mode, parallax correction by the second correction method performed in the second image processing unit 43, that is, a stitching boundary that is bent so as to avoid an image region in which a near-shadow image appears, is set. In the process of cutting out and synthesizing two images along the stitching boundary, the user designates the stitching boundary. In this stitching boundary designation mode, the user performs an input operation using the mouse 32. The operation input information related to the stitching boundary is acquired, and the stitching boundary is set based on the operation input information.

  At this time, in the PC 31, the input / output control unit 47 acquires operation input information corresponding to the operation of the mouse 32, and the second image processing unit 43 acquires the operation input information acquired by the input / output control unit 47. A stitching boundary is set, and an image is cut out and combined along the stitching boundary.

  Specifically, as shown in FIG. 11C, the user designates a stitching boundary so as to avoid an abnormal image region by dragging the mouse 32. At this time, the locus of the mouse pointer 53 becomes a stitching boundary, and image composition processing is performed based on the stitching boundary.

  Here, the stitching boundary specified by the user may be used as it is as the actual stitching boundary, but the stitching based on the cost function is performed with reference to the stitching boundary specified by the user. You may make it optimize a boundary.

  In the optimization of the stitching boundary, the stitching boundary is set so as to avoid an image region where a near-field image such as a person appears, that is, an image region having a high cost function. In other words, the stitching boundary is set so as to pass through an area where a near-field image such as a person does not appear and the pixel value does not change greatly, that is, an image area with a low cost function. At this time, by setting the periphery of the stitching boundary specified by the user as an image region having a particularly low cost function, the peripheral portion of the stitching boundary specified by the user is preferentially set as the stitching boundary.

  As described above, in the stitching boundary designation mode, the user can designate the stitching boundary while looking at the state of the composite image displayed on the screen of the display device 4. In addition, when the stitching boundary is optimized with reference to the stitching boundary specified by the user, the optimization of the stitching boundary is facilitated, and an appropriate stitching boundary can be set efficiently. .

  In this embodiment, the stitching boundary is specified by dragging the mouse 32. However, in an information processing apparatus having a touch panel display such as a tablet terminal, stitching is performed by touching the screen with a finger. It is recommended that the boundary can be specified.

  As mentioned above, although this invention was demonstrated based on specific embodiment, these embodiment is an illustration to the last, Comprising: This invention is not limited by these embodiment. Further, all the components of the image processing apparatus according to the present invention shown in the above-described embodiments are not necessarily essential, and can be appropriately selected as long as they do not depart from the scope of the present invention.

  For example, in the above-described embodiment, an image processing apparatus (including a PC) provided separately from a camera unit including a plurality of cameras performs a parallax correction process and an image synthesis process to output a synthesized image. It can also be configured as a camera unit (imaging device) incorporating an image processing device.

  In the above embodiment, the distance image sensor (depth sensor) is used as the distance detection device for detecting the distance to the subject. However, the distance detection device according to the present invention is applied to an image of a foreground that appears in a captured image. The distance image sensor is not limited as long as it can detect the distance to the corresponding subject.

  The image processing system, the image processing apparatus, and the image processing method according to the present invention are suitable even in a state where a close-up image and a distant view image appear in an image area of a boundary portion corresponding to an overlapping portion of imaging areas of a plurality of cameras. It has the effect of generating a composite image, and displays a panorama composite image by performing image composition processing on multiple captured images acquired from multiple cameras that are arranged so that the imaging areas partially overlap between adjacent cameras This is useful as an image processing system, an image processing apparatus, an image processing method, etc.

DESCRIPTION OF SYMBOLS 1a-1g Camera 2 Camera unit 3 Image processing apparatus 4 Display apparatus 11 Parallax correction amount calculation part 12 Filter 13 Parallax correction amount storage part 14 Panorama image generation part 15 Parallax correction part 16 Image composition part 21 Camera unit 21a Depth sensor (distance detection apparatus) )
23 Image processing device 24 Parallax correction amount calculation unit 32 Mouse (operation input device)
33 Mouse wheel 52 Warning image

Claims (8)

A plurality of cameras arranged such that the imaging areas partially overlap between adjacent cameras;
An image processing system comprising: an image processing device that performs image composition processing on a plurality of captured images acquired from the plurality of cameras and outputs a panorama composite image to a display unit;
The image processing apparatus includes:
Synthesize the captured image of the adjacent camera based on a distant view image that appears in the absence of a near-shadow image in the image area of the boundary corresponding to the overlapping portion of the imaging area,
When the near-shadow image appears in the image area of the boundary where the distant view image appears in the absence of the near-shadow image, an abnormal image area on the simple composite image of the image captured by the adjacent camera is detected, Display a warning about the abnormal image area;
Image processing system. The image processing system according to claim 1,
The image processing apparatus may be configured to use the abnormal image so as to surround an image region in which the near-shadow image appears twice or a part of the near-shadow image disappears as a warning regarding the abnormal image region. Display a warning image that represents the area,
Image processing system. The image processing system according to claim 1 or 2,
The image processing apparatus further includes an input unit for inputting a user designation for the simple composite image,
When the abnormal image area is specified by the input unit, the image of the image area in which the near shadow image appears is deformed so as to be shifted in the horizontal direction with respect to each of the captured images of the adjacent cameras. To
Image processing system. The image processing system according to claim 3,
The image processing apparatus obtains the positional relationship of the distant view image in advance in the absence of the near-shadow image, and combines the captured images of the adjacent cameras;
Image processing system. The image processing system according to claim 1 or 2,
The image processing apparatus further includes an input unit for inputting a user designation for the simple composite image,
When the abnormal image region and the stitching boundary are specified by the input unit, the captured images of the adjacent cameras are synthesized based on the stitching boundary.
Image processing system. The image processing system according to claim 5,
The image processing device sets the stitching boundary based on the stitching boundary specified by the input unit so as to pass through an image region where the near-shadow image does not appear.
Image processing system. An image processing apparatus that performs image synthesis processing on a plurality of captured images acquired from a plurality of cameras arranged so that imaging areas partially overlap between adjacent cameras and outputs a panoramic composite image to a display unit. ,
Synthesize the captured image of the adjacent camera based on a distant view image that appears in the absence of a near-shadow image in the image area of the boundary corresponding to the overlapping portion of the imaging area,
When the near-shadow image appears in the image area of the boundary where the distant view image appears in the absence of the near-shadow image, an abnormal image area on the simple composite image of the image captured by the adjacent camera is detected, Display a warning about the abnormal image area;
Image processing device. An image processing method for performing an image synthesis process on a plurality of captured images acquired from a plurality of cameras arranged so that imaging areas partially overlap between adjacent cameras and outputting a panorama composite image to a display unit. ,
Synthesize the captured image of the adjacent camera based on a distant view image that appears in the absence of a near-shadow image in the image area of the boundary corresponding to the overlapping portion of the imaging area,
When the near-shadow image appears in the image area of the boundary where the distant view image appears in the absence of the near-shadow image, an abnormal image area on the simple composite image of the image captured by the adjacent camera is detected, Display a warning about the abnormal image area;
Image processing method.

Images