JP2018198463A - Image processing device, image processing method, and computer program - Google Patents

Abstract

To suppress deterioration in quality of a composite image generated by composing images photographed by a plurality of cameras with a background image.SOLUTION: An image processing device comprises: a composite image generation part that generates a composite image by superposing a first partial image and a second partial image on a first region and a second region sandwiching a predetermined region set in a background image that is an image photographed at a different position and different time from those of at least any one of a first input image and a second input image; and an image processing unit that compensates the predetermined region between the first partial image and the second partial image. The image processing unit compensates a first range that is a region closer to the first partial image by using the first input image, compensates a second range that is a region closer to the second partial image by using the second input image, and compensates the vicinity of a center of the predetermined region by using any one of the first input image and the second input image.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a technique for processing an image photographed by a plurality of photographing devices.

  In recent years, cameras capable of capturing 360-degree panoramic images (hereinafter referred to as “global cameras”) have begun to spread. A panoramic image captured by the omnidirectional camera (hereinafter referred to as “spherical image”) can be captured by installing the omnidirectional camera at a desired viewpoint position. However, a spherical camera cannot be installed in a competition court such as a soccer court or a basketball court because it interferes with the competitors during the competition. For this reason, it is not possible to take an omnidirectional image during competition at a desired viewpoint position in the competition court.

  Therefore, a virtual viewpoint, which is a virtual viewpoint, is set at a place where the omnidirectional camera cannot be installed, and an omnidirectional image as if taken with the omnidirectional camera at this virtual viewpoint is displayed on the court. There has been proposed a technique obtained by combining images taken by a plurality of cameras installed on the outside (see, for example, Non-Patent Document 1). In the following description, the omnidirectional image at the virtual viewpoint is referred to as a virtual omnidirectional image.

A specific example of a system that obtains a virtual omnidirectional image by combining images captured by a plurality of cameras will be described.
FIG. 5 is a diagram showing a system for obtaining a virtual omnidirectional image in a conventional system. As shown in FIG. 5, the image processing system 1 includes an omnidirectional camera 2 and a plurality of cameras 3-1, 3-2, 3-3,... (N is an integer of 4 or more), an image processing device 4, and a display device 5. When the virtual viewpoint 11 is set in the competition court 10, the image processing system 1 generates a virtual omnidirectional image at the virtual viewpoint 11 by combining the images photographed by the camera group 3 installed outside the competition court 10. obtain.

  The omnidirectional camera 2 is a camera that captures an omnidirectional image. The omnidirectional camera 2 is installed at the position of the virtual viewpoint 11 in the competition court 10 at a timing before the competition is performed. The omnidirectional camera 2 captures an image (hereinafter referred to as “background image”) that is the background of the virtual omnidirectional image from the position of the virtual viewpoint 11. The background image captured by the omnidirectional camera 2 is input to the image processing device 4 and accumulated. As described above, the image processing apparatus 4 stores the background image in advance.

  A camera group 3 is installed around the competition court 10. The camera group 3 is installed around the competition court 10 so as to have an angle of view including the virtual viewpoint 11. The camera group 3 captures an area including the virtual viewpoint 11. The image processing device 4 performs image processing on the image captured by each camera group 3, and generates a virtual omnidirectional image by combining the image after image processing with the background image. The display device 5 is an image display device such as a liquid crystal display, an organic EL (Electro Luminescence) display, or a CRT (Cathode Ray Tube) display. The display device 5 displays the virtual omnidirectional image generated by the image processing device 4.

Next, a specific example of image processing in the image processing system 1 will be described with reference to FIG.
FIG. 6 is a diagram for explaining the flow of image processing in the image processing system 1. FIG. 6A is a diagram illustrating a specific example of the background image 20. In the background image 20, a subject in all directions (360 degrees) with the virtual viewpoint 11 as the center is photographed. Since the background image 20 is an image shot in a state where no person is present in the competition court 10, no person is photographed in the competition court 10.

  FIG. 6B is a diagram illustrating images captured by the cameras 3-1, 3-2, and 3-3. FIG. 6B shows an image 21 taken from the left by the camera 3-1, an image 22 taken by the camera 3-2, and an image 23 taken by the camera 3-3. . The image processing apparatus 4 extracts regions 211, 221, and 231 including the virtual viewpoint 11 from each of the images 21 to 23. The image processing apparatus 4 generates partial images 211a, 221a, and 231a that can be combined with the background image 20 by performing image processing on the extracted images of the regions 211, 221, and 231.

  The image processing device 4 generates the virtual omnidirectional image 24 by combining the partial images 211a, 221a, and 231a with the background image 20. FIG. 6C is a diagram illustrating an example of the virtual omnidirectional image 24 generated by the image processing device 4. As shown in FIG. 9C, partial images 211 a, 221 a, and 231 a are combined with a predetermined area of the virtual omnidirectional image 24. Therefore, an image in which an object (for example, a person) is photographed on the competition court 10 is generated as the virtual omnidirectional image 24.

  In the conventional image processing system 1, the optical center of the camera group 3 used for composition and the optical center of the omnidirectional camera assumed in the virtual viewpoint 11 are different. Therefore, the synthesized virtual omnidirectional image 24 includes a geometrically incorrect image. In order to prevent this, the image processing apparatus 4 performs image processing on the partial images 211 a, 221 a, and 231 a so that consistency is maintained at one point indicating the distance from the virtual viewpoint 11, and is synthesized with the background image 20. There is a need. However, when synthesizing a partial image of an object (for example, a person) that does not exist at a depth where consistency is maintained but exists at another depth, the consistency of depth cannot be maintained by image processing. . Such an object whose depth is inconsistent causes a phenomenon that the virtual omnidirectional image 24 becomes a duplicated image (multiple image) or disappears.

  Hereinafter, with reference to FIG. 7, a phenomenon in which an image of an object is duplicated or disappears in the virtual omnidirectional image 24 will be described. FIG. 7 is a diagram for explaining a problem in the image processing system 1. In FIG. 7, an imaging range 41 indicates the imaging range of the area 211 shown in FIG. 6B in the imaging range of the camera 3-1. The shooting range 42 indicates the shooting range of the area 221 shown in FIG. 6B in the shooting range of the camera 3-2. The shooting range 43 indicates the shooting range of the area 231 shown in FIG. 6B in the shooting range of the camera 3-3. In addition, there are three objects (for example, persons) 49 to 51 having different distances (depths) from the virtual viewpoint 11.

  In the depth 46 which shows the 1st distance from the virtual viewpoint 11 shown with the broken line in FIG. 7, each imaging range 41-43 is located in a line without overlapping. The subject 49 positioned at the depth 46 is a subject that is consistent in the depth without the image being altered or lost. In the depth 47 indicating the second distance from the virtual viewpoint 11, the shooting ranges 41 to 43 overlap as shown by the horizontal line portion 44. The subject 50 positioned at the depth 47 is a subject whose depth is not consistent because the image is duplicated. The depth 48 indicating the third distance from the virtual viewpoint 11 is vacant as indicated by the hatched portion 45 between the imaging ranges 41 to 43. The subject 51 positioned at the depth 48 is a subject whose depth is not consistent because part of the image is lost.

Kosuke Takahashi, 3 others, “Study on virtual spherical image synthesis using multiple camera images”, The Institute of Electronics, Information and Communication Engineers, IEICE Technical Report, June 2015, vol.115, no.76, p. .43-48

  There is a high possibility that a region where an object is present in the virtual omnidirectional image is a region that is watched by a person (hereinafter referred to as “viewing region”). For this reason, if multiple images or disappearance occurs in an object existing in the viewing area, there is a problem that the quality of the virtual omnidirectional image is degraded. Such a problem is not limited to a virtual omnidirectional image, but is a problem common to all synthesized images generated by synthesizing images taken by a plurality of cameras with a background image.

  In view of the above circumstances, an object of the present invention is to provide a technique for suppressing deterioration in quality of a composite image generated by combining images taken by a plurality of cameras with a background image.

  According to one embodiment of the present invention, a predetermined region set in a background image that is an image photographed at a different position and / or at a different time from at least one of the first input image and the second input image. A composite image is generated by superimposing the first partial image and the second partial image generated from the first input image and the second input image on the sandwiched first region and second region. By using the composite image generation unit, the first input image, and the second input image as reference images used for complementing the predetermined region, the superimposed first partial image and the second image An image processing unit that complements the predetermined area between the first partial image and the second partial image, and the first partial image and the second partial image are different from each other in the same subject located at a predetermined depth. A captured image, said One partial image and the second partial image include a part of the subject, and a part of the subject included in the first partial image is a part of the subject included in the second partial image. The predetermined area is different from the first area and the second area when the first partial image and the second partial image are superimposed on the first area and the second area, respectively. A partial area of the subject that is not included in one partial image and not included in the second partial image, and the image processing unit includes the first part of the predetermined area. The first range which is an area closer to the image is complemented by using the first input image, and the second range which is an area closer to the second partial image in the predetermined area. For the above by using the second input image Complement the 2 range, the vicinity of the center of the predetermined area is an image processing apparatus which complemented by the use of any of the first input image and the second input image.

  According to one embodiment of the present invention, a predetermined region set in a background image that is an image photographed at a different position and / or at a different time from at least one of the first input image and the second input image. A composite image is generated by superimposing the first partial image and the second partial image generated from the first input image and the second input image on the sandwiched first region and second region. By using the composite image generation unit, the first input image, and the second input image as reference images used for complementing the predetermined region, the superimposed first partial image and the second image An image processing unit that complements the predetermined area between the first partial image and the second partial image, and the first partial image and the second partial image are different from each other in the same subject located at a predetermined depth. A captured image, said One partial image and the second partial image include a part of the subject, and a part of the subject included in the first partial image is a part of the subject included in the second partial image. The predetermined area is different from the first area and the second area when the first partial image and the second partial image are superimposed on the first area and the second area, respectively. A partial area of the subject that is not included in one partial image and not included in the second partial image, and the image processing unit is an image after the magnification change of the first input image And the image after the magnification change of the second input image are used as the reference image to complement the predetermined area.

  One aspect of the present invention is the above-described image processing device, wherein the background image is an image captured at a different position and / or at a different time from each of the first input image and the second input image. is there.

  According to one embodiment of the present invention, a predetermined region set in a background image that is an image photographed at a different position and / or at a different time from at least one of the first input image and the second input image. A composite image is generated by superimposing the first partial image and the second partial image generated from the first input image and the second input image on the sandwiched first region and second region. By using the composite image generation step and the first input image and the second input image as a reference image used for complementing the predetermined region, the superimposed first partial image and the second image An image processing step for complementing the predetermined area between the first partial image and the second partial image, wherein the first subject image and the second partial image are located at different positions of the same subject located at a predetermined depth. Images taken from The first partial image and the second partial image include a part of the subject, and the part of the subject included in the first partial image is included in the second partial image. The predetermined area is different from a part of the subject, and the predetermined area is obtained when the first partial image and the second partial image are superimposed on the first area and the second area, respectively. Including a partial area of the subject that is not included in the first partial image and that is not included in the second partial image, and in the image processing step, For the first range that is a region closer to the first partial image, the first range is complemented by using the first input image, and the region closer to the second partial image in the predetermined region. For the second range, the second input image Complementing the second range by using, for near the center of the predetermined area is an image processing method for complementing by using either of the first input image and the second input image.

  According to one embodiment of the present invention, a predetermined region set in a background image that is an image photographed at a different position and / or at a different time from at least one of the first input image and the second input image. A composite image is generated by superimposing the first partial image and the second partial image generated from the first input image and the second input image on the sandwiched first region and second region. By using the composite image generation step and the first input image and the second input image as a reference image used for complementing the predetermined region, the superimposed first partial image and the second image An image processing step for complementing the predetermined area between the first partial image and the second partial image, wherein the first subject image and the second partial image are located at different positions of the same subject located at a predetermined depth. Images taken from The first partial image and the second partial image include a part of the subject, and the part of the subject included in the first partial image is included in the second partial image. The predetermined area is different from a part of the subject, and the predetermined area is obtained when the first partial image and the second partial image are superimposed on the first area and the second area, respectively. Includes a partial area of the subject that is not included in the first partial image and that is not included in the second partial image. In the image processing step, the magnification of the first input image In the image processing method, the predetermined region is complemented by using the image after the change and the image after the magnification change of the second input image as the reference image.

  One embodiment of the present invention is a computer program for causing a computer to function as the above-described image processing apparatus.

  According to the present invention, it is possible to suppress deterioration in quality of a synthesized image generated by synthesizing images captured by a plurality of cameras with a background image.

1 is a diagram illustrating a system configuration of an image processing system 100 according to the present invention. It is a figure which shows the specific example of a synthetic | combination information table. It is a figure for demonstrating the process of the image process part. 4 is a flowchart showing a processing flow of the image processing apparatus 80. It is a figure which shows the system for obtaining a virtual omnidirectional image in a conventional system. FIG. 3 is a diagram for explaining a flow of image processing in the image processing system 1; 2 is a diagram for explaining a problem in the image processing system 1. FIG.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
FIG. 1 is a diagram showing a system configuration of an image processing system 100 according to the present invention.
The image processing system 100 includes an omnidirectional camera 60, a plurality of cameras 70-1 to 70-M (M is an integer of 2 or more), and an image processing device 80. In the following description, the cameras 70-1 to 70-M will be referred to as cameras 70 unless otherwise distinguished.

  The omnidirectional camera 60 is installed at the position of the virtual viewpoint 82 in the shooting target area 81. The imaging target area 81 is, for example, a competition court such as a soccer court or a basket court. The virtual viewpoint 82 is a viewpoint virtually set in a predetermined area (in this embodiment, the imaging target area 81). The omnidirectional camera 60 captures an omnidirectional image at the position of the virtual viewpoint 82. The omnidirectional image in the present embodiment includes the entire imaging target region 81 with the virtual viewpoint 82 as the center. The processing by the omnidirectional camera 60 is performed before the processing by the image processing device 80 is started. The omnidirectional camera 60 outputs the captured omnidirectional image to the image processing apparatus 80 as a background image.

  M cameras 70-1, 70-2,..., 70 -M are cameras that are provided outside the imaging target area 81 and shoot an input image as a moving image, and shoot an area including the virtual viewpoint 82. To do. As shown in FIG. 1, a light beam 71 passing over the position of the virtual viewpoint 82 is input to the camera 70-1, and a light beam 72 passing over the position of the virtual viewpoint 82 is input to the camera 70-2. Hereinafter, the light beam input to the camera 70 is referred to as a real light beam. Although not shown in FIG. 1, the camera 70 is installed around the shooting target area 81. That is, the camera 70 is installed so as to surround the photographing target area 81 so that the angle of view includes the virtual viewpoint 82. In FIG. 1, M is an integer equal to or greater than 2. If a virtual omnidirectional image having the same image quality is to be obtained, the value M increases as the shooting target area 81 increases. Further, if the size of the imaging target area 81 is the same, the larger the value of M, the larger the area of the synthesis area (the area where the images from the M cameras 70 are synthesized in the virtual omnidirectional image). Alternatively, if the size of the composite area is the same, the value becomes larger as the image quality of the virtual omnidirectional image that improves the image quality in the composite area is increased.

The image processing device 80 generates a virtual omnidirectional image based on the omnidirectional image captured by the omnidirectional camera 60 and an image captured by the camera 70 (hereinafter referred to as “input image”). .
Next, the functional configuration of the image processing apparatus 80 will be described. The image processing apparatus 80 includes a CPU (Central Processing Unit), a memory, an auxiliary storage device, and the like connected by a bus, and executes an image processing program. By executing the image processing program, the image processing apparatus 80 includes an input image storage unit 801, an object analysis unit 802, a synthesis information storage unit 803, a partial area extraction unit 804, a background image storage unit 805, an image synthesis unit 806, and an image processing unit. It functions as a device having 807. Note that all or part of the functions of the image processing apparatus 80 may be realized using hardware such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA). . The image processing program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. Further, the image processing program may be transmitted / received via a telecommunication line.

The input image storage unit 801 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The input image storage unit 801 stores the input images for each camera 70 in time series in association with the camera ID for identifying each camera 70. The input image includes shooting time and moving image data.
The object analysis unit 802 reads the input image stored in the input image storage unit 801 and uses it as an input. The object analysis unit 802 detects and outputs an object included in the input image that has been input. For example, the object analysis unit 802 detects an object from each of the input images at the same time of two adjacent cameras 70. Here, the object is a person, an object (for example, a ball) or the like that is not included in the background image but is included in the input image. A known image analysis technique is used as a method for analyzing and determining whether the object corresponds to “person”, “ball”, “object A”, or “object B”. For example, the following reference 1 is a document disclosing a technique for detecting a person by analyzing an image.
[Reference 1]: Satoshi Yamauchi and two others, “[Survey Paper] Human Detection by Statistical Learning Method”, IEICE Technical Report, vol.112, no.197, PRMU2012-43, pp. 113-126, September 2012

  The combined information storage unit 803 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The composite information storage unit 803 stores a composite information table. The composite information table is composed of records (hereinafter referred to as “composite information records”) representing information for superimposing an image generated from the input image on the background image (hereinafter referred to as “composite information”).

FIG. 2 is a diagram illustrating a specific example of the synthesis information table.
The composite information table has a plurality of composite information records. The composite information record has each value of camera ID, extraction area information, and conversion information. The value of the camera ID represents identification information for identifying the camera 70. For example, the camera 70 identified by the camera ID “C1” in FIG. 2 is the camera 70-1, and the camera 70 identified by the camera ID “CM” is the camera 70-M.

  The value of the extraction area information represents information regarding an area (hereinafter referred to as “extraction area”) extracted from an image (input image) photographed by the camera 70 identified by the camera ID of the same composite information record. When the extraction area information is a rectangle, a specific example of the extraction area information is a rectangle represented by an upper left coordinate, a width, and a height. The upper left coordinate represents the upper left coordinate in the input image of the extraction area. The width represents the width of the extraction area. The height represents the height of the extraction area. Note that the width and height are set to a range including the virtual viewpoint in the extraction region with the upper left coordinate of the extraction region as a reference. In the present embodiment, the region indicated by the extraction region information is an image of each extraction region extracted from each input image of an adjacent camera 70 (for example, the camera 70 identified by the camera IDs “C1” and “C2”). Hereinafter, when each partial image generated by performing conversion according to conversion information is superimposed on the background image, each partial image has a predetermined width. It is set so as to sandwich a region having a height. In addition, as a method of pinching, an area in which each partial image has a predetermined width and height may be pinched vertically or horizontally.

The value of the conversion information represents information for converting a partial area image extracted according to the extracted area information of the same composite information record into a partial image. The partial image is generated by performing conversion processing such as enlargement, reduction, rotation, and deformation on the partial region image according to the conversion information in order to superimpose the partial region image on the corresponding region of the background image without a sense of incongruity. Is done. This conversion information is, for example, an affine transformation matrix. Note that the affine transformation matrix includes information indicating a region in which the partial image is superimposed on the background image.
The affine transformation matrix is acquired in advance by the following method and stored in the composite information storage unit 803. For example, an image including a chess board photographed by the omnidirectional camera 60 installed at the virtual viewpoint 82 with a lattice-shaped chess board installed at a plurality of types of distances (depths) from the virtual viewpoint 82, and the camera 70 Compare the captured image with the chess board. Then, in both images, an affine transformation matrix for transforming the images so as to correspond to each grid of the captured chess board is obtained. In this way, an affine transformation matrix corresponding to the depth at which the chess board is installed is obtained.

In the example shown in FIG. 2, a plurality of composite information records are registered in the composite information table. In FIG. 2, the composite information record registered at the top of the composite information table has a camera ID value “C1”, an upper left coordinate value “(A, B)”, a width value “C”, The height value is “D” and the conversion information value is “A1 _i ”. That is, an area represented by upper left coordinates (A, B), width C, and height D is extracted from the input image of the camera 70-1 identified by the camera ID “C1”. It is shown that the conversion process of A1 _i ″ is performed.

Returning to FIG. 1, the description of the image processing apparatus 80 will be continued.
The partial area extraction unit 804 receives the input image output from the object analysis unit 802 and the information in the synthesis information table. The partial area extraction unit 804 generates and outputs a partial area image by extracting the partial area from the input image based on the information of the input synthesis information table.
The background image storage unit 805 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The background image storage unit 805 stores the omnidirectional image captured by the omnidirectional camera 60 as a background image.
The image composition unit 806 inputs the composition information table stored in the composition information storage unit 803, the background image stored in the background image storage unit 805, and the partial region image generated by the partial region extraction unit 804. And The image synthesis unit 806 generates a partial image by performing a conversion process on the partial region image based on the affine transformation matrix of the conversion information included in the synthesis information table. The image composition unit 806 generates and outputs a virtual omnidirectional image by superimposing the generated partial image on the background image based on the affine transformation matrix. More specifically, the image composition unit 806 superimposes the partial region image on the background image by replacing the pixel value of the region on the background image with the pixel value of the partial image, thereby superimposing the partial region image on the background image. Generate a sphere image.
The image processing unit 807 receives the virtual omnidirectional image generated by the image composition unit 806 and the input image stored in the input image storage unit 801 as inputs. In the input virtual omnidirectional image, the image processing unit 807 sets a predetermined region (hereinafter referred to as “complementation target region”) between the partial images superimposed on the background image by the image composition unit 806. Are complemented using the input images that are the generation sources of the two partial images.

Hereinafter, specific processing of the image processing unit 807 will be described with reference to FIG. FIG. 3 is a diagram for explaining the processing of the image processing unit 807.
In FIG. 3, an input image 83-1 (first input image) is an image taken by the camera 70-1, and an input image 83-2 (second input image) is taken by the camera 70-2. It is an image. An area 84 in the input image 83-1 represents an extraction area extracted from the input image 83-1 based on the synthesis information table, and an area 85 in the input image 83-2 is derived from the input image 83-2 based on the synthesis information table. Represents the extraction area to be extracted. A partial region image generated by extracting the region represented by the extraction region 84 from the input image 83-1, and a partial region generated by extracting the region represented by the extraction region 85 from the input image 83-2. Partial images 86 (first partial images) and 87 (second partial images) are generated by performing conversion processing according to conversion information on each image. Thereafter, the image composition unit 806 superimposes the partial images 86 and 87 on a predetermined position on the background image 88 indicated by the affine transformation matrix (see the lower diagram in FIG. 3). Here, the region where the partial image 86 is superimposed corresponds to the first region, and the region where the partial image 87 is superimposed corresponds to the second region.

  As described above, the partial images 86 and 87 are superimposed with a predetermined area interposed therebetween. Therefore, a predetermined area 89 is generated between the partial images 86 and 87 superimposed on the background image 88. This predetermined area 89 is a complementation target area that the image processing unit 807 complements. The complement target area includes upper left coordinates and lower left coordinates of the partial image (part image 86 in FIG. 3) superimposed on the first area based on two partial images adjacent to the complement target area, and the second area. Represents a region surrounded by upper right coordinates and lower right coordinates of the partial image (partial image 87 in FIG. 3) superimposed on. As shown in FIG. 3, the partial image 86, the partial image 87, and the complement target area are rectangular, and all have the same height. Furthermore, the partial image 86 and the partial image 87 are arranged so that the bottoms are aligned. Since the partial image 86 and the partial image 87 are arranged so that the bottoms thereof are aligned, the partial image 86 and the partial image 87 sandwich the region to be complemented.

  The image processing unit 807 receives the input image 83-1 that is the generation source of the partial image 86, the input image 83-2 that is the generation source of the partial image 87, and the information on the complement target area as input. Using the images 83-1 and 83-2 as reference images, the complement target area is complemented, and an image complemented with the complement target area is output. Here, the reference image represents an input image used for complementing the complement target area. Specifically, first, the image processing unit 807 selects a small area (patch) including both the complement target area and the area other than the complement target area from the background image 88. As this method, there is a method of selecting patches that exist on the outline of the complementation target area and that have a strong edge strength of pixels in the area other than the complementation target area in the patch area. Next, the image processing unit 807 detects a similar patch from the reference image based on pixels other than the complement target area of the selected patch. Here, the similar patch represents a patch whose pixel value is similar to the pixel value of the pixel of the selected patch. Then, the image processing unit 807 superimposes the detected similar patch on the selected patch. That is, the image processing unit 807 replaces the selected pixel value of the patch with the pixel value of the similar patch. The image processing unit 807 executes the above processing until there is no complement target area.

  At this time, the image processing unit 807 refers to the input image 83-1 that is the generation source of the partial image 86 for the first range that is closer to the partial image 86 than the center of the complementary target region of the complementary target regions. The first range is complemented by using it as an image. The image processing unit 807 also uses the input image 83-2 that is the generation source of the partial image 87 as a reference image for the second range that is closer to the partial image 87 than the center of the complement target region in the complement target region. To supplement the second range. The image processing unit 807 refers to either the input image 83-1 or the input image 83-2 that is the generation source of each of the partial images 86 and 87 adjacent to the complement target region for the vicinity of the center of the complement target region. It complements by using as an image. The reason why the input image that is the generation source of each partial image is used as the reference image is that high-quality complementation is possible from a location close to the original image. In addition, when an object is not included in one partial image of two partial images adjacent to the complement target region, the image processing unit 807 uses an input image that is a generation source of the partial image including the object as a reference image. The complement target area may be complemented by

  In the above example, the patch-based method is described as an example of the complementing method (completion method), but the complementing method is not necessarily limited to this. Other methods may be used as the complementing method. For example, there is a method of complementing using the pixel at the corresponding position in the similar patch of the pixel in the complement target region, or a method of complementing using the weighted average pixel of the pixel at the corresponding position in the plurality of similar patches. is there.

FIG. 4 is a flowchart showing a process flow of the image processing apparatus 80.
The object analysis unit 802 reads the input images at the same time of the adjacent cameras 70 (two cameras 70) from the input image storage unit 801 (step S101). Next, the object analysis unit 802 receives each input image that has been read, and detects each input image object by analyzing each input image (step S102). The object analysis unit 802 determines whether an object is detected from any input image (step S103). When no object is detected from any input image (NO in step S103), the object analysis unit 802 reads the input image at the same time of the camera 70 in another adjacent combination from the input image storage unit 801 (step S104). ). Thereafter, the processing after step S102 is executed.

  On the other hand, when an object is detected from any input image (step S103—YES), the object analysis unit 802 outputs the input image to the partial region extraction unit 804. The partial region extraction unit 804 receives the input image output from the object analysis unit 802 and the composite information table stored in the composite information storage unit 803, and extracts the partial region indicated by the composite information table from each input image. By extracting, a partial area image for each input image is generated (step S105). For example, the partial region extraction unit 804 extracts the region represented by the extraction region information of the composite information record corresponding to the camera 70-1 in the composite information table from the input image of the camera 70-1, thereby A partial area image of the input image 70-1 is generated. The partial area extraction unit 804 outputs each partial area image of each generated input image to the image synthesis unit 806.

  The image composition unit 806 stores each partial region image of each input image output from the partial region extraction unit 804, a composite information table stored in the composite information storage unit 803, and a background image storage unit 805. Each partial image is generated by performing a conversion process on each generated partial area image based on the conversion information in the synthesis information table, and the background image is input to the background image. Superimpose on the corresponding position (step S106). For example, the image composition unit 806 performs a conversion process on the partial region image generated from the input image of the camera 70-1 according to the conversion information of the composite information record corresponding to the camera 70-1 in the composite information table. Thus, a partial image of the partial region image of the camera 70-1 is generated, and the generated partial image is superimposed on the background image based on information indicating a region in which the partial image is superimposed on the background image included in the conversion information. The image composition unit 806 outputs the background image on which the partial images are superimposed to the image processing unit 807.

The image processing unit 807 receives the background image on which the partial image is superimposed and the input image stored in the input image storage unit 801 as input, and each part superimposed on the background image stored in the input image storage unit 801. Using the input image that is the image generation source, the region to be complemented between the superimposed partial images is complemented (step S107). Thereafter, the image processing unit 807 determines whether or not processing has been performed on all input images to be processed (step S108). For example, the image processing unit 807 determines whether or not the processing from step S105 to step S107 has been performed on all input images in which objects are detected. Here, the input image to be processed is an input image stored in the input image storage unit 801.
When the processing from step S105 to step S107 has been performed on all input images in which objects have been detected (step S108—YES), the image processing device 80 ends the processing.
On the other hand, when the processing from step S105 to step S107 has not been performed on all the input images in which the object has been detected (step S108-NO), the image processing device 80 executes the processing subsequent to step S104.

According to the image processing device 80 configured as described above, it is possible to suppress deterioration in quality of a composite image generated by combining images taken by a plurality of cameras with a background image. Hereinafter, this effect will be described in detail.
The image processing apparatus 80 generates a partial area image obtained by extracting a small area from which images are not overlapped at the time of synthesis from images taken by adjacent cameras 70. The image processing device 80 generates a partial image by performing conversion processing on the generated partial region image, and superimposes the generated partial image on the background image. Then, the image processing device 80 complements the complement target area between the superimposed partial images. This complement target area corresponds to an overlap area of two images. That is, since an area corresponding to the overlap area is set as a complement target area and complemented, the occurrence of multiple images and disappearance can be suppressed. For this reason, it is possible to suppress deterioration in quality of a synthesized image generated by synthesizing images captured by a plurality of cameras with a background image.

  The image processing apparatus 80 uses, as a reference image, the input image that is the generation source of the first partial image for the first range in the predetermined region near the first partial image that is superimposed on the first region. To complement the first range. Furthermore, the image processing apparatus 80 uses, as a reference image, the input image that is the generation source of the second partial image for the second range in the predetermined region near the second partial image that is superimposed on the second region. To supplement the second range. As described above, the range in the predetermined region near each partial image can be complemented so as to be more similar to the input image by complementing the input image from which the partial image is generated as a reference image.

<Modification>
The present invention is not limited to an omnidirectional image, but can be applied to a case where a composite image is generated by superimposing images taken by a plurality of cameras on a background image.
In the present embodiment, the configuration in which the input image that is the generation source of the partial image is used as it is as the reference image used for the complement processing of the region to be complemented is shown, but the image processing unit 807 is the generation source of the partial image as the reference image. You may use the image which changed the magnification of a certain input image. For example, the image processing unit 807 enlarges the input image that is the generation source of the partial image as the reference image by a predetermined amount (for example, 10%, 20% expansion, etc.) or reduces the input image by a predetermined amount (for example, 10%). , 20% reduction, etc.) may be used.
In the present embodiment, the width value registered in the composite information table is the same for all camera IDs, but the width value may be different for each camera ID, or may be different for some camera IDs. May be.

The example illustrated in FIG. 3 is an example, and the shape of the partial image and the complement target area, the arrangement of the partial image and the complement target area, and the size of the complement target area need not be limited to the example illustrated in FIG. For example, the shapes of the partial image and the complement target area do not have to be rectangular, and are not necessarily the same shape. However, the size of the partial image and the complement target area needs to be larger than the patch size. Further, the partial image and the complement target area may be arranged vertically or horizontally. Further, the size of the complement target area may be smaller or larger than the partial image. Thus, the size of the complement target area is not affected by the size of the partial image. The reason is that the image quality depends on the reference area.
The method of complementing the region to be complemented is not limited to the above method (the method described in FIG. 3). For example, the image processing unit 807 may supplement the complement target area by the following method. First, the input image of the generation source of the partial image superimposed on the first region is A, the input image of the generation source of the partial image superimposed on the second region is B, and the size of A and B is a: Let b be the ratio. When the center portion is complemented using either the A or B input image, weighting is performed so that similar patches are searched at a ratio of a: b.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  60 ... Spherical camera, 70 (70-1 to 70-M) ... Camera, 4, 80 ... Image processing device, 801 ... Input image storage unit, 802 ... Object analysis unit, 803 ... Composite information storage unit, 804 ... Partial region extraction unit, 805 ... background image storage unit, 806 ... image composition unit, 807 ... image processing unit

Claims (6)

A first region and a first region sandwiching a predetermined region set in a background image, which is an image taken at a different position and / or at a different time from at least one of the first input image and the second input image A composite image generation unit that generates a composite image by superimposing the first partial image and the second partial image generated from the first input image and the second input image on the two regions;
By using the first input image and the second input image as a reference image to be used for complementing the predetermined region, the first partial image and the second partial image are superimposed. An image processing unit that complements the predetermined area of
With
The first partial image and the second partial image are images obtained by imaging the same subject located at a predetermined depth from different positions;
The first partial image and the second partial image include a part of the subject,
A part of the subject included in the first partial image is different from a part of the subject included in the second partial image;
The predetermined area is not included in the first partial image when the first partial image and the second partial image are superimposed on the first area and the second area, respectively. And including a partial region of the subject that is not included in the second partial image,
The image processing unit supplements the first range by using the first input image for a first range that is a region closer to the first partial image in the predetermined region, and For the second range that is closer to the second partial image in the region, the second range is complemented by using the second input image, and the first region is set near the center of the predetermined region. An image processing apparatus that complements by using any one of the input image and the second input image. A first region and a first region sandwiching a predetermined region set in a background image, which is an image taken at a different position and / or at a different time from at least one of the first input image and the second input image A composite image generation unit that generates a composite image by superimposing the first partial image and the second partial image generated from the first input image and the second input image on the two regions;
By using the first input image and the second input image as a reference image to be used for complementing the predetermined region, the first partial image and the second partial image are superimposed. An image processing unit that complements the predetermined area of
With
The first partial image and the second partial image are images obtained by imaging the same subject located at a predetermined depth from different positions;
The first partial image and the second partial image include a part of the subject,
A part of the subject included in the first partial image is different from a part of the subject included in the second partial image;
The predetermined area is not included in the first partial image when the first partial image and the second partial image are superimposed on the first area and the second area, respectively. And including a partial region of the subject that is not included in the second partial image,
The image processing unit supplements the predetermined region by using the image after the magnification change of the first input image and the image after the magnification change of the second input image as the reference image. .   The image processing apparatus according to claim 1, wherein the background image is an image taken at a different position and / or at a different time from each of the first input image and the second input image. A first region and a first region sandwiching a predetermined region set in a background image, which is an image taken at a different position and / or at a different time from at least one of the first input image and the second input image A composite image generation step of generating a composite image by superimposing the first partial image and the second partial image generated from the first input image and the second input image on the two regions;
By using the first input image and the second input image as a reference image to be used for complementing the predetermined region, the first partial image and the second partial image are superimposed. An image processing step of complementing the predetermined region of:
Have
The first partial image and the second partial image are images obtained by imaging the same subject located at a predetermined depth from different positions;
The first partial image and the second partial image include a part of the subject,
A part of the subject included in the first partial image is different from a part of the subject included in the second partial image;
The predetermined area is not included in the first partial image when the first partial image and the second partial image are superimposed on the first area and the second area, respectively. And including a partial region of the subject that is not included in the second partial image,
In the image processing step, for the first range that is a region closer to the first partial image in the predetermined region, the first range is complemented by using the first input image, and the predetermined range For the second range that is closer to the second partial image in the region, the second range is complemented by using the second input image, and the first region is set near the center of the predetermined region. An image processing method complemented by using any one of the input image and the second input image. A first region and a first region sandwiching a predetermined region set in a background image, which is an image taken at a different position and / or at a different time from at least one of the first input image and the second input image A composite image generation step of generating a composite image by superimposing the first partial image and the second partial image generated from the first input image and the second input image on the two regions;
By using the first input image and the second input image as a reference image to be used for complementing the predetermined region, the first partial image and the second partial image are superimposed. An image processing step of complementing the predetermined region of:
Have
The first partial image and the second partial image are images obtained by imaging the same subject located at a predetermined depth from different positions;
The first partial image and the second partial image include a part of the subject,
A part of the subject included in the first partial image is different from a part of the subject included in the second partial image;
The predetermined area is not included in the first partial image when the first partial image and the second partial image are superimposed on the first area and the second area, respectively. And including a partial region of the subject that is not included in the second partial image,
In the image processing step, an image processing method for complementing the predetermined region by using the image after the magnification change of the first input image and the image after the magnification change of the second input image as the reference image. .   A computer program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 3.

Images