JP2017102784A - Image processing system, image processing method and image processing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an image processing system capable of generating a virtual entire celestial sphere image while preventing degradation on viewing quality.SOLUTION: The image processing system determines a coupling line for coupling two image in a common are, when coupling two images; i.e., a first image which is an image acquired from a first image and a second image which is an image acquired from the second image and having a common part in an image area with the first image. The image processing system includes coupling line determination means that determines to dispose the coupling line avoiding an area where there is a high possibility that a person is viewing the images.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an image processing apparatus, an image processing method, and an image processing program for processing image data from a plurality of cameras.

  In recent years, a camera (hereinafter referred to as an omnidirectional camera) that can capture an omnidirectional image that is an omnidirectional image including 360 degrees around the user, and viewing the direction in which the user is facing in viewing the omnidirectional image. The head mounted display (HMD) that can be used is becoming popular. And, a service that distributes omnidirectional images via a network is attracting attention. The omnidirectional image as described above can provide a high sense of realism when viewed with an HMD, and is expected to be used for viewing content such as sports and live performances by artists.

  Generally, these omnidirectional images can be taken by installing an omnidirectional camera at a desired viewpoint. However, it is not possible to install a omnidirectional camera in a soccer court or a basketball court during competition because it would interfere with the competitors if an omnidirectional camera is installed. However, there is a desire to watch videos as if standing in a soccer court or basketball court during competition. Therefore, a virtual viewpoint, which is a virtual viewpoint, is usually set in a place where an omnidirectional camera cannot be installed, and multiple cameras that shoot an area including the virtual viewpoint are installed, and images from these cameras are displayed. A technique for obtaining an omnidirectional image as if taken by an omnidirectional camera at this virtual viewpoint has been devised (see, for example, Non-Patent Document 1). In the following description, the omnidirectional image at the virtual viewpoint is referred to as a virtual omnidirectional image.

  A specific example of an image processing system for obtaining a virtual omnidirectional image by combining images from a plurality of cameras will be described. FIG. 20 is a diagram illustrating an image processing system for obtaining a conventional virtual omnidirectional image. As shown in FIG. 20, the image processing system 1 includes an omnidirectional camera 2 and a plurality of N (N ≧ 2) cameras 3-1, 3-2, 3-3,. , A camera group 3), an image processing device 4, and a display device 5. When the virtual viewpoint 11 is set in the futsal court 10, the image processing system 1 obtains a virtual omnidirectional image at the virtual viewpoint 11 by synthesizing images from the camera group 3 installed outside the court 10.

  The omnidirectional camera 2 is a camera that captures an omnidirectional image. The omnidirectional camera 2 is installed at the position of the virtual viewpoint 11 in the court 10 at a timing before the game is played. The omnidirectional camera 2 captures in advance a background image 20 that is the background of the virtual omnidirectional image from the position of the virtual viewpoint 11. A background image 20 that is an omnidirectional image captured by the omnidirectional camera 2 is input to the image processing device 4 and accumulated.

  A camera group 3 is installed around the court 10. In FIG. 20, N is a natural number of 4 or more. The larger the number of cameras constituting the camera group 3, the better, but the minimum number is two. The camera group 3 is installed around the court 10 so as to have an angle of view including the virtual viewpoint 11. The image processing device 4 performs image processing on the cut-out image including the foreground image output from each camera of the camera group 3 to be combined with the background image 20. The image processing device 4 combines the partial image after image processing with the background image 20 acquired from the omnidirectional camera 2 to generate a virtual omnidirectional image. The display device 5 is a device that displays the virtual omnidirectional image generated by the image processing device 4, and is a liquid crystal display or the like.

  A specific example of image processing in the image processing system 1 will be described. FIG. 21 is a diagram illustrating a specific example of an image subjected to image processing in the image processing system 1. FIG. 21A is a diagram illustrating an example of the background image 20 captured by the omnidirectional camera 2 installed at the position of the virtual viewpoint 11. The image is a 360 degree image centered on the virtual viewpoint 11. Since the background image 20 is an image taken before the start of the competition, no player or the like who competes in the court 10 is shown.

  FIG. 21B shows a partial image 21 captured by the camera 3-1, a partial image 22 captured by the camera 3-2, and a partial image 23 captured by the camera 3-3 from the left. The image processing apparatus 4 cuts out regions 211, 221, and 231 that include the virtual viewpoint 11 and include futsal players from each of the partial images 21 to 23. The image processing apparatus 4 generates partial images 211 a, 221 a, and 231 a that can be pasted on the background image 20 by performing image processing on the cut out images of the areas 211, 221, and 231.

  The image processing device 4 generates the virtual omnidirectional image 24 by combining the background images 20 with the partial images 211a, 221a, and 231a. FIG. 21C is a diagram illustrating an example of the virtual omnidirectional image 24 generated by the image processing device 4. As shown in FIG. 21C, the virtual omnidirectional image 24 has the partial images 211a, 221a, and 231a pasted in a predetermined area, so that the futsal player who is playing the game on the court 10 is shown. It is an image.

  In the conventional image processing system 1, the optical center of the camera group 3 used for composition and the optical center of the virtual omnidirectional camera assumed in the virtual viewpoint 11 are different. For this reason, the synthesized virtual omnidirectional image 24 includes a geometrically incorrect image. In order to prevent this, the image processing device 4 performs image processing so that the consistency is maintained at one point in the depth indicating the distance from the virtual viewpoint 11 and pastes the partial images 211a, 221a, and 231a on the background image 20. There is a need. However, when pasting a partial image of an object (for example, a player who is competing) that does not exist at a depth where consistency is maintained but is located at another depth, the depth consistency is maintained by image processing. I can't. Such an object whose depth is inconsistent causes a phenomenon that the virtual omnidirectional image 24 becomes a duplicated image (multiple image) or disappears.

  Hereinafter, a phenomenon in which an image of an object is duplicated or disappeared in the virtual omnidirectional image 24 will be described with reference to the drawings. FIG. 22 is a diagram for explaining a problem in the image processing system 1. In FIG. 22, the shooting range 41 indicates the shooting range of the region 211 shown in FIG. 21B in the shooting range of the camera 3-1. The shooting range 42 indicates the shooting range of the region 221 shown in FIG. 21B in the shooting range of the camera 3-2. The shooting range 43 indicates the shooting range of the area 231 shown in FIG. 21B in the shooting range of the camera 3-3. In addition, there are three subjects (players) 49 to 51 having different distances (depths) from the virtual viewpoint 11.

  In the depth 46 which shows the 1st distance from the virtual viewpoint 11 shown with the broken line in FIG. 22, each imaging | photography range 41-43 is located in a line without overlapping. The subject 49 positioned at the depth 46 is a subject 49 that is consistent in the depth without the image being duplicated or lost. In the depth 47 indicating the second distance from the virtual viewpoint 11, the shooting ranges 41 to 43 overlap as shown by the horizontal line portion 44. The subject 50 positioned at the depth 47 is a subject 50 that is inconsistent in the depth because the image is duplicated. The depth 48 indicating the third distance from the virtual viewpoint 11 is vacant as indicated by the hatched portion 45 between the imaging ranges 41 to 43. Since the subject 51 located at the depth 48 is partially lost, the subject 51 is not consistent with the depth.

Kosuke Takahashi and three others, “Study on virtual spherical image composition using multiple camera images”, IEICE Technical Report, June 1, 2015, vol.115, no.76, MVE2015-5, p. 43-48

  As described above, a region where a subject is present in the virtual omnidirectional image is highly likely to be a viewing region that is a region watched by the user. There is a problem that the viewing quality of the image is degraded. In addition, when an image is stitched when the images are combined, there is also a problem that the viewing quality of the virtual omnidirectional image is deteriorated.

  The present invention has been made in view of such circumstances, and an object of the present invention is to provide an image processing device, an image processing method, and an image processing program that generate a virtual omnidirectional image that suppresses a decrease in viewing quality. .

  One embodiment of the present invention is a first image that is an image acquired from a first video, and an image that is acquired from a second video and has a part of an imaging region in common with the first image. An image processing apparatus that determines a connection line for combining two images at a common portion when combining two images with a second image, and is viewed by a person viewing the image The image processing apparatus includes a coupling line determination unit that determines to arrange the coupling line while avoiding a region that is highly likely to be present.

  One embodiment of the present invention is a first image that is an image acquired from a first video, and an image that is acquired from a second video and has a part of an imaging region in common with the first image. An image processing apparatus for determining a connection line for combining the two images at a common portion when combining two images with a certain second image, and at the common portion, In the common part, the pixel that is sensitive to the first energy image generating unit that generates a first energy image defined as having a large energy and searching for a pixel having a low energy in the first energy image, A bond line determining means for determining the arrangement of the bond lines by obtaining a series of pixels with low energy, the bond line determining means being viewed by a person viewing the image. While avoiding potential higher regions, an image processing apparatus for determining the binding wire.

  One aspect of the present invention is the image processing apparatus, wherein the common part of the image detects a region that is viewed by a person viewing the image, and detection by the region detection unit. Based on the result, a second energy image generating means for generating a second energy image in which the detected energy of the region is increased, and a third energy by adding the first energy image and the second energy image. The image processing apparatus further includes third energy image generation means for generating an image, and the combination line determination means determines the combination line by obtaining a series of pixels having low energy based on the third energy image.

  One aspect of the present invention is the image processing device, wherein the coupling line determination unit is configured to be orthogonal to a direction aligned when the two images of the common portion obtained by the seam carving process are combined. The seam is the connecting line.

  One aspect of the present invention is the image processing apparatus, wherein the two images are used when generating a virtual omnidirectional image viewed from a virtual viewpoint provided at a predetermined position.

  One embodiment of the present invention is a first image that is an image acquired from a first video, and an image that is acquired from a second video and has a part of an imaging region in common with the first image. An image processing method performed by an image processing apparatus that determines a connection line for combining two images at a common portion when combining two images with a second image, and viewing the image This is an image processing method including a coupling line determination step for determining that the coupling line is arranged while avoiding an area that is likely to be viewed by a person.

  One embodiment of the present invention is a first image that is an image acquired from a first video, and an image that is acquired from a second video and has a part of an imaging region in common with the first image. An image processing method performed by an image processing apparatus that determines a connection line for combining the two images at a common portion when combining two images with a certain second image, wherein An energy image generating step for generating a first energy image that is defined as having high energy for pixels that are sensitive to human eyes, and the common part while searching for an image with low energy in the first energy image A bond line determining step for determining the bond line in step (a), wherein the bond line determining step is configured to determine the bond line while avoiding an area that is likely to be viewed by a person viewing the image. The image processing method of the constant.

  One embodiment of the present invention is an image processing program for causing a computer to function as the image processing apparatus.

  According to the present invention, when two images are combined, the joint between the images is arranged at a position where it is difficult to notice, so that it is possible to generate a virtual omnidirectional image that suppresses a decrease in viewing quality. can get.

1 is a block diagram illustrating a configuration of an image processing apparatus according to an embodiment of the present invention. 2 is a diagram illustrating a basic configuration example of an image processing device 30. FIG. It is a figure which shows an example of the object information stored in the object information storage part 303. FIG. It is a figure which shows the specific example in case overlap occurs in the boundary area | region between adjacent partial images. FIG. 3 is a flowchart showing an operation for creating a virtual omnidirectional image of one frame in the image processing system 1. It is a figure explaining the operation | movement which the image processing apparatus 30 produces the virtual omnidirectional image of a moving image. It is a schematic diagram which shows the production | generation process of a virtual omnidirectional image. It is a schematic diagram which shows the operation | movement which calculates | requires a seam. It is explanatory drawing which shows the process which couple | bonds two partial images. It is a figure which shows the example which produces | generates and processes an energy image based on the advancing direction of a foreground. It is explanatory drawing which shows the production | generation method of an energy image. It is explanatory drawing which shows the production | generation method of an energy image. FIG. 10 is a flowchart illustrating an operation of combining two partial images illustrated in FIG. 9. FIG. It is a figure which shows the example which weights by the direction which has faced. It is a figure which shows the example which weights by the distance from a center. It is explanatory drawing which shows the production | generation method of an energy image. It is a figure which shows the example which calculates | requires energy also based on the speed of the motion of a foreground. It is a figure which shows the example which produced | generated the virtual omnidirectional image. It is explanatory drawing which shows the example of the partial image cut out from the input image of a camera. It is a figure which shows the image processing system for obtaining the conventional virtual omnidirectional image. 3 is a diagram illustrating a specific example of an image to be image processed in the image processing system 1. FIG. 2 is a diagram for explaining a problem in the image processing system 1. FIG.

  Hereinafter, an image processing apparatus according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a system configuration for viewing a virtual omnidirectional image according to the embodiment. In this figure, the same parts as those of the conventional apparatus shown in FIG. A system for viewing a virtual omnidirectional image includes an image processing system 1 and a viewing system 9.

  As shown in FIG. 1, the image processing system 1 includes an omnidirectional camera 2 and a plurality of N (N ≧ 2) cameras 3-1, 3-2, 3-3,. , A camera group 3), an image processing device 30, and a display device 5. When the virtual viewpoint 11 is set in the futsal court 10, the image processing system 1 obtains a virtual omnidirectional image at the virtual viewpoint 11 by synthesizing images from the camera group 3 installed outside the court 10.

  The omnidirectional camera 2 is a camera that captures an omnidirectional image. The omnidirectional camera 2 is installed at the position of the virtual viewpoint 11 in the court 10 at the timing before the competition is performed. The omnidirectional camera 2 captures in advance a background image 20 that is the background of the virtual omnidirectional image from the position of the virtual viewpoint 11. The background image 20 captured by the omnidirectional camera 2 is input to the image processing device 4 and accumulated. The omnidirectional camera 2 is removed from the position of the virtual viewpoint 11 before the start of the competition because the omnidirectional camera 2 becomes a hindrance to the competition if it remains installed at the virtual viewpoint 11 during the competition.

  A camera group 3 is installed around the court 10. Each of the cameras 3-1, 3-2, 3-3,..., 3-N of the camera group 3 is a camera that shoots a moving image (video) including a foreground to be synthesized with the background image 20. 11 is installed so as to surround the periphery of the coat 10 so as to have an angle of view including 11. In FIG. 1, N is an integer equal to or greater than 4. If a virtual omnidirectional image with similar image quality is to be obtained, the larger the coat 10, the larger the value. The higher the image quality of the virtual omnidirectional image, the larger the value.

  The image processing device 30 performs image processing on an input image that is a moving image captured by each of the cameras 3-1, 3-2, 3-3,. A process of synthesizing the partial image after the image processing with the background image 20 acquired from the spherical camera 2 is performed. The display device 5 is a device that displays a virtual omnidirectional image generated by the image processing device 30, and is a liquid crystal display, a head mounted display (HMD), or the like.

  The viewing system 9 includes an image server 6, a network 7, and a plurality of viewing devices 8. The image server 6 is a server that distributes the virtual omnidirectional image generated by the image processing device 30 via the network 7. The network 7 is a communication network such as the Internet. The viewing device 8 is a device that includes a user terminal 81 that can be connected to the network 7 and an HMD 82 that is connected to the user terminal 81. The user terminal 81 receives a virtual omnidirectional image distributed by the image server 6 via the network 7, converts the received virtual omnidirectional image into a video signal that can be viewed on the HMD 82, and outputs the video signal to the HMD 82. With functionality.

  The HMD 82 includes a receiving unit that receives a video signal and the like from the user terminal 81, a screen that includes a liquid crystal display that displays the video signal received through the receiving unit, and a detection unit that detects the movement of the viewer's head. And a transmission unit that transmits a result detected by the detection unit to the user terminal 81. The video displayed on the screen of the HMD 82 is a part of a virtual omnidirectional video based on the virtual omnidirectional image and is called a visual field. The HMD 82 has a function of changing the visual field, which is a range of video to be displayed, according to the viewer's head movement detected by the detection unit.

  Since the video being viewed changes as the head moves up, down, left and right, the viewer wearing the HMD 82 can view the video as if watching the competition from the position of the virtual viewpoint 11. it can. In this way, the viewer wearing the HMD 82 can view a video with a sense of presence as if standing in the virtual viewpoint 11 and watching the competition.

  Since the image processed in the image processing system 1 is the same as the image processed in the conventional image processing system 1 shown in FIG. 20, the operation of the image processing system 1 will be briefly described with reference to FIG. The omnidirectional camera 2 is installed at the virtual viewpoint 11 in the court 10 and shoots the background image 20 shown in FIG. When the competition starts, each camera in the camera group 3 starts shooting. For example, the cameras 3-1, 3-2, and 3-3 in the camera group 3 capture the partial images 21 to 23 illustrated in FIG.

  The image processing apparatus 30 cuts out areas 211, 221, and 231 that include the virtual viewpoint 11 from each of the photographed partial images 21 to 23 and that include players in competition. The image processing apparatus 30 generates partial images 211 a, 221 a, and 231 a that can be pasted on the background image 20 by performing image processing on the images of the extracted areas 211, 221, and 231. The image processing apparatus 30 combines the partial images 211a, 221a, and 231a with the background image 20 to generate a virtual omnidirectional image 24 as shown in FIG.

  The viewing system 9 is not limited to the configuration shown in FIG. The viewing system 9 has a configuration capable of distributing the virtual omnidirectional image via the network 7, such as a configuration including an editing device that edits the virtual omnidirectional image generated by the image processing device 30 and outputs the edited image to the image server 6. I just need it. The configuration of the viewing device 8 may be any configuration as long as the user can view the virtual omnidirectional image received via the network 7.

  Next, the configuration of the image processing apparatus 30 shown in FIG. 1 will be described. FIG. 2 is a diagram illustrating a basic configuration example of the image processing apparatus 30. As shown in FIG. 2, the image processing apparatus 30 includes an object analysis unit 31, a depth acquisition unit 32, a synthesis information acquisition unit 33, an image input unit 34, an image clipping unit 35, an image synthesis unit 36, Each of the cameras in the camera group 3 includes a display processing unit 37, an input unit 38 configured to input information regarding depth, a boundary determination unit 39 that determines a boundary (seam) of partial images to be combined, and a display processing unit 37. A foreground image storage unit 301 that stores a partial image including a captured foreground image, a background image storage unit 302 that stores a background image 20, an object information storage unit 303, and a composite information table 304 are provided.

  The object analysis unit 31 receives a partial image stored in the foreground image storage unit 301 as an input, extracts an object included in the partial image, and outputs it. Here, the object is a person, an object (for example, a ball) or the like that is not included in the background image 20 but is included in the partial image. The object analysis unit 31 assigns an ID that is an identifier for identifying the object to the extracted object.

  The partial images photographed by each camera in the camera group 3 are moving images having a predetermined frame period, and the photographing time is associated with each frame. The object analysis unit 31 assigns the same ID to the same object extracted from a series of frames in the time direction. The object information storage unit 303 stores information about the object including the ID assigned by the object analysis unit 31 in association with the shooting time for each frame of the partial image from which the object is to be extracted.

  For example, the object analysis unit 31 assigns an identifier of ID1 to an object extracted from the partial image 21 that is a series of frames at the photographing times t, t + 1, t + 2,. Similarly, the object analysis unit 31 assigns an identifier of ID2 to the object extracted from the partial image 22 that is a series of frames at the shooting times t, t + 1, t + 2,. ID-3 is assigned to the object extracted from the partial image 23, which is a series of frames at the photographing times t, t + 1, t + 2,.

  When the object analysis unit 31 analyzes the partial image and extracts the object, the object analysis unit 31 acquires a label indicating the attribute of the object and three-dimensional position information that is three-dimensional position information in the space on the court 10 of the object. To do. Specific examples of the label include “person” indicating a person, “ball” indicating a ball, “object A” indicating an object A, and “object B” indicating an object B. ,..., Etc., information for identifying an object that may move within the shooting range of the camera group 3 is used.

The object analysis unit 31 analyzes and determines whether the object corresponds to “person”, “ball”, “object A”, or “object B” by analyzing the partial image in order to extract the object. Then, the determination result is output as a label. It should be noted that a known image analysis technique is used as a method for analyzing and determining whether the object corresponds to “person”, “ball”, “object A”, or “object B”. For example, there is the following publicly known document 1 as a document disclosing a technique for detecting a person by analyzing an image.
Known Document 1: Atsushi Yamauchi and 2 others, “[Survey Paper] Human Detection by Statistical Learning Method”, IEICE Technical Report, vol.112, no.197, PRMU2012-43, pp.113- 126, September 2012

  Further, the object analysis unit 31 is based on information such as the position of the object in the partial image, the positions of a plurality of cameras in the camera group 3 that photographed the object, and the photographing ranges (shooting direction and angle of view) of the plurality of cameras. Thus, the three-dimensional position of the object in the space on the court 10 is acquired. As a method for acquiring the three-dimensional position of the object, a known technique is used. Further, the acquisition position information may be information on a two-dimensional position.

  The object information storage unit 303 receives object information, which is information about the object extracted by the object analysis unit 31, and stores the object information in association with the shooting time. The object information includes an ID for identifying the object, a label indicating the attribute of the object, and the three-dimensional position of the object.

  FIG. 3 is a diagram illustrating an example of object information stored in the object information storage unit 303. As shown in FIG. 3, a plurality of pieces of object information are stored in association with times t, t + 1, t + 2,. At time t, ID1, label 1, and three-dimensional position information 1 are stored as object information of the object 1, and ID2, label 2, and three-dimensional position information 2 are stored as object information of the object 2. The same information is stored at time t + 1 and time t + 2.

  The depth acquisition unit 32 reads out the object information from the object information storage unit 303, specifies a main object that is an important object from among a plurality of objects at each shooting time, and continues from the virtual viewpoint 11 to the specified main object. The depth information related to the depth which is the distance is acquired and output. An important object is, for example, an object that exists in a region in which a viewer gazes in a virtual omnidirectional image.

  The depth acquisition unit 32 specifies the main object at each shooting time in advance. Specifically, the content creator who creates the virtual omnidirectional image uses the input unit 38 to specify information for identifying an area estimated to be watched by the viewer or an object estimated to be watched by the viewer at each shooting time. input. Thereby, the depth acquisition part 32 specifies the main object in each imaging | photography time based on the input information. The method for specifying the main object in the depth acquisition unit 32 is not limited to the method described above, and various methods may be used. For example, the Salientity Map, which is a map showing the degree of interest of the viewer in the captured partial image for each region, is obtained and input to the depth acquisition unit 32. The depth acquisition unit 32 may identify an object that exists in a visually noticeable region as a main object based on the input Salinity Map. In addition, the test subject is allowed to view a video that is a partial image in advance, a viewing log indicating which region was viewed at each shooting time is acquired, the viewing log is input to the depth acquisition unit 32, and based on the input viewing log The main object may be specified.

In addition, the method for obtaining the Saliency Map is a known technique. For example, the technique described in the following known document 2 may be used.
Known Document 2: Laurent Itti, Christof Koch, and Ernst Niebur, "A Model of Saliency-Based Visual Attention for Rapid Scene Analysis", IEEE Transactions on Pattern Analysis and Machine Intelligence, 20 (11): 1254-1259 (1998)

  The composite information table 304 is cut-out area information that is information about a cut-out area for cutting out an area including the virtual viewpoint 11 from a partial image, and information for converting an image cut out according to the cut-out area into a partial image. Composite information including certain conversion information is stored. The partial image is generated by subjecting the cut-out image to deformation processing such as enlargement, reduction, and rotation according to the conversion information in order to paste the cut-out image to the corresponding region of the background image 20 without a sense of incongruity. This deformation process is performed, for example, by performing affine transformation on the image. The conversion information when performing affine transformation on an image is, for example, an affine transformation matrix. The following shows an example of using affine transformation as the deformation processing performed on the partial area image. However, the deformation processing is not limited to affine transformation, and image conversion by enlargement, reduction, rotation, etc. is performed according to conversion information. Any process may be used as long as the process is performed. The composite information table 304 includes a camera code that identifies a camera that has captured a partial image to be processed in the camera group 3, a depth from the virtual viewpoint 11, conversion information that is an affine transformation matrix corresponding to the depth, and It is a table which stores in association with cut-out area information according to depth.

  The affine transformation matrix is acquired in advance by the following method and stored in the synthesis information table 304. For example, an image including a chess board photographed by the omnidirectional camera 2 installed at the virtual viewpoint 11 by installing a lattice-patterned chess board at a plurality of types of distances (depths) from the virtual viewpoint 11, and the camera group 3 Compare the image with the chess board taken in. Then, in both images, an affine transformation matrix for transforming the images so as to correspond to each grid of the photographed chess board is obtained. In this way, an affine transformation matrix corresponding to the depth at which the chess board is installed is obtained.

  The cut-out area information is acquired in advance by the following method and stored in the synthesis information table 304. For example, if there is an overlapping area where the same subject (chessboard) exists in partial images taken by two adjacent cameras in the camera group 3, both cameras remain so that only one area remains. The cutout area for the image of is set. The cutout area is obtained for each camera included in the camera group 3 with respect to a plurality of types of distances (depths) from the virtual viewpoint 11 to the subject (chess board). Note that the cutout area may be set so that an overlapping area having a width of several pixels to several tens of pixels is left in the images of both cameras.

  The composite information acquisition unit 33 uses the depth acquired by the depth acquisition unit 32 as an input, and based on the depth, from the composite information table 304, a cutout region and an affine transformation corresponding to a partial image captured by each camera of the camera group 3 Obtain and output composite information including a matrix. Since there are several to several tens of depths stored in the composite information table 304, it is assumed that there is no depth table having the same value as the depth acquired by the depth acquisition unit 32. In such a case, the composite information acquisition unit 33 combines information corresponding to the two depth values recorded in the composite information table 304 that are values before and after the depth acquired by the depth acquisition unit 32 (cutout area information). And conversion information), the combined information corresponding to the depth acquired by the depth acquisition unit 32 is calculated. Specifically, the coordinate value of the clip region in the clip region information corresponding to the two recorded depth values is linearly interpolated to identify the clip region located between the two. By linearly interpolating each coefficient of the affine transformation matrix corresponding to the two recorded depth values, an affine transformation matrix serving as an intermediate value is calculated.

  The foreground image storage unit 301 stores a partial image including a foreground image captured by each camera of the camera group 3 in association with a camera code that identifies each camera. The partial image includes shooting time and moving image data. The foreground image storage unit 301 stores, for example, the partial image 21 shown in FIG. 21B in association with the camera code that identifies the camera 3-1, and the partial image 22 that identifies the camera 3-3. And the partial image 23 is stored in association with the camera code that identifies the camera 3-3.

  The background image storage unit 302 stores the background image 20 that is an omnidirectional image captured by the omnidirectional camera 2. The background image storage unit 302 stores, for example, the background image 20 shown in FIG. 21A photographed by the celestial camera 2 installed at the virtual viewpoint 11 in the court 10. The background image 20 to be stored may be image data for one frame or moving image data for a predetermined time. When storing image data for a predetermined time, a portion that periodically changes in the background image 20 (for example, a portion in which an electric bulletin board is reflected and a portion in which the display content of the electric bulletin board is periodically changed) .), Image data for a time corresponding to the cycle may be stored as the background image 20.

  The configuration in which the image processing apparatus 30 acquires the background image 20 from the omnidirectional camera 2 may be any configuration. For example, the image processing device 30 may include a communication unit that can communicate with the omnidirectional camera 2 in a wired or wireless manner, and the background image 20 may be acquired via the communication unit. In addition, the background image 20 is recorded on the recording medium using a recording medium that can be attached to and removed from the omnidirectional camera 2, and the recorded recording medium is connected to the image processing apparatus 30. A configuration in which the background image 20 is acquired from the background image 20 may be obtained. Further, the configuration in which the image processing device 30 acquires the partial image from the camera group 3 may be any configuration as in the case of the omnidirectional camera 2.

  The image input unit 34 acquires a partial image from the partial image storage unit 301, acquires the background image 20 from the background image storage unit 302, outputs the partial image to the image cutout unit 35, and outputs the background image 20 to the image composition unit To 36. The image cutout unit 35 specifies a cutout region corresponding to the partial image from each camera of the camera group 3 based on the cutout region information included in the composite information acquired by the composite information acquisition unit 33, and specifies from the partial image. The cut out area is cut out, and the cut out image is output to the image composition unit 36. For example, the image cutout unit 35 performs a process of cutting out the cutout areas 211, 221, and 231 from each of the partial images 21 to 23 illustrated in FIG.

  The image synthesizing unit 36 receives the image cut out by the image cutout unit 35, the combination information acquired by the synthesis information acquisition unit 33, and the background image as input, and performs the synthesis information acquisition unit 33 on the image cut out by the image cutout unit 35. The transformation processing is performed based on the affine transformation matrix of the transformation information included in the composite information acquired by generating a partial image. The image synthesizing unit 36 generates and outputs a virtual omnidirectional image by pasting the generated partial image to the background image 20 based on the affine transformation matrix and synthesizing it. Note that the affine transformation matrix includes information indicating an area where the partial image is pasted in the background image 20. The image composition unit 36 has a function of transmitting the generated virtual omnidirectional image to the image server 6.

  For example, the image composition unit 36 performs a deformation process based on the affine transformation matrix on the images obtained by cutting out the cutout areas 211, 221, and 231 from the partial images 21 to 23 illustrated in FIG. Partial images 211a, 221a, and 231a are generated. The image composition unit 36 generates the virtual omnidirectional image 24 shown in FIG. 21C by, for example, pasting the partial images 211a, 221a, and 231a on the background image 20 and combining the partial images 211a, 221a, and 231a. .

  When the partial image is pasted on the background image 20 and the virtual omnidirectional image 24 is generated, there may be an overlap in the boundary region between the adjacent partial images. FIG. 4 is a diagram illustrating a specific example in the case where overlap occurs in a boundary region between adjacent partial images. As shown in FIG. 4, the partial image 211 b and the partial image 221 b pasted on the virtual omnidirectional image 24 overlap in the boundary region 25. Note that the partial image 211b and the partial image 221b shown in FIG. 4 are different from the partial image 211a and the partial image 221a shown in FIG. 21C in that there are overlapping areas in both images.

As illustrated in FIG. 4, when the partial image 211 b and the partial image 221 b overlap in the boundary region 25, the image composition unit 36 performs blending (Blending) described below for the overlapping boundary region 25. ) Process. The image composition unit 36 determines a blending parameter α, and calculates the value of each pixel in the overlap region 25 based on (Equation 1).
g (x, y) = αI _i (x, y) + (1−α) I _{i + 1} (x, y) (Equation 1)

In (Expression 1), x and y are horizontal and vertical coordinates on the virtual omnidirectional image 24. g (x, y) is the value of the pixel value of the coordinates (x, y) in the boundary region 25. I _i (x, y) and I _{i + 1} (x, y) are coordinates of partial images generated based on the partial images photographed by the cameras 3-i and 3- (i + 1) in the camera group 3. x, y) represents the value of the pixel value. Further, the value of α is constant in the overlapping region 25, but may be changed as shown in the following (Equation 2).
α (x) = (x−xs) / (xe−xs) (Formula 2)
In (Expression 2), xs and xe are the x coordinates of both ends of the overlapping region 25 as shown in FIG. 4, and xs <xe.

  The display processing unit 37 receives the virtual omnidirectional image output from the image synthesis unit 36, converts the virtual omnidirectional image into a video signal that can be displayed on the display device 5, and outputs the video signal. As shown in FIG. 21C, the virtual omnidirectional image 24 is an image including distortion and an image including a landscape of 360 degrees with the virtual viewpoint 11 as the center. It has a function of cutting out an image in a range to be displayed on the display device 5 from the virtual omnidirectional image and correcting distortion of the cut out image.

  The image processing apparatus 30 includes the foreground image storage unit 301 and the background image storage unit 302, but is not limited thereto. For example, an image storage device including a foreground image storage unit 301 and a background image storage unit 302 is separately provided, and the image processing device 30 acquires the foreground image storage unit 301 and the background image storage unit 302 from the image storage device. Also good.

  The boundary determination unit 39 receives the virtual celestial sphere image output from the image synthesis unit 36 and the synthesis information as input, and does not perform blending processing on the boundary region described above, but sets the boundary line to be difficult to see. Thus, a boundary region for performing natural image composition is determined and output.

  Next, an operation for creating a virtual omnidirectional image of one frame in the image processing system 1 will be described. FIG. 5 is a flowchart showing an operation of creating a virtual omnidirectional image of one frame in the image processing system 1. The operation shown in FIG. 5 includes a process of acquiring object information, composite information, background image 20 and partial image in advance before the process of generating a virtual omnidirectional image at each shooting time.

  After the omnidirectional camera 2 is installed at the virtual viewpoint 11 and the chess board is installed at a predetermined distance (depth) from the virtual viewpoint 11, the omnidirectional camera 2 captures an omnidirectional image including the chess board ( Step S101). The omnidirectional camera 2 is removed from the virtual viewpoint 11, and the shooting range including the virtual viewpoint 11 and the chess board is taken by each camera of the camera group 3, and is included in the omnidirectional image taken by the omnidirectional camera 2. The composite information for associating the chess board to be matched with the chess board included in the image photographed by one camera in the camera group 3 is obtained (step S102). Note that the shooting of the chess board in steps S101 and S102 is performed by installing the chess board at a plurality of types of distances from the virtual viewpoint 11.

  After the omnidirectional camera 2 is installed at the virtual viewpoint 11, the omnidirectional camera 2 captures the background image 20 (step S103). The captured background image 20 is stored in the background image storage unit 302. After the omnidirectional camera 2 is removed from the virtual viewpoint 11, the camera group 3 starts photographing, for example, when the competition starts. As a result, the image processing apparatus 30 stores the partial image captured by the camera group 3 in the foreground image storage unit 301. The object analysis unit 31 reads out the partial image from the foreground image storage unit 301 and performs analysis processing, and stores the analysis result in the object information storage unit 303. The depth acquisition unit 32 specifies a main object based on information input from the input unit 38 from among the objects stored in the object information storage unit 303. The depth acquisition unit 32 acquires depth information from the virtual viewpoint 11 to the identified main object (step S104).

  The composite information acquisition unit 33 receives the depth acquired by the depth acquisition unit 32 as input, and acquires composite information including a cutout region and an affine transformation matrix corresponding to each partial image from the composite information table 304 based on the depth. Are output (step S105). In step S <b> 105, when there is no depth table having the same value as the depth acquired by the depth acquisition unit 32, the composite information acquisition unit 33 combines the depth corresponding to the depth that is the value before and after the depth acquired by the depth acquisition unit 32. Based on the information, composite information corresponding to the depth acquired by the depth acquisition unit 32 is obtained.

  The image cutout unit 35 receives the cutout region information included in the composite information acquired by the composite information acquisition unit 33, and based on the cutout region information, the cutout region corresponding to the partial image from each camera in the camera group 3 Is extracted, the specified cutout region is cut out from the partial image, and the cutout image is output to the image composition unit 36. The image composition unit 36 receives the image cut out by the image cutout unit 35, the combination information acquired by the combination information acquisition unit 33, and the background image, and converts the image cut out by the image cutout unit 35 into the conversion information included in the combination information. A deformation process is performed based on the affine transformation matrix of information to generate a partial image. The image compositing unit 36 generates and outputs a virtual omnidirectional image by pasting the generated partial image to the background image 20 based on the affine transformation matrix and compositing (step S106).

  When overlapping in the boundary region between the two partial images pasted on the background image 20, the image composition unit 36 performs blending processing on the overlapping boundary region (step S107).

  Next, a basic operation in which the image processing apparatus 30 creates a virtual omnidirectional image of a moving image will be described. FIG. 6 is a diagram illustrating an operation in which the image processing apparatus 30 creates a virtual omnidirectional image of a moving image. In the operation of FIG. 6, it is assumed that the processes up to capturing of the partial image in steps S <b> 101 to S <b> 104 shown in FIG. 5 have already been completed. As shown in FIG. 6, the image processing apparatus 30 starts processing for the frame at the first photographing time (step S201).

  The image input unit 34 acquires a partial image from the foreground image storage unit 301, acquires the background image 20 from the background image storage unit 302, outputs the partial image to the image clipping unit 35, and outputs the background image 20 to the image composition unit 36 (step S202). The depth acquisition unit 32 specifies a main object from the objects stored in the object information storage unit 303 based on information input from the input unit 38, and acquires the depth to the specified main object ( Step S203).

  The composite information acquisition unit 33 receives the depth acquired by the depth acquisition unit 32 as input, acquires composite information corresponding to each partial image from the composite information table 304 based on the depth, and outputs the composite information (step S204). The image cutout unit 35 receives the combination information acquired by the combination information acquisition unit 33, cuts out a cutout region from the partial image based on the combination information, and outputs the cutout image to the image composition unit 36. The image composition unit 36 receives the image cut out by the image cutout unit 35, the combination information acquired by the combination information acquisition unit 33, and the background image as input, and the image clipped by the image cutout unit 35 includes the affine included in the combination information. A deformation process is performed based on the transformation matrix to generate a partial image. The image composition unit 36 combines the generated partial image with the background image 20 based on the affine transformation matrix to generate and output a virtual omnidirectional image (step S205). If there is a partial image at the next shooting time, the image processing apparatus 30 returns to step S201 to continue the loop, and if there is no partial image at the next shooting time, the loop ends (step S206).

  As described above, the image processing apparatus 30 obtains a depth corresponding to the main object that the viewer is interested in, generates a partial image corresponding to the obtained depth, and pastes the generated partial image on the background image 20. Thus, a virtual omnidirectional image can be generated. Thereby, the image processing apparatus 30 can suppress the occurrence of alternation or disappearance in the subject that is the main object included in the virtual omnidirectional image. The image processing device 30 can suppress the occurrence of a parting in the subject of interest of the viewer included in the virtual omnidirectional image by performing the composition processing according to the depth of the subject of interest of the viewer. It is possible to provide a viewer with a virtual omnidirectional image in which the deterioration of quality is suppressed.

<First Embodiment>
Next, an image processing apparatus according to the first embodiment of the present invention will be described. In the first embodiment, the boundary process performed during the above-described synthesis process is modified. In the first embodiment, as described above, the blending process is not performed, but a boundary that is not easily noticed when the partial images are combined is used. Here, the virtual omnidirectional image generation processing will be briefly described with reference to FIG. FIG. 7 is a schematic diagram illustrating a virtual omnidirectional image generation process. First, input images respectively acquired from respective moving images (videos) photographed by the cameras C _i−1 , C _i , and C _{i + 1} are acquired in advance. A moving image (video) photographed by the camera is composed of images of a plurality of frames, and here, an image of a processing target frame is acquired as an input image. In addition, in order to generate an energy image, an image of one or more frames past the processing target frame is also acquired in advance. Then, cut-out images S _i−1 , S _i , and S _{i + 1} that are the foreground are cut out from the obtained input images.

Next, the cut-out images S _i−1 , S _i , S _{i + 1} are subjected to image conversion using the affine transformation matrices A _i−1 , A _i , A _{i + 1} obtained in advance, and the partial images S ′ _i−1 , S ′ _i and S ′ _{i + 1} are generated. Then, a synthesis process is performed with the omnidirectional image B that has been captured in advance. At this time, instead of arranging the partial images so as to overlap each other, a seam carving process is applied so that the joint between the partial images and the partial images is not noticeable. By trying to synthesize like this. A virtual omnidirectional image viewed from the virtual viewpoint Pv can be generated. By directing a line of sight toward the scene that the user wants to see with this HDM 82, the virtual omnidirectional image can be viewed as if it were a futsal game from the virtual viewpoint 11 in the court 10.

  Here, seam carving used in the present embodiment will be described. The definition of a seam is an image sequence in which pixels that are not visually important are connected, and this is called a seam. A seam connected so as to connect the upper end to the lower end of an image is called a vertical seam. Seam carving also has a horizontal seam, but since only the vertical seam is used here, the vertical seam will be described. Seam carving is a technique used when enlarging or reducing an image, and the image can be reduced by deleting the seam. Moreover, it is applicable also to the expansion of an image by duplicating this seam. In seam carving, it is important to detect seams that are not visually important. To do this, we introduce the idea of image energy. The basic idea is that pixels that are sensitive to the human eye have high energy. An image to which energy values are assigned for each pixel is referred to as an energy image.

FIG. 8 is a schematic diagram showing an operation for obtaining a seam. First, energy calculation (FIG. 8B) is performed on a partial image in which no seam is defined. As a result, an energy image (FIG. 8C) is obtained. A vertical seam is obtained by performing a process for obtaining a vertical seam among the known seam carving processes on the energy image. Since the seam carving process is a known technique (refer to publicly known document 3), a detailed description is omitted here, but generally a boundary with low accumulated energy is obtained as a seam.
Known Document 3: Avidan, Shai, and Ariel Shamir. "Seam carving for content-aware image resizing." ACM Transactions on graphics (TOG). Vol. 26. No. 3. ACM, 2007.

  Then, a seam that can be used as a seam can be defined by superimposing the seam obtained on the energy image on the original partial image (FIG. 8A).

  Next, processing for combining two partial images will be described with reference to FIG. FIG. 9 is an explanatory diagram showing processing for combining two partial images. As described above, the two partial images 1 and 2 have a region where this part overlaps (this is called a boundary region). With respect to this boundary region, the vertical seam is obtained by performing only the processing for obtaining the vertical seam of the seam carving process using any one of the partial regions 1 and 2. The obtained vertical seam is used as a connecting line for connecting the two partial images 1 and 2. Here, a vertical seam has been described as an example. However, when images to be combined are arranged vertically, a horizontal seam is obtained. That is, a seam in a direction orthogonal to the direction in which two images are arranged is obtained. By doing so, the blending process described above becomes unnecessary, and the joining line is set at a place where it is difficult to notice with the eyes, so that the two partial images 1 and 2 can be joined naturally. Is possible.

  However, the following problems arise when the seam carving is simply performed by finding a joint line (seam). That is, there is a problem that it is easy to be noticed when there is a boundary (seam) between partial images in an area that is viewed by the viewer. Therefore, in the present embodiment, based on the assumption that the viewer should be looking at the direction in which the foreground such as a person moves, no seam is created in the direction in which the object that is the foreground image such as a person moves. I did it. FIG. 10 is a diagram illustrating an example in which an energy image is generated and processed based on the traveling direction of the foreground. As shown in FIG. 10 (A), the viewer looks toward the area in the moving direction of the person shown in the image (the area surrounded by the broken line), and if a seam is created in that area, the viewer is based on the seam. As a result, the combined line becomes conspicuous.

  This is because an energy image is simply generated from the target partial image shown in FIG. 10B, and a seam is formed in the region in the direction of movement of the person shown in the image. On the other hand, as shown in FIG. 10C, a second energy image is newly generated, and the second energy image is added so as not to make a seam in the foreground progression method. The second energy image is an energy image in which the energy of an area where the seam is not desired (here, the area in the direction in which the foreground is going to advance) is increased based on the foreground motion.

  Here, an energy image generation method will be described. 11 and 12 are explanatory diagrams showing a method for generating an energy image. For a certain boundary region, an optical flow is calculated for each pixel using an image of the past s (≧ 1) frames (FIG. 11A). Since the method for calculating the optical flow is known, detailed description thereof is omitted here.

Next, the optical flow obtained previously is clustered using K-means or the like regarding the position and direction (FIG. 11B). Subsequently, an energy image is obtained for each cluster. It is assumed that the energy image is an image having the same size as the boundary region. As shown in FIG. 12, if the average of the optical flows included in the cluster K _i (j is the class number, 1 ≦ j) is the optical flow of the cluster f _i and the center of gravity of K _i is g _i , the average (g _i An energy image is created according to a normal distribution with an x coordinate of + Δv) and a variance of 1.

Note that Δv = αf _i and the horizontal axis of the normal distribution is parallel to f _i as shown in FIG. α is a coefficient, and an appropriate numerical value may be given. However, it is desirable to give a value such that a moving object existing in the vicinity of the cluster K _i comes near g _i + Δv at the next time. In addition, the maximum value e _{max of} energy is determined in advance (such as 255), and is normalized so that the maximum value of energy obtained by the normal distribution is equal to e _max . Note that e _max may be changed according to the number of optical flows (Nf _i ) included in K _i and the magnitude (Pf _i ) of optical flows as e _max = Nf _i + Pf _i (FIG. 11 (C )).

Next, the energy images created in each cluster are added to obtain an energy image of the boundary region. Note that f _i obtained in FIG. 11C may be a maximum value, a minimum value, a median value, or the like of the optical flows included in the cluster K _i .

  Next, an operation for combining the two partial images shown in FIG. 9 will be described with reference to FIG. FIG. 13 is a flowchart showing an operation of combining the two partial images shown in FIG. First, the boundary determination part 39 produces | generates an energy image by the method mentioned above (step S301).

  Next, the boundary determination unit 39 adds the two energy images to generate a third energy image (step S302). Then, the boundary determining unit 39 obtains a vertical seam from the added energy image by the seam carving process (step S303). By this processing, the vertical seam shown in FIG. 9B is defined.

  Next, the boundary determination unit 39 determines a boundary using the obtained vertical seam (step S304). Subsequently, the boundary determination unit 39 cuts out the partial image based on the obtained boundary (step S305). This image clipping is performed on two partial images. Then, the boundary determination unit 39 combines the two partial images so as to overlap the two combined lines (step S306).

  Next, the boundary determination unit 39 determines whether or not the processing is completed depending on whether or not there is a boundary region that has not yet been processed (step S307). If there is an unprocessed boundary area, the process is repeated.

  By such processing, it is possible to prevent a partial image from being jointed in the direction in which the foreground such as a person moves, and thus it is possible to generate a virtual omnidirectional image that suppresses a decrease in viewing quality.

Second Embodiment
Next, an image processing apparatus according to a second embodiment of the present invention will be described. In the second embodiment, as in the first embodiment, a blending process is not performed, but a boundary that is not easily noticed when combining partial images is used. As described above, the following problems occur when seam carving simply seeks and joins seams. That is, there is a problem that it is easy to be noticed when there is a boundary (seam) between partial images in an area (line-of-sight direction) that is viewed by the viewer. Therefore, in the present embodiment, a seam is not created in the direction in which the viewer is paying attention (the region that is likely to be viewed by the viewer, which is approximately the center of the field of view). FIG. 14 is a diagram illustrating an example in which weighting is performed in the facing direction. As shown in FIG. 14A, if the viewer creates a seam in the direction of the line of sight, the combined line based on the seam becomes conspicuous.

  This is because the energy image is simply generated from the target partial image shown in FIG. 14B, and a seam is formed on the left side of the person shown in the image. On the other hand, as shown in FIG. 14C, a second energy image is newly generated, and the second energy image is added so as not to create a seam in the direction of the line of sight. The second energy image is an energy image in which the energy of a region where a seam is not desired (here, a region that is highly likely to be viewed by the viewer, that is, a region in the middle of the visual field) is increased. It is.

Here, an energy image generation method will be described. It is assumed that the point D = (dx, dy) on the image is known as information indicating the direction in which the head is facing, and the point D is included in the boundary region. This point D is acquired from information on the direction of the HMD 82. The boundary determination unit 39 creates an energy image according to a normal distribution with an average dx and a variance of 1. At this time, the horizontal axis of the normal distribution is assumed to be horizontal. Note that the maximum value e _{max of} energy is determined in advance and normalized so that the maximum value of energy obtained by the normal distribution is equal to e _max .

  Since the operation of combining the two partial images shown in FIG. 9 in the second embodiment is the same as the processing operation shown in FIG. 13, a detailed description thereof is omitted here.

  By such processing, it is possible to prevent a partial image from being jointed in the viewer's line-of-sight direction, and thus it is possible to generate a virtual omnidirectional image in which deterioration in viewing quality is suppressed.

<Third Embodiment>
Next, an image processing apparatus according to a third embodiment of the present invention will be described. In the third embodiment, as in the first embodiment, the blending process is not performed, but the geometric consistency is prevented from being lost when the joint (seam) is separated from the center of the partial image. This is a combination method. As described above, the following problems occur when seam carving simply seeks and joins seams. That is, there is a problem in that the geometrical consistency of the image is lost if the combined portion is generated at a position away from the center of the partial image. Therefore, in this embodiment, the coupling portion is provided at a position where the consistency is not lost. FIG. 15 is a diagram illustrating an example in which weighting is performed based on the distance from the center. Due to the nature of this method, a geometrically correct image is obtained when the joint (seam) is set at the center (FIG. 15A).

  Therefore, when the connecting portion (seam) is set by the conventional method, the connecting portion (seam) is formed at a position away from the center of the image (FIG. 15B). On the other hand, as shown in FIG. 15C, a second energy image is newly generated, and the second energy image is added so as not to create a seam at a position away from the center. The second energy image is an energy image in which the energy of a region where a seam is not desired (here, a region away from the center) is increased.

Here, an energy image generation method will be described. FIG. 16 is an explanatory diagram showing a method for generating an energy image. As shown in FIG. 16, in the partial images taken by the cameras C _i and C _{i + 1} , the positions of the virtual viewpoints are v _i and v _{i + 1} , respectively, and the midpoints of v _{i + 1} and v _i are c = (cx, cy). To do. An energy image is created according to a normal distribution with mean cx and variance 1. At this time, the horizontal axis of the normal distribution is assumed to be horizontal. Note that the maximum value e _{max of} energy is determined in advance and normalized so that the maximum value of energy obtained by the normal distribution is equal to e _max .

  The operation of combining the two partial images shown in FIG. 9 in the third embodiment is the same as the processing operation shown in FIG.

  By such processing, the seam of the partial image is set near the center of the image, so that it is possible to prevent the geometric consistency from being lost. An image can be generated.

<Fourth Embodiment>
Next, an image processing apparatus according to a fourth embodiment of the present invention will be described. In the fourth embodiment, as in the first embodiment, the blending process is not performed, but the geometric alignment is prevented from being lost when the joint (seam) is separated from the center of the partial image. This is a combination method. As described above, the following problems occur when seam carving simply seeks and joins seams. That is, there is a problem in that the geometrical consistency of the image is lost if the combined portion is generated at a position away from the center of the partial image. Therefore, in this embodiment, the coupling portion is provided at a position where the consistency is not lost. FIG. 17 is a diagram illustrating an example in which energy is also obtained based on the speed of the foreground motion. Due to the nature of this method, a geometrically correct image is obtained when a joint (seam) is set at the center.

  Therefore, when the joint portion (seam) is set by the conventional method, the joint portion (seam) is formed at a position away from the center of the image (FIG. 17B). On the other hand, as shown in FIG. 17C, a second energy image is newly generated, and the second energy image is added so as not to create a seam at a position away from the center. The second energy image is an energy image in which the energy of a region where a seam is not desired (here, avoiding a slowly moving foreground and avoiding a fast moving foreground) is increased. The fast moving foreground (for example, a ball) takes advantage of the fact that it is difficult to notice even if a double image or disappearance occurs.

  Here, an energy image generation method will be described. For a certain boundary region, an optical flow is calculated for each pixel from an image of a past s (≧ 1) frame. Then, the obtained optical flow is clustered by applying K-means or the like regarding the position and direction.

Next, for each cluster, the average of the optical flows included in the cluster K _j is set to f _j as in the third embodiment. At this time, when the size of f _j (| f _j |) is equal to or less than the threshold value tv (that is, in the case of a slowly moving object), the energy image is obtained in the same manner as that obtained by the above-described processing. On the other hand, when the size of f _j (| f _j |) is equal to or larger than tv (that is, in the case of a fast moving object), an energy image for this cluster K _j is not obtained. Then, the energy images created in each cluster are added to obtain an energy image of the overlapping region.

  The operation of combining the two partial images shown in FIG. 9 in the fourth embodiment is the same as the processing operation shown in FIG.

  By such processing, the seam of the partial image is set near the center of the image, so that it is possible to prevent the geometric consistency from being lost. An image can be generated. In particular, since a slowly moving object (foreground) is avoided and a fast moving object (foreground) is avoided, a virtual omnidirectional image in which deterioration in viewing quality is suppressed can be generated.

  As described above, when joining two partial images, the joint of the images is set to a position that is not noticed by the viewer, so that a virtual omnidirectional image that suppresses degradation in viewing quality is generated. can do. Further, since the geometric consistency is prevented from being lost when the two partial images are combined, a virtual omnidirectional image in which deterioration of viewing quality is similarly suppressed can be generated.

  By repeating the process of combining two partial images by the method described above, a virtual omnidirectional image as shown in FIG. 18 can be obtained. FIG. 18 is a diagram illustrating an example in which a virtual omnidirectional image is generated. The left side of FIG. 18 shows an example of a virtual omnidirectional image according to the prior art. The right side of FIG. 18 shows that a high-quality virtual celestial sphere image can be generated by applying the seam carving process and optimizing the position of the joint (seam).

  FIG. 19 is an explanatory diagram illustrating an example of a partial image cut out from an input image of the camera. As shown in FIG. 19, in the left figure, the person who is the subject is reflected in all three cameras, and thus a triple image is generated. On the other hand, as shown in the right figure, the area of the partial image can be enlarged by the seam carving process, so that the person who is the subject can be contained in one partial image, and a triple image is generated. This can be prevented.

  You may make it implement | achieve all or one part of the image processing apparatus in embodiment mentioned above with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be for realizing a part of the functions described above, and may be a program capable of realizing the functions described above in combination with a program already recorded in the computer system. It may be realized using hardware such as PLD (Programmable Logic Device) or FPGA (Field Programmable Gate Array).

  As mentioned above, although embodiment of this invention has been described with reference to drawings, the said embodiment is only the illustration of this invention, and it is clear that this invention is not limited to the said embodiment. is there. Therefore, additions, omissions, substitutions, and other modifications of the components may be made without departing from the technical idea and scope of the present invention.

  When two images are combined, they are arranged at positions where it is difficult to notice the seam of the images, so that it is also possible to apply to applications where it is indispensable to generate a virtual omnidirectional image that suppresses degradation in viewing quality.

  DESCRIPTION OF SYMBOLS 10 ... Coat, 11 ... Virtual viewpoint, 1 ... Image processing system, 2 ... Spherical camera, 3 ... Camera group, 5 ... Display apparatus, 30 ... Image processing Device 20 ... background image 6 ... image server 7 ... network 8 ... viewing device 81 ... user terminal 82 ... HMD 9 ... viewing system 31 ... Object analysis unit, 32 ... Depth acquisition unit, 33 ... Composition information acquisition unit, 34 ... Image input unit, 35 ... Image cutout unit, 36 ... Image composition unit, 37 Display processing unit 38 ... Input unit 39 ... Boundary determination unit 301 ... Foreground image storage unit 302 ... Background image storage unit 303 ... Object information storage unit 304 ..Composite information table

Claims (8)

2 of the 1st image which is an image acquired from the 1st image, and the 2nd image which is an image acquired from the 2nd image, and is an image with which a part of image pick-up field is common to the 1st image. An image processing apparatus for determining a connection line for combining the two images at a common portion when combining two images,
An image processing apparatus comprising a combined line determination unit that determines to arrange the combined line while avoiding an area that is likely to be viewed by a person viewing the image. 2 of the 1st image which is an image acquired from the 1st image, and the 2nd image which is an image acquired from the 2nd image, and is an image with which a part of image pick-up field is common to the 1st image. An image processing apparatus for determining a connection line for combining the two images at a common portion when combining two images,
In the common part, a first energy image generating means for generating a first energy image in which a pixel that is sensitive to the human eye is defined as having a large energy;
A search for pixels with low energy in the first energy image, and a combination line determination means for determining to arrange the connection lines by obtaining a series of pixels with low energy in the common part; and
The coupling line determination means includes
An image processing apparatus that determines the coupling line while avoiding an area that is highly likely to be viewed by a person viewing the image. An area detecting means for detecting an area viewed by a person viewing the image in the common part of the image;
Second energy image generation means for generating a second energy image in which the energy of the detected area is increased based on a detection result by the area detection means;
A third energy image generating means for generating a third energy image by adding the first energy image and the second energy image;
The image processing apparatus according to claim 2, wherein the coupling line determination unit determines the coupling line by obtaining a series of pixels having the low energy based on the third energy image.   The image processing apparatus according to claim 3, wherein the joining line determination unit uses a seam in a direction orthogonal to a direction arranged when joining the two images of the common portion obtained by the seam carving process as a joining line. .   The image processing apparatus according to claim 1, wherein the two images are used when generating a virtual omnidirectional image viewed from a virtual viewpoint provided at a predetermined position. 2 of the 1st image which is an image acquired from the 1st image, and the 2nd image which is an image acquired from the 2nd image, and is an image with which a part of image pick-up field is common to the 1st image. An image processing method performed by an image processing apparatus for determining a connection line for combining the two images at a common portion when combining two images,
An image processing method comprising: a joint line determination step for deciding to place the joint line while avoiding an area where a person who is viewing the image is likely to be viewing. 2 of the 1st image which is an image acquired from the 1st image, and the 2nd image which is an image acquired from the 2nd image, and is an image with which a part of image pick-up field is common to the 1st image. An image processing method performed by an image processing apparatus for determining a connection line for combining the two images at a common portion when combining two images,
In the common part, an energy image generating step for generating a first energy image in which a pixel that is sensitive to human eyes is defined as having high energy;
A bond line determining step for determining the bond line in the common part while searching for an image with a low energy of the first energy image;
The coupling line determination step includes:
An image processing method for determining the coupling line while avoiding an area that is highly likely to be viewed by a person viewing the image.   An image processing program for causing a computer to function as the image processing apparatus according to claim 1.