JP7458735B2 - Image processing device, image processing method, and program - Google Patents

Description

The present invention relates to an image processing device, an image processing method, and a program, and more particularly to display control of a virtual viewpoint image generated based on multiple images.

Patent Document 1 describes that in a multi-view image content viewing system, it is determined whether the image content includes an image in a "non-display area" that should not be displayed to the viewer. . If it is determined that the image content includes a "non-display area", processing is performed to change the image including the image of the non-display area to an image from a different viewpoint that does not include the non-display area. This prevents the non-display area from being displayed to the viewer when viewing the image content.

Japanese Patent Application Publication No. 2012-120031

However, when the technology disclosed in Patent Document 1 is applied to a virtual viewpoint image generation system in which a user can arbitrarily specify a viewpoint, changing the viewpoint may give the user a sense of discomfort. That is, in Patent Document 1, when changing the viewpoint so that the non-display area is not displayed, the image is switched to a predetermined image that is significantly different from the image displayed before changing the viewpoint. Therefore, switching of images regardless of the user's viewpoint operation may give the user a sense of discomfort depending on the content, and as a result, the usability of the viewpoint operation may be reduced.

An object of the present invention is to reduce the sense of discomfort given to a user while preventing display of a specific area regarding display control of a virtual viewpoint image generated based on a plurality of images.

In order to achieve the above object, one form of the present invention is an image processing device that generates a virtual viewpoint image based on images of multiple viewpoints, the device comprising: a viewpoint input unit that inputs virtual viewpoint information in the virtual viewpoint image; , a display determination unit that determines whether or not the viewpoint specified by the virtual viewpoint information satisfies a prohibition condition regarding an object that should not be displayed; an output control unit that controls to output an image according to a viewpoint specified before the specification of the viewpoint when it is determined that the prohibition condition is satisfied , and the prohibition condition is set as a target for generating a virtual viewpoint image. A space is divided into a plurality of regions, and each of the divided regions is defined, and the position and direction of the viewpoint for viewing the object that should not be displayed are defined .

According to the above embodiment, regarding the display control of the virtual viewpoint image generated based on a plurality of images, it is possible to reduce the sense of discomfort given to the user while not displaying a specific area.

1 is a block diagram showing a virtual viewpoint image generating device according to a first embodiment of the present invention. 2 is a block diagram showing a hardware configuration of the virtual viewpoint image generating device shown in FIG. 1 . 1 is a diagram illustrating a system configuration for capturing images input to a virtual viewpoint image generation device according to the present embodiment. FIG. 4 is a diagram illustrating divided regions obtained by dividing a target three-dimensional space and in which prohibited viewpoint information is defined, according to an embodiment of the present invention. FIG. 3 is a diagram illustrating an example of setting prohibited viewpoint information according to an embodiment of the present invention. FIG. 7 is a diagram illustrating another example of setting prohibited viewpoint information in divided areas according to the embodiment of the present invention. 7 is a flowchart illustrating image output determination processing based on a virtual viewpoint according to an embodiment of the present invention. FIG. 2 is a block diagram showing a virtual viewpoint image generation device according to a second embodiment of the present invention. 7 is a flowchart showing image output determination processing based on a virtual viewpoint according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

(First embodiment)
A first embodiment of an image processing device according to the present invention relates to a virtual viewpoint image generation device. Specifically, in a virtual viewpoint image generation device, a part of an image is generated based on a preset condition (hereinafter also referred to as a "prohibition condition") associated with the viewpoint of a virtual camera (hereinafter also referred to as a virtual viewpoint). This controls the hiding of .

FIG. 1 is a block diagram showing a virtual viewpoint image generation device according to a first embodiment of the present invention. The virtual viewpoint image generation device 10 includes an image input section 100, a virtual camera viewpoint input section 101, a virtual viewpoint image generation section 102, an image output section 103, an image display prohibition condition setting section 104, an image display judgment section 105, and an image output control section. 106.

The image input unit 100 receives images captured by multiple cameras (multiple viewpoint images) from each camera, and outputs the received images to the virtual viewpoint image generation unit 102. The virtual camera viewpoint input unit (viewpoint input unit) 101 inputs information on the viewpoint of the virtual camera through a user's operation or the like via an operation unit (not shown), and outputs the input information to the virtual viewpoint image generation unit 102 and the image display determination unit 105. Here, the information on the viewpoint of the virtual camera represents, but is not limited to, the attitude, position, angle of view, etc. of the virtual camera.

The virtual viewpoint image generating unit 102 generates an image seen from the viewpoint of the virtual camera based on the images obtained by the above-mentioned multiple cameras and the viewpoint information of the virtual camera, and outputs it to the image output unit 103. The image generating method is not particularly limited, but the image can be generated by the following method, for example. First, each image from the multiple cameras is separated into a foreground and a background. This separation can be performed by comparing with an image of only the background captured in advance. In addition, a three-dimensional model of an object existing in the target space is calculated using a volume intersection method based on the foreground image obtained by the multiple cameras and the position and orientation information of each camera. Then, based on the viewpoint information of the virtual camera, the appearance of the three-dimensional model of the object is calculated by three-dimensional calculation, and the color of the foreground image is applied to the three-dimensional model to generate a foreground image. For the background, a background model to which a background image calculated in advance is applied is used, and the background image is applied to the three-dimensional model based on the position and orientation of each camera. Furthermore, a background image is generated by calculating an image seen from the viewpoint of the virtual camera by three-dimensional calculation. Then, the background image and the foreground image are synthesized to generate the final virtual viewpoint image. Here, the viewpoint of the virtual camera (virtual viewpoint) is set to any position in the generated virtual viewpoint image, i.e., a position that is unrelated to the positions of the multiple cameras used.

The image output unit 103 outputs the virtual viewpoint image input from the virtual viewpoint image generation unit 102 to, for example, an image viewing device. Although the output method is not limited, for example, the HDMI (registered trademark) method (High-Definition Multimedia Interface) can be used. Further, the image output unit 103 stops or starts image output based on the control of the image output control unit 106.

The image display prohibition condition setting unit 104 sets conditions under which image output is prohibited in the target space of virtual viewpoint image generation. Details of this setting method will be described later with reference to FIG. 3 and subsequent figures. The image display determining unit 105 determines whether or not to stop image output based on the image display prohibition condition set by the image display prohibition condition setting unit 104, and outputs the determination result to the image output control unit 106. . Details of this determination will be described later with reference to FIG. 7 and the like.

The image output control unit 106 controls the image output of the image output unit 103 based on the judgment result of the image display judgment unit 105. When stopping the image output, the image output unit 103 may stop the image signal itself and output a black screen or the frame immediately before the virtual viewpoint. In this embodiment, the immediately previous frame is output. This makes it possible to produce an image whose content is closest to the image to be viewed from the virtual viewpoint at that time, and minimizes the sense of discomfort felt by the viewer when the displayed image is changed. It also reduces the sense of discomfort felt by the viewer when the image signal is interrupted or when the screen is switched to black.

FIG. 2 is a block diagram showing the hardware configuration of the virtual viewpoint image generation device 10 shown in FIG. 1. The virtual viewpoint image generation device 10 includes a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, a display section 215, an operation section 216, a communication I/F 217, and a bus 218 as a hardware configuration.

The CPU 211 controls the entire virtual viewpoint image generation device 10 using computer programs and data stored in the ROM 212 and RAM 213. Thereby, the functions of each of the elements 100 to 106 of the virtual viewpoint image generation device 10 shown in FIG. 1 are realized. Note that the virtual viewpoint image generation device 10 may include one or more dedicated hardware different from the CPU 211, and the dedicated hardware may execute at least part of the processing by the CPU 211. Examples of specialized hardware include ASICs (Application Specific Integrated Circuits), FPGAs (Field Programmable Gate Arrays), and DSPs (Digital Signal Processors). The ROM 212 stores programs that do not require modification. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, data supplied from the outside via the communication I/F 217, and the like. The auxiliary storage device 214 is composed of, for example, a hard disk drive, and stores various data such as image data and audio data.

The display unit 215 is configured with, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) for the user to perform operations related to the virtual viewpoint image generation device 10. The operation unit 216 includes, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and inputs various instructions to the CPU 211 in response to user operations. The CPU 211 operates as a display control unit that controls the display unit 215 and an operation control unit that controls the operation unit 216. The operation unit 216 and the display unit 215 realize part of the functions related to input of virtual camera viewpoint information by the virtual camera viewpoint input unit 101 shown in FIG.

The communication I/F 217 is used for communication with devices external to the virtual viewpoint image generation device 10. For example, when the virtual viewpoint image generation device 10 is connected to an external device by wire, a communication cable is connected to the communication I/F 217. When the virtual viewpoint image generation device 10 has a function of wirelessly communicating with an external device, the communication I/F 217 includes an antenna. The bus 218 connects each part of the virtual viewpoint image generation device 10 and transmits information. This communication I/F 217 realizes part of the image input/output functions of the image input section 100 and the image output section 103 shown in FIG.

In this embodiment, the display unit 215 and the operation unit 216 are assumed to exist inside the virtual viewpoint image generating device 10, but at least one of the display unit 215 and the operation unit 216 may exist as a separate device outside the virtual viewpoint image generating device 10.

Furthermore, the virtual viewpoint image generation device 10 may include one or more dedicated hardware or GPU (Graphics Processing Unit) different from the CPU 211, and at least part of the processing may be performed by the GPU or dedicated hardware. good. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits) and DSPs (Digital Signal Processors).

FIG. 3 is a diagram showing a system for capturing images input to the virtual viewpoint image generation device 10 of this embodiment. The example shown in FIG. 3 shows that the target three-dimensional space 300 is photographed by eight cameras 11 to 18, respectively. The object to be photographed is assumed to be a competition such as soccer, and in this case, the object three-dimensional space 300 includes a soccer stadium where the players compete. Of course, the subject and three-dimensional space to be photographed are not limited to these, and may include, for example, a concert with several people in a studio, or an outdoor location such as a golf course. Good too.

As shown in FIG. 3, the target three-dimensional space 300 is defined as a space of -100m to 100m along the x-axis of the zyz coordinate system, -50m to 50m along the y-axis, and 0m to 10m along the z-axis. has been done. In this defined space, control is performed to hide a portion of the virtual viewpoint image, which will be described later with reference to FIG. 4 and subsequent figures.

In this embodiment, an image display prohibition condition associated with a virtual viewpoint specified by a user or the like is set in advance, and based on this prohibition condition, non-display control of a part of the image (prohibited area) is performed. The image display prohibition condition setting unit 104 (FIG. 1) sets this prohibition condition. In this embodiment, settings can be made using two setting methods. The first method (hereinafter also referred to as "first method") is a method of setting a prohibited viewpoint for a specified virtual viewpoint. The second method (hereinafter also referred to as "second method") is a method in which a part of the target three-dimensional space is set as a prohibited area, and this prohibited area is hidden in relation to the virtual viewpoint. be.

The first method is to divide the target three-dimensional space 300 shown in FIG. (hereinafter also referred to as "prohibited viewpoint information") are defined in advance. Then, if the virtual viewpoint specified by the user or the like satisfies the prohibited viewpoint information specified in the divided area including the position of the virtual viewpoint, that is, if the prohibition condition is satisfied, the image viewed from that virtual viewpoint is hidden. Take control.

FIG. 4 is a diagram illustrating divided regions obtained by dividing the target three-dimensional space 300 and in which prohibited viewpoint information is defined. In this embodiment, the target three-dimensional space 300 is divided so that the divided regions become rectangular parallelepipeds of 10 m x 10 m x 10 m. As a result, the target three-dimensional space 300 is divided into 200 regions (divided regions). Then, prohibited viewpoint information is set in advance for each of the divided regions obtained by dividing in this way.

FIG. 5 is a diagram illustrating setting of prohibited viewpoint information in a divided area according to the first method of the present embodiment, and shows an arbitrary divided area 300k in the target three-dimensional space 300. In the divided region 300k, virtual axes Ix, Iy, and Iz are defined that pass through the center I0 of the rectangular parallelepiped and are parallel to the x, y, and z axes, respectively. Using these three virtual axes, the divided region 300k can be divided into eight regions of the first to eighth quadrants, which are octants. These eight areas can be specified, for example, in correspondence with the coordinates (xk, yk, zk), ..., (xk+1, yk+1, zk+1) of the vertices of the divided area 300k, respectively. . Alternatively, one area may include a plane defined by the 90° angle between the Ix axis and the Iy axis, a plane defined by the 90° angle between the Ix axis and the Iz axis, and a 90° angle between the Iy axis and the Iz axis. It can be specified as a plane defined by ° and an area surrounded by.

For divided region 300k, prohibited viewpoint information is defined as follows. In the prohibited viewpoint information, the position of the viewpoint can be defined by the above eight coordinates. That is, the position of the viewpoint is defined as a position inside a region determined by the magnitude relationship with the x, y, and z values of each of the above eight coordinates. In addition, the direction (orientation) of the viewpoint can be defined by whether it is facing toward any of the regions in the first to eighth quadrants from the center I0 of the above rectangular parallelepiped.

Specifically, first, an area of an object that should not be displayed in the target three-dimensional space 300 (hereinafter, "prohibited area"; not shown in FIG. 5) is identified. For example, a user uses a UI showing the three-dimensional space of a stadium to enclose an object that should not be displayed and set a prohibited area. Then, for each of the set prohibited areas, it is determined whether the prohibited area is visible from the center IO of the divided area from each divided area 300k. If it is visible, a position inside the divided area is defined as the position of the prohibited viewpoint information. In addition, the quadrant that includes the visible direction among the first to eighth quadrants is defined as the viewpoint direction. Note that, depending on the divided area, the prohibited area may not be visible, and in that case, it goes without saying that prohibited viewpoint information is not defined or set for that divided area.

When a virtual viewpoint Ie is specified in one of the divided regions in which prohibited viewpoint information is defined as described above, the direction of the virtual viewpoint Ie in that divided region is based on the prohibition condition, as will be described later in FIG. will be judged. In other words, does the direction of the virtual viewpoint Ie have the same (parallel) vector as the direction of the viewpoint defined in the divided region (direction from the center IO toward any of the first to eighth quadrants)? It is determined whether or not. When they have the same vector, it is determined that the virtual viewpoint Ie falls within the range indicated by the prohibited viewpoint information of the divided area, that is, satisfies the prohibition condition (S1001 in FIG. 7).

In this embodiment, as described above, the user can set the prohibited viewpoint information in advance via a GUI (not shown) or the like. For example, the user selects and specifies an area to be set as a prohibited area in the target three-dimensional space 300 displayed in the GUI, and sets the prohibited viewpoint information for each divided area 300k of the target three-dimensional space 300 as described above.

Note that a plurality of divided regions in the target three-dimensional space 300 may be selected and orientations that are prohibited at once (prohibited viewpoint information) may be set. Further, for example, for a divided area in which prohibited viewpoint information is not set, the next process may be performed without determining whether or not the prohibition condition described later with reference to FIG. 7 is satisfied.

Further, in the above example, the size of the rectangular parallelepiped that is the divided area is 10 m on each side, and the number of divisions in the direction is 8, but it is needless to say that the present invention is not limited to this. By increasing or decreasing the number of divisions in the rectangular parallelepiped size and orientation of the divided regions, it is possible to increase or decrease the resolution of the viewpoint to be set.

Further, in the above example, prohibited viewpoint information is set using a rectangular parallelepiped as a unit of divided region, but the shape is not limited to this. For example, it may be spherical or cylindrical. Furthermore, although the target three-dimensional space 300 is equally divided into a certain size to define divided regions, the present invention is not limited to this form. For example, in the case of soccer, the size of a rectangular parallelepiped that corresponds to the field is divided into other regions. It can also be set smaller. This is based on the fact that signs outside the field are often larger than the players on the field.

According to the first method of setting prohibition conditions explained above, when specifying a virtual viewpoint, by simply referring to the prohibition viewpoint information specified for the corresponding divided area, the prohibition area can be set with respect to the specified virtual viewpoint. You can determine whether it exists or not. Thereby, as described in Patent Document 1, when generating a virtual viewpoint image, it is not necessary to determine whether a prohibited area exists in the image, and the processing load can be reduced.

Next, a second method for setting prohibition conditions will be explained. In this method, a prohibition condition is set by setting an area (prohibition area) in which generation of a virtual viewpoint image is prohibited in the target three-dimensional space 300. Similar to the first method described above, this method is a process that is set in advance before generating a virtual viewpoint image.

For example, similar to the first method, the user uses a UI that shows the three-dimensional space of the stadium to surround an area in which the virtual viewpoint image generation is desired to be prohibited. The area formed by this enclosing is defined as a prohibited area. Information on the defined prohibited area can be expressed as coordinates in the target three-dimensional space 300 (FIG. 3).

Figure 6 is a diagram for explaining the setting of prohibited areas according to this embodiment, showing prohibited areas defined in the target three-dimensional space 300 shown in Figure 3. Figure 6 shows an example in which three prohibited areas 601, 602, and 603 are defined.

When a virtual viewpoint Ie is specified in the target three-dimensional space 300 in which a prohibited area is set as described above, the image display determination unit 105 determines whether the virtual viewpoint, which will be described later in FIG. 7, satisfies the prohibition condition. The judgment is as follows.

The image display determination unit 105 uses coordinate transformation to place the vertices of each of the set prohibited areas 601 to 603 in the image captured by the virtual camera (viewed from the virtual viewpoint) based on the position and orientation of the virtual viewpoint. project. Note that in the following, in order to simplify the explanation, determination of prohibition conditions for one prohibition area will be described. First, the coordinate values of the vertices of the prohibited area are converted to the coordinate values of the camera coordinate system of the virtual camera. The rotation matrix R is determined by the attitude (orientation) of the virtual camera in the coordinate system of the target three-dimensional space 300, and the translation vector t is determined by the position of the camera. Note that the rotation matrix R is a 3×3 matrix. Using the coordinate values Xw, Yw, and Zw of the world coordinate system, the coordinate values Xc, Yc, and Zc of the camera coordinate system are calculated by the following equation (1).

Next, the three-dimensional coordinates of the camera coordinate system are transformed into two-dimensional coordinates of the camera image by perspective projection transformation.

If the coordinates on the camera image are (u, v), then they can be calculated from the three-dimensional coordinates Xc, Yc, and Zc in the camera coordinate system using the following formula (2).

Thereby, the coordinates of each vertex of the prohibited area projected onto the image captured by the virtual camera can be obtained. Then, based on these coordinates, it is determined whether a prohibited area exists in the image of the virtual camera. The image display determination unit 105 determines that the prohibited area exists, that is, that the prohibition condition is satisfied (S1001 in FIG. 7).

According to the second method described above, since this method sets a prohibited area in the target three-dimensional space 300, it is compared to setting a planar area as disclosed in Patent Document 1. , it becomes possible to set the area in more detail.

Note that in the above judgment, the judgment may be made not only based on whether or not a prohibited area exists in the image, but also depending on the size and position of the prohibited area. For example, generation of the image may be stopped only when the range in which the prohibited area exists is larger than a predetermined threshold value.

In this embodiment, one of the prohibition conditions set by the two setting methods described above is used. For example, if the number of prohibited areas in the target three-dimensional space 300 is greater than a predetermined value, the prohibition conditions by the first method may be used. As another example, one of the prohibition conditions may be set depending on the content of the image to be displayed. Also, for example, the prohibition conditions may be changed every predetermined time of the content image. In that case, the prohibition conditions may be changed depending on the shooting time linked to the content, etc.

FIG. 7 is a flowchart showing image output determination processing based on a virtual viewpoint according to the present embodiment. Note that the series of processes shown in the flowchart is performed by the CPU 211 of the virtual viewpoint image generation device 10 loading a control program stored in the ROM 212 or the auxiliary storage device 214 into the RAM 213 and executing it. Alternatively, some or all of the functions of the steps in the flowchart may be realized by hardware such as an ASIC or an electronic circuit. Further, the symbol "S" in the description of the flowchart means a "step" in the flowchart. The same applies to other flowcharts.

The virtual camera viewpoint input unit 101 receives virtual camera viewpoint information (virtual viewpoint information), and outputs the virtual camera viewpoint information to the image display determination unit 105 (S1000). The image display determining unit 105 determines whether the virtual viewpoint satisfies the prohibition condition based on the information on the viewpoint of the virtual camera and the information on the prohibition condition described above (S1001).

If the virtual viewpoint satisfies the prohibition condition, the image display determination unit 105 notifies the image output control unit 106 of the determination to prohibit image display. In response, the image output control unit 106 controls the image output unit 103 to stop outputting the image from that viewpoint and controls the image output unit 103 to output the previous frame (S1002). That is, the image output control unit 106 outputs the frame generated and output at the viewpoint before moving to the virtual viewpoint. As a result, the person viewing the image re-views the image that was viewed before attempting to switch the viewpoint, and the sense of discomfort given to the viewer can be reduced compared to the case of switching the viewpoint.

On the other hand, if the specified virtual viewpoint does not satisfy the prohibition condition, the image display determination unit 105 notifies the image output control unit 106 of the determination that image output is permitted. In response to this, the image output control unit 106 does not perform any particular control over the image output unit 103, and the image output unit 103 outputs the frame generated by the virtual viewpoint image generation unit 102 as is. (S1003).

Note that in the above-described embodiment, a configuration in which there is one image output unit 103 has been described, but in the case where there is a plurality of image output units, the processing based on the above-described judgment may be controlled independently of each other. For example, it is also possible to control only images to be disclosed to the outside so that objects that meet the prohibition conditions are not displayed.

According to the first embodiment described above, it is possible to control the image output from the virtual viewpoint based on the prohibition condition based on the viewpoint of the virtual camera, which is set in advance by the user. As a result, for example, it becomes possible to perform control so that objects that the image content provider does not want to display are not displayed.

(Second embodiment)
The virtual viewpoint image generation system according to the second embodiment of the present invention performs processing such as removing an image of a target (object) in a prohibited area in an image from a virtual viewpoint in generating an image from a virtual viewpoint. Output.

FIG. 8 is a block diagram showing a virtual viewpoint image generation device 20 according to the second embodiment of the present invention. Elements that perform the same functions as those in the first embodiment are given the same numbers and their explanations are omitted.

In addition to the functions described in the first embodiment, the virtual viewpoint image generating unit 110 of this embodiment controls image generation based on the prohibited conditions set by the image display prohibition condition setting unit 104 and the virtual camera viewpoint input from the virtual camera viewpoint input unit 101. Specifically, when generating the foreground, objects present in the area set by the prohibited conditions are removed before generation so that the object is not displayed in the virtual viewpoint image that is finally generated. As described above, this object is an object that should not be displayed, and a prohibited area has been set by the user to surround this object.

The process of removing objects from the foreground will be specifically described below. As described above in the first embodiment, a prohibited area is set in the target three-dimensional space 300 as a method for setting the image display prohibition condition. In addition, the foreground image generation method calculates the appearance of the 3D model of the target space through 3D calculations based on information on the viewpoint of the virtual camera, and then applies the color of the foreground image to the 3D model to generate the foreground image. generate. Whether the three-dimensional model to be generated as a foreground image exists in the prohibited area can be determined by comparing the respective three-dimensional coordinates. As a result of comparing the 3D coordinates, the 3D model within the prohibited area is not targeted for foreground generation, that is, by not projecting it onto the foreground image from the virtual camera viewpoint, the 3D model is extracted from the foreground image. Can be removed. Note that in addition to excluding all of one object as a three-dimensional model, for example, in the case of a three-dimensional model composed of a point group, it is also possible to exclude only points included in a prohibited area. As a result, a virtual viewpoint image in which only objects within the prohibited area are not displayed can be displayed to the user, and even if the prohibited area is set, the user can operate and view the virtual viewpoint image as usual.

In the above example, the method of removing the object included in the prohibited area from the foreground has been described, but the object may be removed from the background, or from both the foreground and the background. It is also possible to display only the background or only the foreground so that objects in the prohibited area are not displayed.

In addition, instead of completely removing the three-dimensional object, it is possible to display it after processing it by thinning out the points of the three-dimensional object and reducing the resolution of the object displayed in the virtual viewpoint image.

Figure 9 is a flowchart showing the image output determination process based on the virtual viewpoint according to this embodiment.

The virtual viewpoint image generation unit 110 receives images from multiple cameras from the image input unit 100, separates the foreground and background, and generates a three-dimensional model. Further, a virtual camera viewpoint is received from the virtual camera viewpoint input unit 101 (S2000). Next, the virtual viewpoint image generation unit 110 compares the prohibited area set in advance by the image display prohibition condition setting unit 104 and the three-dimensional model using three-dimensional coordinates (S2001). Then, as a result of the comparison, the three-dimensional model existing in the prohibited area is excluded from the foreground generation target (S2002).

Furthermore, a foreground image is generated based on the three-dimensional model other than the excluded one, and a background image is further generated. The foreground image and the background image are combined to generate a virtual viewpoint image and output to the image output unit 103 (S2003). The image output unit 103 outputs the generated virtual viewpoint image (S2004).

According to the second embodiment of the present invention described above, when the specified virtual viewpoint satisfies the prohibition condition, an image excluding objects that should not be displayed is displayed. As a result, compared to a mode in which the object is not displayed by switching the viewpoint, it is possible to reduce the sense of discomfort given to the user. Further, objects in the prohibited area can be hidden while giving the user the same impression in terms of operability and display images as normal viewing of virtual viewpoint images.

(Third embodiment)
In the third embodiment of the present invention, a process will be described in which the operation of the virtual viewpoint is controlled based on a prohibition condition based on the viewpoint of the virtual camera set in advance by the user, so that it is impossible to input a viewpoint from which image output is prohibited.

The configuration of the virtual viewpoint image generation device according to this embodiment is almost the same as the configuration shown in FIG. 8, and only the function of the virtual viewpoint image generation unit 110 is different. Below, only the functions of the virtual viewpoint image generation unit 110 will be described.

The virtual viewpoint image generation unit 110 determines whether the virtual camera viewpoint received from the virtual camera viewpoint input unit 101 satisfies the prohibition condition set by the image display prohibition condition setting unit 104. If the prohibition condition is not satisfied and output is permitted, a virtual viewpoint image at the designated virtual camera viewpoint is directly generated. When it is determined that the prohibition condition is satisfied, a virtual viewpoint image is generated at the virtual camera viewpoint while maintaining the virtual camera viewpoint that was previously permitted to be output. As a result, even if the operating user specifies a viewpoint that satisfies the prohibition conditions, the user cannot move to that viewpoint. In other words, it is possible to restrict movement to viewpoints that satisfy the prohibition conditions.

Note that in the above example, the judgment criterion was whether the virtual viewpoint satisfies the prohibition condition set by the above-mentioned first method, but regarding the prohibition condition set by the above-mentioned second method, Of course, it is also possible to determine whether the virtual viewpoint satisfies the prohibition conditions.

As described above, according to the third embodiment of the present invention, the operation of the virtual viewpoint can be controlled according to the prohibition condition based on the virtual viewpoint set in advance by the user, and the viewpoint from which image output is prohibited can be made impossible to input.

(Other embodiments)
In the embodiment described above, a method was described in which the user sets the prohibition condition, but the method is not limited to this, and the prohibition condition may be set automatically. For example, by recognizing characters, etc., we can recognize billboards that are different from those of the sponsors or posts that are inappropriate for support, or detect abnormal behavior such as being naked, and set prohibition conditions based on these. Good too.

The present invention provides a system or device with a program that implements one or more of the functions of the embodiments described above via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. This can also be achieved by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

10, 20 Virtual viewpoint image generation device 11 to 18 Camera 100 Image input section 101 Virtual camera viewpoint input section 102, 110 Virtual viewpoint image generation section 103 Image output section 104 Image display prohibition condition setting section 105 Image display judgment section 106 Image output control Part 300 Target three-dimensional space 601, 602, 603 Prohibited area

Claims (10)

An image processing device that generates a virtual viewpoint image based on images from multiple viewpoints,
a viewpoint input unit that inputs virtual viewpoint information in the virtual viewpoint image;
a display determination unit that determines whether a viewpoint specified by the virtual viewpoint information satisfies a prohibition condition regarding an object that should not be displayed;
When the display determining unit determines that the specified viewpoint satisfies the prohibition condition, an output control unit controls to output an image according to a viewpoint specified before the specified viewpoint;
Equipped with
The prohibition condition is defined for each divided region in which a target space for generating a virtual viewpoint image is divided into a plurality of regions, and is defined as viewing the object that should not be displayed, the position of the viewpoint, and the direction of the viewpoint. An image processing device characterized in that : An image processing device that generates a virtual viewpoint image based on images from multiple viewpoints,
a viewpoint input unit that inputs virtual viewpoint information in the virtual viewpoint image;
a display determination unit that determines whether virtual viewpoint information input by the viewpoint input unit satisfies a prohibition condition regarding an object that should not be displayed;
If the display determining unit determines that the information on the virtual viewpoint satisfies the prohibition condition, it controls to process the image of the object that should not be displayed in the image corresponding to the virtual viewpoint and output the image. an output control section;
Equipped with
The prohibition condition is defined for each divided region in which a target space for generating a virtual viewpoint image is divided into a plurality of regions, and is defined as viewing the object that should not be displayed, the position of the viewpoint, and the direction of the viewpoint. An image processing device characterized in that : The image processing apparatus according to claim 2, wherein the processing is to remove the object that should not be displayed from the image according to the virtual viewpoint. 3. When the display determining section determines that the information on the virtual viewpoint satisfies the prohibition condition, the output controlling section removes the image of the object that should not be displayed from the foreground or the background. The image processing device described in . 3. The image processing apparatus according to claim 2, wherein the processing is to reduce the resolution of the image of the target that should not be displayed from the image corresponding to the virtual viewpoint. An image processing device that generates a virtual viewpoint image based on images from multiple viewpoints,
a viewpoint input unit that inputs virtual viewpoint information in the virtual viewpoint image;
a display determination unit that determines whether a viewpoint specified by the virtual viewpoint information satisfies a prohibition condition regarding an object that should not be displayed; a display judgment unit that is defined for each divided region, in which a space is divided into a plurality of regions, and defined as a viewpoint position and a viewpoint direction for viewing the object that should not be displayed;
If the display determining unit determines that the specified viewpoint is within the range of the position and direction of the viewpoint that views the object that should not be displayed, it stops outputting the image according to the viewpoint. an output control section to
An image processing device comprising: An image processing method for generating a virtual viewpoint image based on images of multiple viewpoints, the method comprising:
an image display determination step of determining whether a viewpoint specified by virtual viewpoint information in the virtual viewpoint image satisfies a prohibition condition regarding an object that should not be displayed;
If the image display determining step determines that the designated viewpoint satisfies the prohibition condition, an output control step of controlling to output an image according to the viewpoint specified before the designation of the viewpoint;
has
The prohibition condition is defined for each divided region in which a target space for generating a virtual viewpoint image is divided into a plurality of regions, and is defined as viewing the object that should not be displayed, the position of the viewpoint, and the direction of the viewpoint. An image processing method characterized by: An image processing method for generating a virtual viewpoint image based on images of multiple viewpoints, the method comprising:
an image display determination step of determining whether virtual viewpoint information in the virtual viewpoint image satisfies a prohibition condition regarding an object that should not be displayed;
In the image display determination step, if it is determined that the information of the virtual viewpoint satisfies the prohibition condition, control is performed to process the image of the object that should not be displayed in the image corresponding to the virtual viewpoint and output the image. an output control process to
has
The prohibition condition is defined for each divided region in which a target space for generating a virtual viewpoint image is divided into a plurality of regions, and is defined as viewing the object that should not be displayed, the position of the viewpoint, and the direction of the viewpoint. An image processing method characterized by: An image processing method for generating a virtual viewpoint image based on images of multiple viewpoints, the method comprising:
an image display judgment step of determining whether or not a viewpoint specified by virtual viewpoint information in the virtual viewpoint image satisfies a prohibition condition regarding an object that should not be displayed, the prohibition condition forming a virtual viewpoint image; an image display determination step, which is defined for each divided region, in which a target space for displaying the image is divided into a plurality of regions, and is defined as a viewpoint position and a viewpoint direction for viewing the object that should not be displayed;
If the image display determining step determines that the specified viewpoint is within the range of the position and direction of the viewpoint that views the object that should not be displayed, outputting an image according to the viewpoint. an output control process to stop;
An image processing method comprising: A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 6 .

Images