JP2018129787A - Determination device, image processing device, determination method, and determination program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To obtain an image of a background observed when a specific object is temporarily removed in a real space, which is shielded by the specific object and cannot be viewed by a user.SOLUTION: In a determination device, an imaging device whose position and orientation are determined so as to image a specific object placed in a real space is arranged in the real space, the arrangement position of the imaging device that outputs an image including a background image which is a still image or a moving image of a background observed as if a specific object is removed in a space of an image displayed on a screen of a display device and is shielded by the specific object and cannot be viewed from a user, and the determination device includes an arrangement determination unit that determines an arrangement position as a movement destination of the imaging device capable of imaging without losing any background on the basis of position information of the imaging device and information on the background when there is no still image or moving image captured without losing any background in the still image or the moving image captured by the imaging device.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a determination device, an image processing device, a determination method, and a determination program.

  Conventionally, augmented reality (AR) and mixed reality (MR) are known as image processing techniques. Augmented reality is a technology that expresses an expanded reality by giving information to an object that exists in a real space. Mixed reality is a technology in which an image processing device changes an image of an object existing in a real space, and the image processing device combines and expresses the changed object image and the image of an object existing in a virtual space. .

  In the above technical fields, an image without an object in the real space is generated, and the generated image is superimposed on the real space and shown to the user, so that it appears as if the object has been removed from the real space. it can. As a method for generating an image with no object in the real space, it can be roughly classified into two. As a first method, there is a method of restoring from an area around an object using an image in which the object is reflected (for example, see Non-Patent Document 1). As a second method, there is a method of filling an object region using a plurality of camera images capturing the background of the object (see, for example, Non-Patent Document 2).

Norihiko Kawai and two others, “Hidden reality using image restoration”, 17th Image Recognition and Understanding Symposium Hayato Enomoto, 1 other person, "Diminished Reality using multiple handy cameras", Symposium on Image Recognition and Understanding, July 2007

  However, in the method of Non-Patent Document 1, since the background is filled with similar colors or textures around the object, the correct background of the object may not be restored. In the method of Non-Patent Document 2, it is possible to take a plurality of images from different viewpoints in order to acquire the background of the object and make it difficult to see objects other than the background by blending, but the object remains partially. May end up. As described above, in the conventional method, there is a problem when generating an image having no object in the real space.

  On the other hand, a background image that is observed when an object is temporarily removed in real space and that is blocked by a specific object and cannot be seen by the user is captured and presented to the user. However, if the specific object is a non-movable object such as a pillar, a background image cannot be taken. Therefore, there is a technology that can acquire a background image that is observed when a specific object is temporarily removed in real space and that cannot be seen by the user because it is blocked by the specific object. It is requested.

  In view of the above circumstances, the present invention is an image of a background that is observed when a specific object is temporarily removed in real space, and is a background image that is blocked by the specific object and cannot be seen by the user. The purpose is to provide technology that can be acquired.

  According to one aspect of the present invention, an imaging device whose position and orientation are determined so as to capture a specific object placed in a real space is placed in the real space, and the image is displayed in the space of the image displayed on the screen of the display device. A background image that is a still image or a moving image of a background that is observed as if a specific object is removed, and that includes the background image that is blocked by the specific object and cannot be seen by the user. A determination device for determining an arrangement position of the imaging device, wherein the imaging is performed when a still image or a moving image captured by the imaging device does not include a still image or a moving image that is captured without the background missing. An arrangement determination unit including an arrangement determination unit that determines an arrangement position as a movement destination of an imaging apparatus capable of imaging without losing the background based on position information of the apparatus.

  One aspect of the present invention is the above-described determination device, wherein the arrangement determination unit converts a still image or a moving image captured by the imaging device into a still image or a moving image viewed from the direction of the user's line of sight. The arrangement position of the movement destination of the image pickup apparatus is determined so that the influence at the time of doing is reduced.

  One aspect of the present invention is the above-described determination device, in which the arrangement determination unit is a position that can be imaged without losing the background and includes the background within the range of the depth of field. The arrangement position of the movement destination of the imaging device is determined.

  One aspect of the present invention is the above-described determination device, wherein the arrangement determination unit includes a still image or a moving image that is captured without the background being missing, and the position of the imaging device is the background. When the position is not within the depth of field range, the position where the background is included within the depth of field is determined as the arrangement position of the movement destination of the imaging device.

  One embodiment of the present invention is the determination device described above, and when there are a plurality of imaging devices, the imaging device is selected from the plurality of imaging devices based on an angle formed by the user's line-of-sight direction and the optical axis of the imaging device. And a selection unit that determines the arrangement position of the selected imaging device.

  One aspect of the present invention is the above-described determination device, wherein the selection unit includes an angle formed by the user's line-of-sight direction and the optical axis of the imaging device that is equal to or less than a predetermined angle, and the position of the user The imaging device is selected such that the difference between the distance from the specific position of the background to the specific position of the background and the distance from the position of the imaging device to the specific position of the background is equal to or less than a predetermined value.

  One aspect of the present invention is the above-described determination device, and when there are a plurality of imaging devices, evaluation values for a plurality of parameters in an image are calculated for each of the imaging devices, and the evaluation values are set according to priority. The image processing apparatus further includes a selection unit that calculates a total evaluation value for each of the imaging devices by performing weighting, and selects an imaging device having the calculated total evaluation value equal to or greater than a threshold, and the placement determination unit includes the selected imaging device. Determine the placement position.

  One aspect of the present invention includes the determination device, an image acquisition unit that acquires a still image or a moving image captured by the imaging device, and a clip that cuts out the background image from the acquired still image or moving image. An image processing apparatus comprising: an output unit; and an output unit that outputs the cut-out background image to the display device.

  According to one aspect of the present invention, an imaging device whose position and orientation are determined so as to capture a specific object placed in a real space is placed in the real space, and the image is displayed in the space of the image displayed on the screen of the display device. A background image that is a still image or a moving image of a background that is observed as if a specific object is removed, and that includes the background image that is blocked by the specific object and cannot be seen by the user. A determination method performed by a determination device that determines an arrangement position of the image pickup device, wherein a still image or a moving image picked up by the image pickup device does not have a still image or a moving image picked up without missing the background And an arrangement determining step of determining an arrangement position that is a movement destination of the imaging apparatus capable of imaging without losing the background based on position information of the imaging apparatus.

  According to one aspect of the present invention, an imaging device whose position and orientation are determined so as to capture a specific object placed in a real space is placed in the real space, and the image is displayed in the space of the image displayed on the screen of the display device. A background image that is a still image or a moving image of a background that is observed as if a specific object is removed, and that includes the background image that is blocked by the specific object and cannot be seen by the user. As a determining device for determining the arrangement position of the imaging device, the imaging is performed when there is no still image or moving image that is captured without missing the background in the still image or moving image captured by the imaging device. A determination program for executing an arrangement determination step for determining an arrangement position as a movement destination of an imaging apparatus capable of imaging without losing the background based on position information of the apparatus .

  According to the present invention, it is possible to acquire a background image that is observed when a specific object is temporarily removed in real space, and that cannot be seen by the user because it is blocked by the specific object. It becomes.

It is a figure which shows the example of a structure of the image processing system 1a in 1st Embodiment. It is an overhead view which shows the example of arrangement | positioning of the object and camera 2 in 1st Embodiment. It is a figure which shows the example of the variable regarding arrangement | positioning of the camera. It is a figure which shows the example of a display of a display image. It is a figure which shows the specific example of the image imaged with the camera. It is a figure for demonstrating the process of the cutting part. It is a figure which shows the example of a display of the display image after a display. It is a flowchart which shows the flow of a process of the image processing apparatus 3a in 1st Embodiment. It is a figure which shows the example of a structure of the image processing system 1b in 2nd Embodiment. It is an overhead view which shows the example of arrangement | positioning of the object and camera 2-1 to 2-N in 2nd Embodiment. It is a figure which shows the specific example of the image imaged with the cameras 2-1 to 2-3. It is a flowchart which shows the flow of a process of the image processing apparatus 3b in 2nd Embodiment. It is a figure which shows the example of a structure of the image processing system 1c in 3rd Embodiment. It is a figure which shows the specific example of a rotation information table. It is a flowchart which shows the flow of a process of the image processing apparatus 3c in 3rd Embodiment. It is a figure which shows the example of a structure of the image processing system 1d in 4th Embodiment. It is a flowchart which shows the flow of a process of the image processing apparatus 3d in 4th Embodiment. It is a figure which shows the example of a structure of the image processing system 1e in 5th Embodiment. It is an overhead view which shows the example of arrangement | positioning of the object and camera 2-1 to 2-N in 5th Embodiment. It is a flowchart which shows the flow of a process of the image processing apparatus 3e in 5th Embodiment. It is a figure which shows the example of a structure of the image processing system 1f in 6th Embodiment. It is a flowchart which shows the flow of a process of the image processing apparatus 3f in 6th Embodiment. It is a figure which shows the example of a structure of the image processing system 1g in 7th Embodiment. It is a flowchart which shows the flow of a process of the image processing apparatus 3g in 7th Embodiment. It is a figure which shows the example of a structure of the image processing system 1h in 8th Embodiment. It is a flowchart which shows the flow of a process of the image processing apparatus 3h in 8th Embodiment.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings.
(First embodiment)
FIG. 1 is a diagram illustrating an example of a configuration of an image processing system 1a according to the first embodiment. The image processing system 1a is a system that performs image processing on an image. The image may be either a still image or a moving image. The image processing system 1a includes a camera 2, an image processing device 3a, and a display device 4.

  The camera 2 is arranged at a predetermined position in a predetermined direction in the real space where the object is arranged. The camera 2 is moved according to the control of the image processing device 3a. The camera 2 images a real space through an optical system that is a lens. The optical system may be a wide-angle lens, a fish-eye lens, or an omnidirectional lens. The camera 2 outputs an image that is a captured still image or moving image to the image processing device 3a.

  The image processing device 3a is an information processing device such as a personal computer device, a server device, a smartphone terminal, or a tablet terminal. The image processing device 3a and the display device 4 may be separate or integrated. The image processing device 3a moves the camera 2 to a predetermined position. For example, the image processing apparatus 3a moves the camera 2 to a predetermined position while maintaining the optical axis direction of the camera 2 at the position before the movement. The image processing device 3a performs image processing on the image output from the camera 2. For example, the image processing device 3a may execute image processing at an image acquisition period (frame rate) of the camera. The image processing device 3a outputs an image on which image processing has been performed to the display device 4. The image processing device 3a may execute image output processing at, for example, the display cycle (refresh rate) of the display device. That is, the image processing device 3a may perform moving image processing on a moving image including a plurality of frame images.

  The display device 4 is a display device such as a head mounted display. The screen of the display device 4 transmits light that reaches the screen of the display device 4 from the outside of the display device 4. For example, the screen of the display device 4 is a liquid crystal display having light transparency. Hereinafter, an optical image of an object by light transmitted through the screen of the display device 4 is referred to as an “object optical image”. The screen of the display device 4 displays the image output by the image processing device 3a so as to hide the object optical image on the screen. Accordingly, the display device 4 can display the image output from the image processing device 3a. Hereinafter, an image displayed on the screen of the display device 4 is referred to as a “display image”.

  An arrangement example of the object and the camera 2 in the image processing system 1a is shown in FIG. FIG. 2 is an overhead view showing an example of the arrangement of the object and the camera 2 in the first embodiment. As shown in FIG. 2, a camera 2 and an object 6 are arranged in a room 100 having a real space where an observer 5 wearing the display device 4 is located. In FIG. 2, the observer 5 is facing the line of sight 51. That is, the observer 5 faces the direction of the object 6 arranged in the room 100. Thereby, the observer 5 observes the object optical image of the object 6 arranged in the room 100 through the display device 4 with the light transmitted through the screen of the display device 4. It is assumed that the observer 5 is viewing the range from the position a to the position d in the room 100 via the display device 4. That is, a line 52-1 connecting the position a and the position of the observer 5, a line 52-2 connecting the position d and the position of the observer 5, and a line connecting the position a and the position d. Is the range of the visual field of the observer 5.

  The camera 2 is arranged at a predetermined position in the room 100 in a predetermined direction. In the first embodiment, the camera 2 is arranged so that the direction of the line of sight 51 of the observer 5 and the direction of the optical axis 20 of the camera 2 are horizontal. The camera 2 moves in accordance with the control of the image processing device 3a. For example, the camera 2 moves in the direction indicated by the arrow 7 in accordance with the control of the image processing device 3a. The moving direction of the camera 2 may be set in advance or may be instructed by the image processing device 3a.

  The object 6 is, for example, a box or a pillar. In the following description, a case where the object 6 is a cylinder will be described as an example. Since the object 6 is positioned within the field of view of the observer 5, the observer 5 cannot see the wall C from the position b in the room 100 to the position c. That is, the observer 5 cannot see the background behind the object 6 by the object 6. Here, the background represents an object that is shielded by a specific object (for example, the object 6) and cannot be seen by the observer 5. The background may be, for example, an object, a wall in real space, or a virtual wall set at a virtual position in real space. The object 6 is disposed at a predetermined actual object position in the room 100. The actual object position represents the actual position of the object in the real space. The object 6 may be stationary in real space. In the following description, the case where the object 6 is removed and the wall C as the background of the object 6 is displayed on the display device 4 will be described as an example.

  Next, returning to FIG. 1, an example of the configuration of the image processing apparatus 3a will be described. The image processing device 3a includes a CPU (Central Processing Unit), a memory, an auxiliary storage device, and the like connected by a bus, and executes an image processing program. By executing the image processing program, the image processing apparatus 3 a includes an image acquisition unit 30, an operation unit 31, a variable determination unit 32, a storage unit 33, an arrangement determination unit 34, a movement unit 35, a cutout unit 36, and an output unit 37. Functions as a device. Note that all or part of the functions of the image processing apparatus 3a may be realized by using hardware such as an application specific integrated circuit (ASIC), a programmable logic device (PLD), or a field programmable gate array (FPGA). . The image processing program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. Further, the image processing program may be transmitted / received via a telecommunication line.

  The image acquisition unit 30 acquires an image captured by the camera 2. The image acquisition unit 30 records the acquired image in the storage unit 33. The operation unit 31 is an operation device having operation keys and the like. For example, the operation unit 31 is a keyboard, a mouse, or a touch panel. The operation unit 31 accepts an operation on the operation unit 31 to thereby arrange the arrangement information of the camera 2 in the real space, the arrangement information of the object in the real space, the position information of the observer 5, the position information of the background, the pasting position information, and the background. Get information.

  The arrangement information of the camera 2 in the real space is, for example, information such as coordinates and an optical axis direction of the camera 2 in the real space. The optical axis direction is expressed by an angle based on the x-axis of coordinates defined in the real space as shown in FIG. The arrangement information of the object in the real space is, for example, information on the coordinates of the object (for example, the object 6 in FIG. 2) arranged in the real space. The position information of the observer 5 is information on the coordinates of the observer 5. The background position information is information on background coordinates.

The pasting position information may be represented by, for example, information on the upper left coordinates, width, and height in the display image. Further, for example, the pasting position information may be information such as “center”, “right”, and “left”, which are areas defined in advance in association with the coordinates of the display image. The position information of the observer 5 is, for example, information on the coordinates of the observer's standing position in the real space and the line of sight of the observer 5. The background information is information related to the background, for example, a feature amount of the background.
The operation unit 31 outputs the acquired arrangement information of the camera 2, the position information of the observer 5, and the position information of the background to the variable determination unit 32, and the arrangement information of the camera 2, the position information of the observer 5, and The object arrangement information is output to the arrangement determining unit 34. In addition, the operation unit 31 outputs the acquired pasting position information and background information to the cutout unit 36.

  The variable determination unit 32 acquires the arrangement information of the camera 2, the position information of the observer 5, and the background position information from the operation unit 31. The variable determination unit 32 determines a variable related to the arrangement of the camera 2 based on the acquired arrangement information of the camera 2, the position information of the observer 5, and the position information of the background.

  The variables relating to the arrangement of the camera 2 are, for example, the current position of the camera 2, the angle formed by the observer's 5 line-of-sight direction and the optical axis of the camera 2 (hereinafter referred to as “camera angle”), and the position of the observer 5. The difference between the distance to the specific position of the background and the distance from the position of the camera 2 to the specific position of the background (hereinafter referred to as “distance difference”). The specific position of the background may be set in advance. For example, when the background is the wall C in FIG. 2, the midpoint of the wall C may be a specific position of the background. For example, when the background is an object, the position of the object may be set as a specific position of the background. The variable determination unit 32 records variable information related to the arrangement of the camera 2 in the storage unit 33.

  FIG. 3 is a diagram illustrating an example of variables related to the arrangement of the camera 2. The variable determination unit 32 records variable information related to the arrangement of the camera 2 in the storage unit 33. For example, the variable determination unit 32 records the current position of the camera 2 in the storage unit 33. For example, the variable determination unit 32 records the camera angle in the storage unit 33. For example, the variable determination unit 32 records the distance difference in the storage unit 33.

  Returning to FIG. 1, the description of the configuration example of the image processing apparatus 3a will be continued. The storage unit 33 is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The storage unit 33 stores the image acquired by the image acquisition unit 30 and variable information related to the arrangement of the camera 2. The storage unit 33 may store arrangement information (coordinates, optical axis direction) of the camera 2. The storage unit 33 may store the object arrangement information, the position information of the observer 5, and the background information.

  The arrangement determining unit 34 functions as a determining device that determines the arrangement position of the camera 2. The arrangement determination unit 34 stores the image and variable information stored in the storage unit 33, the position information of the observer 5 input via the operation unit 31, the object arrangement information, and the arrangement information of the camera 2. input. The arrangement determining unit 34 determines the arrangement position of the camera 2 based on the input image and variable information, the arrangement information of the camera 2, the position information of the observer 5, and the arrangement information of the object. Specifically, the arrangement determining unit 34 first determines whether or not the input image is captured without any background. Here, the lack of background means that a part or all of the background that is projected when a camera captures a space without an object is shielded by the presence of the object and is not projected, that is, the background in the image. The state in which an object is reflected in part or all of.

  When the input image is captured without any background missing, the arrangement determining unit 34 determines the current position of the camera 2 as the arrangement position of the camera 2. The arrangement determining unit 34 outputs information on the determined arrangement position (hereinafter referred to as “determined arrangement position information”) to the cutting unit 36. The determined arrangement position information represents, for example, the coordinates of the arrangement position of the camera 2 determined by the arrangement determining unit 34.

  On the other hand, when the input image is not captured without a background, the arrangement determining unit 34 determines the position where the camera 2 is moved by a predetermined distance from the current position of the camera 2 as the arrangement position of the camera 2. Here, the position where the camera 2 is moved by a predetermined distance is calculated based on the arrangement information of the camera 2 by calculating an area that is imaged on the same plane as the background and moving the camera 2 in a direction closer to the calculated background area. (For example, about 10% of the distance from the initial position of the camera to the background). Thereafter, the arrangement determining unit 34 outputs the determined arrangement position information to the moving unit 35.

  The moving unit 35 receives the determined arrangement position information output from the arrangement determining unit 34. The moving unit 35 moves the camera 2 to the position indicated by the determined arrangement position information based on the input determined arrangement position information. For example, the moving unit 35 moves the camera 2 to the position indicated by the determined arrangement position information while maintaining the optical axis direction of the camera 2 at the position before the movement. The moving unit 35 moves the camera 2 by driving an actuator such as a motor. In addition, the movement part 35 may detect the positional information on the camera 2 after a movement based on the detection result of a position sensor.

The cutout unit 36 inputs the determined arrangement position information output from the arrangement determination unit 34 and the pasting position information and background information output from the operation unit 31. The cutout unit 36 acquires an image captured by the camera 2 from the storage unit 33. Using the image acquired from the storage unit 33, the cutout unit 36 uses the image of the camera 2 indicated by the determined arrangement position information and the background region image calculated from the background information (hereinafter referred to as “background partial image”). ). The cutout unit 36 determines the display position of the background partial image on the display screen based on the pasting position information output from the operation unit 31. For example, when the pasting position information represents a region “center” defined in advance in association with the coordinates of the display image, the cutout unit 36 displays the display position of the background partial image on the display screen as the display image. It is determined at the center of the screen of the display device 4 defined in advance in association with the coordinates. The cutout unit 36 outputs the background partial image and the display position information that is display position information to the output unit 37.
The output unit 37 inputs the background partial image output from the cutout unit 36 and the display position information. The output unit 37 performs image processing on the background partial image based on the input display position information. The output unit 37 outputs the background partial image subjected to the image processing and the display position information to the display device 4.

  FIG. 4 is a diagram illustrating a display example of a display image. The display device 4 transmits light that reaches the screen of the display device 4 from the object 6. The display device 4 displays a display image 40 including an object optical image of the object 6. As shown in FIG. 4, the observer 5 can see the object optical image of the object 6 at the center of the display image 40. In addition, as shown in FIG. 4, the observer 5 cannot view the object optical image of the background behind the object 6 on the display image 40.

  FIG. 5 is a diagram illustrating a specific example of an image captured by the camera 2. 5A is a camera image 21 output to the image processing apparatus 3a by the camera 2 before movement (in the case of FIG. 2, the solid-line camera 2), and FIG. 5B is the camera after movement. 2 (in the case of FIG. 2, the dotted line camera 2) is the camera image 21 a output to the image processing device 3 a.

  As shown in FIG. 5A, in the camera image 21, a part of the range of the wall C (between the backgrounds bc) is shielded by the object 6, and the entire wall C is not imaged. On the other hand, in the camera image 21a, the wall C is captured without being missing. In this way, the image processing apparatus 3a moves the camera 2 to a position where the background of the specific object 6 viewed from the observer 5 can be captured without being lost.

  FIG. 6 is a diagram for explaining the processing of the cutout unit 36. The cutout unit 36 cuts out the background partial image 50 from the camera image 23 output by the camera 2. Further, the cutout unit 36 determines the display position of the cut out background partial image 50 on the display screen as a position corresponding to the object 6. The output unit 37 outputs the background partial image 50 subjected to the image processing and the display position information to the display device 4. The screen of the display device 4 displays the background partial image 50 subjected to the image processing so as to be superimposed on the object optical image at the position indicated by the display position information so as to hide the object optical image on the screen. A display example of the display image after display is shown in FIG.

  FIG. 7 is a diagram illustrating a display example of a display image after display. As shown in FIG. 7, the display device 4 displays a background partial image 50 on which image processing has been performed on a display image 40 including an object optical image. As a result, an image in which the object 6 displayed on the display image 40 in FIG. 4 is removed can be presented to the observer 5.

FIG. 8 is a flowchart showing the flow of processing of the image processing apparatus 3a in the first embodiment. In the description of FIG. 8, it is assumed that the process by the variable determination unit 32 has already been performed. In addition, FIG. 8 illustrates processing when a still image is acquired from the camera 2.
The image acquisition unit 30 acquires an image from the camera 2 (step S101). The image acquisition unit 30 records the acquired image in the storage unit 33. The arrangement determining unit 34 acquires an image stored in the storage unit 33. Based on the acquired image, the position information of the observer 5 input via the operation unit 31, the object arrangement information, and the arrangement information of the camera 2, the arrangement determining unit 34 captures an image without missing a background. It is determined whether or not there is an image being processed (step S102).

  Specifically, the arrangement determination unit 34 first determines the range of the background invisible to the observer 5 due to the shielding of the object 6 in the visual field range of the observer 5 based on the position information of the observer 5 and the arrangement information of the object. Is calculated. And the arrangement | positioning determination part 34 is based on the arrangement | positioning information of the camera 2, the visual field range of the camera 2, and the arrangement | positioning information of an object, and the image currently imaged without missing the range of the background which the observer 5 cannot see is missing. Judge whether there is.

  When there is no image captured without a background missing (step S102-NO), the arrangement determining unit 34, the input image and variable information, the arrangement information of the camera 2, the position information of the observer 5, and the object Based on the placement information, the placement position of the camera 2 is determined (step S103). The arrangement determining unit 34 outputs the determined arrangement position information to the moving unit 35. The moving unit 35 moves the camera 2 to the position indicated by the determined arrangement position information based on the determined arrangement position information output from the arrangement determining unit 34 (step S104). The image acquisition unit 30 acquires an image captured by the camera 2 after movement (step S105). For example, the image acquisition unit 30 may acquire an image captured by the camera 2 as an image captured by the camera 2 after movement after receiving a notification that the movement of the camera 2 is completed from the movement unit 35. Good. The image acquisition unit 30 records the acquired image in the storage unit 33.

  The arrangement determination unit 34 acquires an image captured by the camera 2 after movement, which is stored in the storage unit 33. Based on the acquired image, the position information of the observer 5 input via the operation unit 31, the object arrangement information, and the arrangement information of the camera 2, the arrangement determining unit 34 captures an image without missing a background. It is determined whether there is an image that has been displayed (step S106). When there is no image captured without a missing background (step S106—NO), the image processing device 3a repeatedly executes the processing after step S103.

On the other hand, when there is an image that has been imaged with no background missing (step S <b> 106 -YES), the arrangement determining unit 34 outputs the determined arrangement position information to the cutting unit 36. The arrangement determining unit 34 outputs the determined arrangement position information of the camera 2 to the cutout unit 36 at the time when it is determined that the image is captured without missing the background. When the determined arrangement position information is acquired from the arrangement determining unit 34, the cutting unit 36 acquires the image captured by the camera 2 from the storage unit 33. The cutout unit 36 cuts out the background partial image from the acquired image. Further, the cutout unit 36 determines the display position of the background partial image on the display screen based on the pasting position information. For example, the cutout unit 36 determines the area specified by the pasting position information as the display position of the background partial image on the display screen.
The cutout unit 36 outputs the background partial image and the display position information to the output unit 37.

The output unit 37 inputs the background partial image and the display position information output from the cutout unit 36. The output unit 37 performs image processing on the input background partial image. Specifically, the output unit 37 converts the background partial image into an image viewed from the direction of the line of sight 51 of the observer 5. For example, the output unit 37 performs affine transformation processing using an affine transformation matrix on the background partial image. The affine transformation matrix may include information indicating a region to be subjected to processing such as rotation, enlargement, or reduction. Thereafter, the output unit 37 outputs the converted background partial image and display position information to the display device 4 (step S107).
Further, in the process of step S102, when the image is captured with no background (step S102-YES), the arrangement determining unit 34 determines the current position of the camera 2 as the arrangement position of the camera 2. Thereafter, the arrangement determining unit 34 outputs the determined arrangement position information to the cutout unit 36. Thereafter, the image processing device 3a executes the process of step S107.

  Differences from the processing shown in FIG. 8 will be described in the case where the image acquisition unit 30 acquires a moving image. In step S101, the image acquisition unit 30 acquires a moving image from the camera 2 and acquires each frame image from the acquired moving image. In step S <b> 102, the arrangement determining unit 34 determines whether there is an image that has been captured without missing a background, based on the first frame image of each frame image of the acquired moving image. If there is an image that is captured without any background missing, in step S107, the cutout unit 36 cuts out a background partial image for each frame image of the moving image acquired from the camera 2. Next, the cutout unit 36 determines the display position of the background partial image so as to be a common display position in each frame image. Thereafter, the output unit 37 outputs the background partial images corresponding to the number of frame images and the display position information common to the frame images to the display device 4.

On the other hand, in step S102, when there is no image captured without a missing background, in step S105, the image acquisition unit 30 acquires a moving image from the camera 2 after movement, and each frame image from the acquired moving image. To get. In step S <b> 106, the arrangement determining unit 34 determines whether there is an image that has been captured without missing the background, based on the first frame image of each frame image of the acquired moving image. If there is an image that is captured without any background missing, in step S107, the cutout unit 36 cuts out a background partial image for each frame image of the moving image acquired from the camera 2 after movement. Next, the cutout unit 36 determines the display position of the background partial image so as to be a common display position in each frame image. Thereafter, the output unit 37 outputs the background partial images corresponding to the number of frame images and the display position information common to the frame images to the display device 4.
On the other hand, in step S106, when there is no image captured without a missing background, the image processing device 3a executes the processing from step S103 onward.

  As described above, in the image processing device 3a of the first embodiment, the camera 2 whose position and orientation are determined so as to image the object 6 placed in the real space is placed in the real space, and the display device 4 Arrangement of the camera 2 that outputs a background image that is observed as if the object 6 has been removed in the space of the image displayed on the screen and that is hidden by the object 6 and cannot be seen by the user A determination device for determining the position is provided. The determining apparatus according to the first embodiment includes an arrangement determining unit 34. The arrangement determination unit 34 is a camera 2 that is a movement destination of the camera 2 that can capture an image without missing a background that is not visible to the observer 5 with a specific object in an area that the observer 5 is looking through the display device 4. Determine the placement position. As a result, by acquiring an image captured from the determined arrangement position, the image processing device 3a can acquire a background image that is blocked by a specific object and cannot be seen by the user.

<Modification>
The image processing device 3a may execute the process of FIG. 8 when the specific object moves or the specific object is changed, or may execute the process of FIG. 8 when the observer 5 moves. Good.
The image acquisition unit 30 may output an image acquired from the moved camera 2 to the cutout unit 36. When configured in this way, the moving unit 35 outputs a notification indicating that the movement of the camera 2 is completed to the image acquisition unit 30. When the image acquisition unit 30 acquires the notification from the moving unit 35 and then acquires the image from the camera 2, the image acquisition unit 30 outputs the acquired image to the cutout unit 36. The cutout unit 36 inputs the image output from the image acquisition unit 30 and the background information and pasting position information input via the operation unit 31. The cutout unit 36 cuts out the background partial image from the input image. The cutout unit 36 determines the display position of the background partial image on the display screen based on the pasting position information. For example, the area specified by the pasting position information specified by the operation unit 31 is determined as the display position of the background partial image on the display screen.

(Second Embodiment)
In the first embodiment, the configuration in which one camera is provided in the image processing system and the camera is moved to a position where the entire background can be imaged is shown. On the other hand, in the second embodiment, the image processing system includes a plurality of cameras, and the image processing apparatus selects one camera from the plurality of cameras, and the selected camera is located at a position where all the backgrounds can be captured. It is different from the first embodiment in that it is moved. In the second embodiment, only differences from the first embodiment will be described.

  FIG. 9 is a diagram illustrating an example of the configuration of the image processing system 1b according to the second embodiment. The image processing system 1b includes cameras 2-1 to 2-N (N is an integer equal to or greater than 2), an image processing device 3b, and a display device 4. The image processing device 3b is an information processing device such as a personal computer device, a server device, a smartphone terminal, or a tablet terminal. The image processing device 3b and the display device 4 may be separate or integrated. In the following, N is 3 as an example.

  The image processing device 3b performs image processing on an image output from the camera 2-n (1 ≦ n ≦ N) selected from the cameras 2-1 to 2-N. The image processing device 3b may execute image processing at, for example, a camera image acquisition period (frame rate). The image processing device 3b outputs the image subjected to the image processing to the display device 4. For example, the image processing device 3b may execute an image output process at a display cycle (refresh rate) of the display device. That is, the image processing device 3b may perform moving image processing on a moving image including a plurality of frame images.

  The image processing device 3b includes a CPU, a memory, an auxiliary storage device, and the like connected by a bus, and executes an image processing program. By executing the image processing program, the image processing apparatus 3b has an image acquisition unit 30, an operation unit 31, a variable determination unit 32, a storage unit 33, an arrangement determination unit 34b, a movement unit 35, a cutout unit 36, an output unit 37, and a selection unit. It functions as a device including the unit 38. Note that all or part of the functions of the image processing apparatus 3b may be realized using hardware such as an ASIC, PLD, or FPGA. The image processing program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. Further, the image processing program may be transmitted / received via a telecommunication line.

  The image processing device 3b is different in configuration from the image processing device 3a in that it includes a placement determination unit 34b instead of the placement determination unit 34 and a selection unit 38. The image processing device 3b is the same as the image processing device 3a in other configurations. Therefore, the description of the entire image processing apparatus 3b is omitted, and the arrangement determination unit 34b and the selection unit 38 will be described.

  The arrangement determining unit 34b functions as a determining device that determines the arrangement position of the camera 2-n. The arrangement determination unit 34b includes the identification information of the camera 2-n selected by the selection unit 38, images and variable information captured by the cameras 2-1 to 2-N stored in the storage unit 33, and operations. The position information of the observer 5 input via the unit 31, the arrangement information of the object in the real space, and the arrangement information of the cameras 2-1 to 2-N in the real space are input. Based on the input plurality of images and variable information, the placement information of each of the cameras 2-1 to 2 -N, the position information of the observer 5, and the placement information of the object, the placement determination unit 34 Determine the placement position of -n.

  Specifically, the arrangement determining unit 34b first determines whether or not there is an image that is captured without missing the background among the plurality of input images. When there is an image that is captured without a background missing, the arrangement determining unit 34b determines the current positions of the cameras 2-1 to 2-N as the arrangement positions of the cameras 2-1 to 2-N. In this case, the arrangement determining unit 34b outputs a notification (hereinafter referred to as “camera presence notification”) indicating that there is a camera 2-n that captures an image without missing a background to the selection unit 38. The camera presence notification includes the identification information of the camera 2-n that captures an image without missing the background and the determined arrangement position information of each camera 2-n.

  On the other hand, when there is no image captured without missing the background, the arrangement determining unit 34b notifies that there is no camera 2-n that is imaged without missing the background (hereinafter referred to as “no camera notification”). ) Is output to the selection unit 38. Further, the arrangement determining unit 34b determines a position moved by a predetermined distance from the current position of the camera 2-n selected by the selection unit 38 as the arrangement position of the selected camera 2-n. When there is no image captured without a background missing, the arrangement determining unit 34b outputs the identification information of the selected camera 2-n and the determined arrangement position information to the moving unit 35.

  The selection unit 38 functions as a determination device that determines the camera 2-n to be selected. The selection unit 38 receives the camera presence notification or the camera non-notification output from the arrangement determination unit 34 b and the variable information stored in the storage unit 33. When the camera non-notification is acquired from the arrangement determination unit 34b, the selection unit 38 selects a camera 2-n having a camera angle equal to or smaller than a predetermined angle from the cameras 2-1 to 2-N based on the input variable information. select. The predetermined angle is, for example, 30 degrees. Note that when there are a plurality of cameras 2-n having a camera angle equal to or smaller than a predetermined angle, the selection unit 38 selects the camera 2-n based on the distance difference. Specifically, the selection unit 38 selects a camera 2-n whose distance difference is a predetermined value or less. The predetermined value of the distance difference is, for example, about 10% of the distance from the initial position of the camera to the background. Thereafter, the selection unit 38 outputs the identification information of the selected camera 2-n to the arrangement determination unit 34b.

  In addition, when the camera presence notification is acquired from the arrangement determination unit 34b, the selection unit 38, based on the input variable information, the camera 2 having a camera angle equal to or smaller than a predetermined angle among the cameras 2-1 to 2-N. Select n. The predetermined angle is, for example, 30 degrees. Note that when there are a plurality of cameras 2-n having a camera angle equal to or smaller than a predetermined angle, the selection unit 38 selects the camera 2-n based on the distance difference. Specifically, the selection unit 38 selects a camera 2-n whose distance difference is a predetermined value or less. The predetermined value of the distance difference is, for example, about 10% of the distance from the initial position of the camera to the background. The selection unit 38 selects the camera 2-n having the smallest distance difference when there are a plurality of cameras 2-n having a camera angle smaller than the predetermined angle or when there are a plurality of cameras 2-n having a distance difference equal to or smaller than a predetermined value. You may choose. Thereafter, the selection unit 38 outputs the identification information of the selected camera 2-n and the determined arrangement position information of the selected camera 2-n to the cutting unit 36.

  An arrangement example of objects and cameras 2-1 to 2-N in the image processing system 1b is shown in FIG. FIG. 10 is an overhead view showing an example of the arrangement of objects and cameras 2-1 to 2-N in the second embodiment. As shown in FIG. 10, cameras 2-1 to 2-3 and an object 6 are arranged in a room 100 having a real space where an observer 5 wearing the display device 4 is located. In FIG. 10, the observer 5 is facing the direction of the line of sight 51. That is, the observer 5 faces the direction of the object 6 arranged in the room 100. Thereby, the observer 5 observes the object optical image of the object 6 arranged in the room 100 through the display device 4 with the light transmitted through the screen of the display device 4. It is assumed that the observer 5 is viewing the range from the position a to the position d in the room 100 via the display device 4. That is, a line 52-1 connecting the position a and the position of the observer 5, a line 52-2 connecting the position d and the position of the observer 5, and a line connecting the position a and the position d. Is the range of the visual field of the observer 5.

  The cameras 2-1 to 2-3 are arranged at predetermined positions in the room 100 in a predetermined direction. In the second embodiment, the camera 2-1 is arranged so that the direction of the line of sight 51 of the observer 5 and the direction of the optical axis 20-1 of the camera 2-1 are horizontal, and the cameras 2-2 and 2. -3 is arranged so that the direction of the line of sight 51 of the observer 5 is different from the direction of the optical axis 20-2 of the camera 2-2 and the direction of the optical axis 20-3 of the camera 2-3.

  FIG. 11 is a diagram illustrating a specific example of images captured by the cameras 2-1 to 2-3. 11A shows a camera image 22 output from the camera 2-1 to the image processing apparatus 3b, and FIG. 11B shows a camera image 23 output from the camera 2-2 to the image processing apparatus 3b. FIG. 11C shows a camera image 24 output from the camera 2-3 to the image processing apparatus 3b. As shown in FIGS. 11A to 11C, in the camera images 22 to 24, a part of the range of the wall C (between the backgrounds b and c) is shielded by the object 6, and the entire wall C is imaged. Not. In such a case, the image processing device 3b according to the second embodiment moves the camera 2-n selected from the plurality of cameras 2-1 to 2-N to a position where the image can be captured without missing the background.

FIG. 12 is a flowchart illustrating a processing flow of the image processing apparatus 3b according to the second embodiment. In the description of FIG. 12, it is assumed that the processing by the variable determination unit 32 has already been performed. In addition, FIG. 12 illustrates processing when still images are acquired from a plurality of cameras 2-1 to 2-N.
The image acquisition unit 30 acquires images from each of the plurality of cameras 2-1 to 2-N (step S201). The image acquisition unit 30 records the acquired images in the storage unit 33 in association with the identification information of each of the cameras 2-1 to 2 -N. The arrangement determination unit 34 b acquires images captured by the cameras 2-1 to 2 -N and stored in the storage unit 33. The arrangement determining unit 34b is based on the acquired images, the position information of the observer 5 input via the operation unit 31, the object arrangement information, and the arrangement information of the cameras 2-1 to 2-N. Then, it is determined whether or not there is an image that has been imaged with no background missing (step S202).

  For example, the arrangement determination unit 34b captures all the images of the acquired cameras 2-1 to 2-N or a part thereof without any background, or the entire images of the cameras 2-1 to 2-N. It is determined whether or not the background is missing. When all or some of the acquired images of the cameras 2-1 to 2-N are captured without any background, the arrangement determining unit 34b determines that there is an image that is captured without any background missing. . On the other hand, when all of the acquired images of the cameras 2-1 to 2-N are captured with the background missing, the arrangement determining unit 34b determines that there is no image captured without the background missing.

  When there is no image captured without a background missing (step S202—NO), the arrangement determining unit 34b outputs a camera non-notification to the selecting unit 38. When the camera non-notification is acquired from the arrangement determination unit 34b, the selection unit 38 selects a camera 2-n whose camera angle is equal to or smaller than a predetermined angle based on the variable information stored in the storage unit 33 (step S40). S203). Thereafter, the selection unit 38 outputs the identification information of the selected camera 2-n to the arrangement determination unit 34b. The arrangement determining unit 34b moves the position moved by a predetermined distance from the current position of the camera 2-n identified by the identification information of the camera 2-n output from the selection unit 38 to the arrangement position of the selected camera 2-n. (Step S204). Thereafter, the arrangement determining unit 34b outputs the identification information of the selected camera 2-n and the determined arrangement position information to the moving unit 35.

  The moving unit 35 indicates the camera 2-n identified by the identification information based on the identification information of the camera 2-n output from the arrangement determining unit 34 and the determined arrangement position information, by the determined arrangement position information. The position is moved (step S205). The image acquisition unit 30 acquires an image captured by the camera 2-n after movement (step S206). The image acquisition unit 30 records the acquired image in the storage unit 33.

  The arrangement determining unit 34b acquires an image captured by the moved camera 2-n stored in the storage unit 33. Based on the acquired image, the positional information of the observer 5 input via the operation unit 31, the positional information of the object, and the positional information of the camera 2-n, the layout determining unit 34b has no background. It is determined whether there is a captured image (step S207). If there is no image captured without a missing background (step S207—NO), the image processing device 3b repeatedly executes the processing after step S204.

  On the other hand, when there is an image that has been captured without any background missing (step S207—YES), the arrangement determination unit 34b outputs a camera presence notification to the selection unit 38. When the camera presence notification is acquired from the arrangement determination unit 34b, the selection unit 38 selects the camera 2-n from the cameras 2-n that capture images without missing the background. Thereafter, the selection unit 38 outputs the identification information of the selected camera 2-n and the determined arrangement position information of the selected camera 2-n to the cutting unit 36. The cutout unit 36 acquires an image captured by the camera 2-n identified by the identification information output from the selection unit 38 from the storage unit 33. The cutout unit 36 cuts out the background partial image from the acquired image. Further, the cutout unit 36 determines the display position of the background partial image on the display screen based on the pasting position information. For example, the cutout unit 36 determines the area specified by the pasting position information as the display position of the background partial image on the display screen. The cutout unit 36 outputs the background partial image and the display position information to the output unit 37.

  The output unit 37 inputs the background partial image and the display position information output from the cutout unit 36. The output unit 37 performs image processing on the input background partial image. Specifically, the output unit 37 converts the background partial image into an image viewed from the direction of the line of sight 51 of the observer 5. For example, the output unit 37 performs affine transformation processing using an affine transformation matrix on the background partial image. The affine transformation matrix may include information indicating a region to be subjected to processing such as rotation, enlargement, or reduction. Thereafter, the output unit 37 outputs the converted background partial image and display position information to the display device 4 (step S208).

  Further, in the process of step S202, when there is an image that is captured with no background missing (step S202—YES), the image processing device 3b executes the process of step S208.

  When the image acquisition unit 30 acquires a moving image, differences from the processing illustrated in FIG. 12 will be described. In step S201, the image acquisition unit 30 acquires a moving image from each of the plurality of cameras 2-1 to 2-N, and acquires each frame image from each of the acquired moving images. In step S202, the arrangement determining unit 34b determines whether there is an image that is captured without any background missing, based on the first frame image of each frame image of each acquired moving image. If there is an image that is captured without any background missing, in step S208, the cutout unit 36 cuts out a background partial image for each frame image of the moving image acquired from the selected camera 2-n. Next, the cutout unit 36 determines the display position of the background partial image so as to be a common display position in each frame image. Thereafter, the output unit 37 outputs the background partial images corresponding to the number of frame images and the display position information common to the frame images to the display device 4.

On the other hand, in step S202, when there is no image captured without a missing background, in step S206, the image acquisition unit 30 acquires a moving image from the plurality of cameras 2-n after movement, and acquires the acquired moving image. Each frame image is acquired from. In step S207, the arrangement determining unit 34b determines whether or not there is an image that is captured without any background missing based on the first frame image of each frame image of the acquired moving image. When there is an image that is captured without any background missing, in step S208, the cutout unit 36 cuts out a background partial image for each frame image of the moving image acquired from the camera 2-n after movement. Next, the cutout unit 36 determines the display position of the background partial image so as to be a common display position in each frame image. Thereafter, the output unit 37 outputs the background partial images corresponding to the number of frame images and the display position information common to the frame images to the display device 4.
On the other hand, in step S207, when there is no image captured without a missing background, the image processing device 3b executes the processing after step S204.

  As described above, in the image processing device 3b of the second embodiment, the camera 2 whose position and orientation are determined so as to image the object 6 placed in the real space is placed in the real space, and the display device 4 Arrangement of the camera 2 that outputs a background image that is observed as if the object 6 has been removed in the space of the image displayed on the screen and that is hidden by the object 6 and cannot be seen by the user A determination device for determining the position is provided. The determination apparatus according to the second embodiment includes an arrangement determination unit 34b and a selection unit 38. The selection unit 38 selects the camera 2-n based on the camera angle among the plurality of cameras 2-1 to 2-N. The arrangement determination unit 34b can capture an image of the selected camera 2-n without missing a background that is not visible to the observer 5 with a specific object in an area viewed by the observer 5 via the display device 4. The arrangement position of the camera 2-n which is the movement destination of the camera 2-n is determined. Thus, by acquiring an image captured from the determined arrangement position, the image processing device 3b can acquire a background image that is blocked by a specific object and cannot be seen by the user.

<Modification>
The image processing apparatus 3b may execute the process of FIG. 12 when the specific object moves or the specific object is changed, or may execute the process of FIG. 12 when the observer 5 moves. Good.
The image acquisition unit 30 may output an image acquired from the moved camera 2-n to the cutout unit 36. When configured in this way, the moving unit 35 outputs a notification indicating that the movement of the camera 2-n is completed to the image acquiring unit 30. The notification output by the moving unit 35 includes, for example, identification information of the camera 2-n. When the image acquisition unit 30 acquires the notification from the moving unit 35 and then acquires an image from the camera 2-n identified by the identification information, the image acquisition unit 30 outputs the acquired image to the cutout unit 36. The cutout unit 36 inputs the image output from the image acquisition unit 30 and the background information and pasting position information input via the operation unit 31. The cutout unit 36 cuts out the background partial image from the input image. The cutout unit 36 determines the display position of the background partial image on the display screen based on the pasting position information. For example, the area specified by the pasting position information specified by the operation unit 31 is determined as the display position of the background partial image on the display screen. In this case, the arrangement determining unit 34b may not output the identification information of the camera 2-n that has determined the arrangement position to the cutting unit 36.

(Third embodiment)
In 1st Embodiment and 2nd Embodiment, the structure which moves a camera to the position which can image all the backgrounds was shown. On the other hand, in the third embodiment, the position and orientation of the destination of the camera so that the entire background can be imaged and the influence upon conversion to an image viewed from the direction of the observer's line of sight is reduced. This is different from the first and second embodiments in that it is controlled. In the third embodiment, only differences from the first embodiment and the second embodiment will be described.

FIG. 13 is a diagram illustrating an example of a configuration of an image processing system 1c according to the third embodiment. The image processing system 1c includes a camera 2c, an image processing device 3c, and a display device 4.
The camera 2c is arranged at a predetermined position in a predetermined direction in the real space where the object is arranged. The movement and orientation of the camera 2c are changed according to the control of the image processing device 3c. The camera 2c images real space through an optical system that is a lens. The optical system may be a wide-angle lens, a fish-eye lens, or an omnidirectional lens. The camera 2c outputs an image that is a captured still image or moving image to the image processing device 3c.

  The image processing device 3c is an information processing device such as a personal computer device, a server device, a smartphone terminal, or a tablet terminal. The image processing device 3c and the display device 4 may be separate or integrated. The image processing device 3c performs image processing on the image output from the camera 2c. For example, the image processing device 3c may execute image processing at an image acquisition period (frame rate) of the camera. The image processing device 3c outputs the image subjected to the image processing to the display device 4. For example, the image processing device 3c may execute an image output process at a display cycle (refresh rate) of the display device. That is, the image processing device 3c may perform moving image processing on a moving image including a plurality of frame images.

  The image processing device 3c includes a CPU, a memory, an auxiliary storage device, and the like connected by a bus, and executes an image processing program. By executing the image processing program, the image processing apparatus 3c includes an image acquisition unit 30, an operation unit 31, a variable determination unit 32, a storage unit 33c, an arrangement determination unit 34c, a cutout unit 36, an output unit 37, and an arrangement unit 39. Functions as a device. Note that all or part of the functions of the image processing apparatus 3c may be realized using hardware such as an ASIC, PLD, or FPGA. The image processing program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. Further, the image processing program may be transmitted / received via a telecommunication line.

  The image processing device 3c is different from the image processing device 3a in that the storage unit 33c and the arrangement determination unit 34c are provided in place of the storage unit 33 and the arrangement determination unit 34, the arrangement unit 39 is newly provided, and the movement unit 35 is not provided. And the configuration is different. The image processing device 3c is the same as the image processing device 3a in other configurations. Therefore, description of the entire image processing apparatus 3c is omitted, and the storage unit 33c, the arrangement determination unit 34c, and the arrangement unit 39 will be described.

  The storage unit 33c is configured using a storage device such as a magnetic hard disk device or a semiconductor storage device. The memory | storage part 33c memorize | stores the image acquired by the image acquisition part 30, and the variable information regarding arrangement | positioning of the camera 2c. The storage unit 33c may store arrangement information (coordinates, optical axis direction) of the camera 2c. Further, the storage unit 33c stores a table (hereinafter referred to as “rotation information table”) representing information related to the rotation of the camera 2c.

  FIG. 14 is a diagram illustrating a specific example of the rotation information table. The rotation information table is a table in which the rotation angle of the camera 2c is set in advance for each position and camera angle of the camera 2c. In the rotation information table, the position represents the position where the camera 2c is currently arranged when the camera 2c is not moving, and the moved camera 2c is arranged when the camera 2c is moving. Represents the position. The camera angle represents an angle formed between the viewing direction of the observer 5 and the optical axis of the camera 2c. The rotation angle represents the rotation angle of the camera 2c. For example, the rotation angle represents how much the camera 2c is rotated from the current angle.

  The arrangement determining unit 34c functions as a determining device that determines the arrangement position and orientation of the camera 2c. The arrangement determining unit 34c includes an image, variable information and rotation information table stored in the storage unit 33c, position information of the observer 5 input via the operation unit 31, and object arrangement information in real space. The arrangement information of the camera 2c is input. The arrangement determination unit 34c determines the arrangement position and orientation of the camera 2c based on the input image, variable information and rotation information table, the arrangement information of the camera 2c, the position information of the observer 5, and the arrangement information of the object. decide. Specifically, the arrangement determining unit 34c first determines whether or not the input image is captured without a background. When the input image is captured without any background missing, the arrangement determining unit 34c determines the current position of the camera 2c as the arrangement position of the camera 2c. The arrangement determining unit 34 c outputs the determined arrangement position information to the cutting unit 36.

  On the other hand, if the input image is not captured without a background, the arrangement determining unit 34c determines the position where the camera 2c is moved by a predetermined distance from the current position of the camera 2c as the arrangement position of the camera 2c. Here, the position where the camera 2c is moved by a predetermined distance refers to the area where the camera 2c is imaged on the same plane as the background from the arrangement information of the camera 2c, and the camera 2c is determined in a direction closer to the calculated background area. (For example, set to about 10% of the distance from the initial position of the camera to the background).

  Further, the arrangement determining unit 34c refers to the rotation information table, and acquires the rotation angle based on the position of the camera 2c and the camera angle. The arrangement determining unit 34c determines the acquired rotation angle as the orientation of the camera 2c. The arrangement determination unit 34 c outputs camera movement destination arrangement information to the arrangement unit 39. The camera movement destination arrangement information is information representing the coordinates and rotation angle of the arrangement position determined by the arrangement determination unit 34c.

  The placement unit 39 receives the camera movement destination placement information output from the placement determination unit 34c. First, the arrangement unit 39 moves the camera 2c to a position indicated by the camera movement destination arrangement information based on the input camera movement destination arrangement information. For example, the placement unit 39 moves the camera 2c to a predetermined position while maintaining the optical axis direction of the camera 2c at the position before the movement. Thereafter, the arrangement unit 39 arranges the camera 2c in the determined direction by rotating the camera 2c by the rotation angle included in the camera movement destination arrangement information. For example, the placement unit 39 moves and places the camera 2c by driving an actuator such as a motor. In addition, the arrangement | positioning part 39 may detect the positional information on the camera 2c after a movement based on the detection result of a position sensor. The placement unit 39 may detect the orientation (posture information) of the camera 2c based on the detection results of the acceleration sensor and the orientation sensor.

FIG. 15 is a flowchart illustrating a processing flow of the image processing apparatus 3c according to the third embodiment. In the description of FIG. 15, it is assumed that the processing by the variable determination unit 32 has already been performed. 15, processes similar to those in FIG. 8 are denoted by the same reference numerals as those in FIG. 8, and description thereof is omitted. In addition, FIG. 15 illustrates processing when a still image is acquired from the camera 2c.
In the process of step S102, when there is no image captured without a missing background (step S102-NO), the arrangement determining unit 34c inputs the input image, variable information and rotation information table, arrangement information of the camera 2c, Based on the position information of the observer 5 and the arrangement information of the object, the arrangement position and orientation of the camera 2c are determined (step S301). The arrangement determination unit 34 c outputs camera movement destination arrangement information to the arrangement unit 39. The arrangement unit 39 moves the camera 2c to the camera movement destination position based on the coordinates of the arrangement position included in the camera movement destination arrangement information output from the arrangement determination unit 34c (step S302).

  Thereafter, the arrangement unit 39 rotates the camera 2c by an angle indicated by the rotation angle based on the rotation angle included in the camera movement destination arrangement information (step S303). The image acquisition unit 30 acquires an image captured by the camera 2c after movement and rotation (step S304). For example, the image acquisition unit 30 receives the notification that the movement and rotation of the camera 2c has been completed from the placement unit 39, and then converts the image captured by the camera 2c as an image captured by the camera 2c after movement and rotation. You may get it. The image acquisition unit 30 records the acquired image in the storage unit 33c.

The arrangement determination unit 34c acquires an image captured by the camera 2c after movement and rotation, which is stored in the storage unit 33c. Based on the acquired image, variable information and rotation information table, the placement information of the camera 2c, the position information of the observer 5, and the placement information of the object, the placement determination unit 34c is imaged without missing a background. It is determined whether there is an image (step S305). When there is no image captured without a missing background (step S305—NO), the image processing device 3c repeatedly executes the processing after step S301.
On the other hand, when there is an image that has been imaged with no background missing (step S305—YES), the image processing device 3c executes the process of step S107.

  Differences from the process shown in FIG. 15 will be described in the case where the image acquisition unit 30 acquires a moving image. In step S101, the image acquisition unit 30 acquires a moving image from the camera 2c, and acquires each frame image from the acquired moving image. In step S102, the arrangement determining unit 34c determines whether or not there is an image that is captured without any background missing, based on the first frame image of each frame image of the acquired moving image. If there is an image that is captured with no background missing, in step S107, the cutout unit 36 cuts out a background partial image for each frame image of the moving image acquired from the camera 2c. Next, the cutout unit 36 determines the display position of the background partial image so as to be a common display position in each frame image. Thereafter, the output unit 37 outputs the background partial images corresponding to the number of frame images and the display position information common to the frame images to the display device 4.

On the other hand, in step S102, when there is no image captured without a missing background, in step S304, the image acquisition unit 30 acquires a moving image from the camera 2c after movement and rotation, and each image is acquired from the acquired moving image. Get a frame image. In step S <b> 305, the arrangement determining unit 34 c determines whether there is an image that has been captured without any background missing, based on the first frame image of each frame image of the acquired moving image. When there is an image that is captured without any background missing, in step S107, the cutout unit 36 cuts out a background partial image for each frame image of the moving image acquired from the moved and rotated camera 2c. Next, the cutout unit 36 determines the display position of the background partial image so as to be a common display position in each frame image. Thereafter, the output unit 37 outputs the background partial images corresponding to the number of frame images and the display position information common to the frame images to the display device 4.
On the other hand, in step S305, when there is no image captured without a missing background, the image processing device 3c executes the processing after step S301.

  As described above, in the image processing device 3c according to the third embodiment, the camera 2c whose position and orientation are determined so as to image the object 6 placed in the real space is placed in the real space, and the display device 4 Arrangement of a camera 2c that outputs a background image that is observed as if the object 6 has been removed in the space of the image displayed on the screen and that is hidden by the object 6 and cannot be seen by the user A determination device for determining the position is provided. The determining apparatus according to the third embodiment includes an arrangement determining unit 34c. The arrangement determination unit 34c determines the position and orientation of the movement destination of the camera 2c so that the influence when converting the image captured by the camera 2c into an image viewed from the direction of the line of sight of the observer 5 is reduced. As a result, the image processing device 3c is shielded by a specific object and can be viewed from the user by acquiring an image captured by the imaging device arranged at the determined arrangement position and arranged in the determined orientation. It becomes possible to obtain a background image that cannot be obtained.

<Modification>
The image processing apparatus 3c may execute the process of FIG. 15 when the specific object moves or the specific object is changed, or may execute the process of FIG. 15 when the observer 5 moves. Good.
The image acquisition unit 30 may output an image acquired from the camera 2c after the movement and orientation are adjusted to the cutout unit 36. When configured in this way, the arrangement unit 39 outputs a notification indicating that the movement and orientation adjustment of the camera 2c has been completed to the image acquisition unit 30. When the image acquisition unit 30 acquires an image from the camera 2 c after acquiring the notification from the arrangement unit 39, the image acquisition unit 30 outputs the acquired image to the cutout unit 36. The cutout unit 36 inputs the image output from the image acquisition unit 30 and the background information and pasting position information input via the operation unit 31. The cutout unit 36 cuts out the background partial image from the input image. The cutout unit 36 determines the display position of the background partial image on the display screen based on the pasting position information. For example, the area specified by the pasting position information specified by the operation unit 31 is determined as the display position of the background partial image on the display screen.

(Fourth embodiment)
In the third embodiment, a single camera is provided in the image processing system so that the entire background can be imaged, and the influence upon conversion to an image viewed from the direction of the observer's line of sight is reduced. A configuration for controlling the position and orientation of the camera destination is shown. On the other hand, in the fourth embodiment, the image processing system is provided with a plurality of cameras, and the image processing apparatus selects one camera from the plurality of cameras so that the background of the selected camera can be captured. In addition, the third embodiment is different from the third embodiment in that the position and orientation of the movement destination of the camera are controlled so as to reduce the influence upon conversion to an image viewed from the direction of the line of sight of the observer. In the fourth embodiment, only differences from the third embodiment will be described.

  FIG. 16 is a diagram illustrating an example of a configuration of an image processing system 1d according to the fourth embodiment. The image processing system 1d includes cameras 2c-1 to 2c-N, an image processing device 3d, and a display device 4. The image processing device 3d is an information processing device such as a personal computer device, a server device, a smartphone terminal, or a tablet terminal. The image processing device 3d and the display device 4 may be separate or integrated. In the following, N is 3 as an example.

  The image processing device 3d performs image processing on an image output from the camera 2c-n selected from the cameras 2c-1 to 2c-N. The image processing device 3d may execute image processing at, for example, a camera image acquisition period (frame rate). The image processing device 3d outputs the image subjected to the image processing to the display device 4. The image processing device 3d may execute image output processing at, for example, the display cycle (refresh rate) of the display device. That is, the image processing device 3d may perform moving image processing on a moving image including a plurality of frame images.

  The image processing device 3d includes a CPU, a memory, an auxiliary storage device, and the like connected by a bus, and executes an image processing program. By executing the image processing program, the image processing apparatus 3d causes the image acquisition unit 30, the operation unit 31, the variable determination unit 32, the storage unit 33c, the arrangement determination unit 34d, the arrangement unit 39, the cutout unit 36, the output unit 37, and the selection. It functions as a device including the unit 38d. Note that all or part of the functions of the image processing device 3d may be realized using hardware such as an ASIC, PLD, or FPGA. The image processing program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. Further, the image processing program may be transmitted / received via a telecommunication line.

  The image processing apparatus 3d is different in configuration from the image processing apparatus 3c in that it includes an arrangement determination unit 34d instead of the arrangement determination unit 34c and a selection unit 38d. The other configuration of the image processing device 3d is the same as that of the image processing device 3c. Therefore, the description of the entire image processing apparatus 3d is omitted, and the arrangement determination unit 34d and the selection unit 38d will be described.

  The arrangement determining unit 34d functions as a determining device that determines the arrangement position and orientation of the cameras 2c-n. The arrangement determination unit 34d includes the identification information of the camera 2c-n selected by the selection unit 38d, and the images, variable information, and rotation information captured by the cameras 2c-1 to 2c-N stored in the storage unit 33c. The table, the position information of the observer 5 input via the operation unit 31, the object arrangement information in the real space, and the arrangement information of the cameras 2c-1 to 2c-N in the real space are input. The arrangement determination unit 34d is based on the plurality of input images, variable information and rotation information tables, the arrangement information of the cameras 2c-1 to 2c-N, the position information of the observer 5, and the arrangement information of the object. Thus, the arrangement position and orientation of the camera 2c-n are determined.

  Specifically, the arrangement determining unit 34d first determines whether or not there is an image that has been captured without any background loss among the plurality of input images. When there is an image that is captured without a background missing, the arrangement determining unit 34d determines the current positions of the cameras 2c-1 to 2c-N as the arrangement positions of the cameras 2c-1 to 2c-N. In this case, the arrangement determination unit 34d outputs a camera presence notification to the selection unit 38d.

  On the other hand, when there is no image captured without a missing background, the arrangement determination unit 34d outputs a camera non-notification to the selection unit 38d. Further, the arrangement determining unit 34d determines a position moved by a predetermined distance from the current position of the camera 2c-n selected by the selection unit 38d as the arrangement position of the selected camera 2c-n. Further, the arrangement determining unit 34d refers to the rotation information table, and acquires a corresponding rotation angle value from the arrangement position and camera angle of the selected camera 2c-n. The arrangement determining unit 34d determines the acquired rotation angle value as the direction of the camera 2c-n. When there is no image captured without a background missing, the arrangement determining unit 34 d outputs the identification information of the selected camera 2 c-n and the camera movement destination position information to the arrangement unit 39.

  The selection unit 38d functions as a determination device that determines the camera 2c-n to be selected. The selection unit 38d receives the camera presence notification or the camera non-notification output from the arrangement determination unit 34d and the variable information stored in the storage unit 33c. When the camera non-notification is acquired from the arrangement determination unit 34d, the selection unit 38d selects the camera 2c-n based on the input variable information so that the projection conversion amount is reduced. For example, the selection unit 38d selects the camera 2c-n whose camera angle is equal to or smaller than a predetermined angle among the cameras 2c-1 to 2c-N. The predetermined angle is, for example, 30 degrees. The selection unit 38d selects the camera 2c-n based on the distance difference when there are a plurality of cameras 2c-n having a camera angle equal to or smaller than a predetermined angle. Specifically, the selection unit 38d selects the camera 2c-n whose distance difference is equal to or smaller than a predetermined value. The predetermined value of the distance difference is, for example, about 10% of the distance from the initial position of the camera to the background. The selection unit 38d selects the camera 2-n having the smallest distance difference when there are a plurality of cameras 2-n having a camera angle smaller than the predetermined angle or when there are a plurality of cameras 2-n having a distance difference equal to or smaller than a predetermined value. You may choose. Thereafter, the selection unit 38d outputs the identification information of the selected camera 2c-n to the arrangement determination unit 34d.

  In addition, when the camera presence notification is acquired from the arrangement determination unit 34d, the selection unit 38d, based on the input variable information, the camera 2c- whose camera angle is equal to or less than a predetermined angle among the cameras 2c-1 to 2c-N Select n. The selection unit 38d selects the camera 2c-n based on the distance difference when there are a plurality of cameras 2c-n having a camera angle equal to or smaller than a predetermined angle. Specifically, the selection unit 38d selects the camera 2c-n whose distance difference is equal to or smaller than a predetermined value. The selection unit 38d selects the camera 2-n having the smallest distance difference when there are a plurality of cameras 2-n having a camera angle smaller than the predetermined angle or when there are a plurality of cameras 2-n having a distance difference equal to or smaller than a predetermined value. You may choose. Thereafter, the selection unit 38d outputs the identification information of the selected camera 2c-n and the determined arrangement position information of the selected camera 2c-n to the cutting unit 36.

  FIG. 17 is a flowchart illustrating a process flow of the image processing apparatus 3d according to the fourth embodiment. In the description of FIG. 17, it is assumed that the process by the variable determination unit 32 has already been performed. In FIG. 17, the same processes as those in FIG. 12 are denoted by the same reference numerals as those in FIG. In FIG. 17, processing when still images are acquired from a plurality of cameras 2c-1 to 2c-N will be described.

  In the process of step S202, when there is no image captured without a missing background (step S202—NO), the arrangement determination unit 34d outputs a camera non-notification to the selection unit 38d. When the camera non-notification is acquired from the arrangement determination unit 34d, the selection unit 38d selects a camera 2c-n having a camera angle equal to or less than a predetermined angle based on the variable information stored in the storage unit 33c. When there are a plurality of cameras 2c-n having a predetermined angle or less, the selection unit 38d selects the cameras 2c-n having a distance difference of a predetermined value or less (step S401). The predetermined value of the distance difference is, for example, about 10% of the distance from the initial position of the camera to the background. Thereafter, the selection unit 38d outputs the identification information of the selected camera 2c-n to the arrangement determination unit 34d. In step S401, the selection unit 38d determines that the camera having the smallest distance difference is present when there are a plurality of cameras 2c-n whose camera angle is smaller than the predetermined angle or when there are a plurality of cameras 2c-n whose distance difference is equal to or smaller than the predetermined value. 2-n may be selected.

  The arrangement determination unit 34d determines a position moved by a predetermined distance from the current position of the camera 2c-n identified by the identification information output from the selection unit 38d as the arrangement position of the selected camera 2c-n. Further, the arrangement determining unit 34d refers to the rotation information table and determines the orientation of the camera 2c-n from the determined arrangement position and the camera angle of the selected camera 2c-n (step S402). The arrangement determining unit 34d outputs the identification information of the selected camera 2c-n and the camera movement destination arrangement information to the arrangement unit 39.

  The placement unit 39 moves the camera 2c-n identified by the identification information to the camera destination position based on the identification information of the camera 2c-n output from the placement determination unit 34d and the camera destination location information. (Step S403). Thereafter, the placement unit 39 selects the camera 2c-n identified by the identification information based on the identification information of the camera 2c-n output from the placement determination unit 34d and the rotation angle included in the camera movement destination placement information. Then, it is rotated by the angle indicated by the rotation angle (step S404). The image acquisition unit 30 acquires an image captured by the camera 2c-n after movement and rotation (step S405). The image acquisition unit 30 records the acquired image in the storage unit 33c.

The arrangement determining unit 34d acquires an image captured by the camera 2c-n after movement and rotation, which is stored in the storage unit 33c. The arrangement determination unit 34d has no background based on the acquired image, the position information of the observer 5 input via the operation unit 31, the arrangement information of the object, and the arrangement information of the camera 2c-n. It is determined whether or not there is an image captured in step S406. If there is no image captured without a missing background (step S406—NO), the image processing device 3d repeatedly executes the processing after step S402.
On the other hand, when there is an image that has been imaged with no background missing (step S406—YES), the image processing device 3d executes the process of step S208.

  When the image acquisition unit 30 acquires a moving image, differences from the processing illustrated in FIG. 17 will be described. In step S201, the image acquisition unit 30 acquires a moving image from each of the plurality of cameras 2c-1 to 2c-N, and acquires each frame image from each of the acquired moving images. In step S202, the arrangement determining unit 34d determines whether or not there is an image that is captured without any background missing, based on the first frame image of each frame image of each acquired moving image. If there is an image that is captured without any background missing, in step S208, the cutout unit 36 cuts out a background partial image for each frame image of the moving image acquired from the selected camera 2c-n. Next, the cutout unit 36 determines the display position of the background partial image so as to be a common display position in each frame image. Thereafter, the output unit 37 outputs the background partial images corresponding to the number of frame images and the display position information common to the frame images to the display device 4.

On the other hand, in step S202, when there is no image captured without a missing background, in step S406, the image acquisition unit 30 acquires and acquires moving images from the plurality of cameras 2c-n after movement and rotation. Each frame image is acquired from the moving image. In step S406, the arrangement determining unit 34d determines whether or not there is an image that is captured without missing a background, based on the first frame image of each frame image of the acquired moving image. When there is an image that is captured without any background missing, in step S208, the cutout unit 36 cuts out a background partial image for each frame image of the moving image acquired from the moved and rotated camera 2c-n. Next, the cutout unit 36 determines the display position of the background partial image so as to be a common display position in each frame image. Thereafter, the output unit 37 outputs the background partial images corresponding to the number of frame images and the display position information common to the frame images to the display device 4.
On the other hand, in step S406, when there is no image captured without a missing background, the image processing device 3d executes the processing from step S402 onward.

  As described above, in the image processing device 3d of the fourth embodiment, the camera 2c whose position and orientation are determined so as to image the object 6 placed in the real space is placed in the real space, and the display device 4 Arrangement of a camera 2c that outputs a background image that is observed as if the object 6 has been removed in the space of the image displayed on the screen and that is hidden by the object 6 and cannot be seen by the user A determination device for determining the position is provided. The determination apparatus according to the fourth embodiment includes an arrangement determination unit 34d and a selection unit 38d. The selection unit 38d selects the camera 2c-n based on the camera angle, the positions of the cameras 2c-1 to 2c-N, the position of the user, and the distance among the plurality of cameras 2c-1 to 2c-N. The arrangement determination unit 34d can capture an image of the selected camera 2c-n without missing a background that is not visible to the observer 5 with a specific object in the region that the observer 5 is viewing via the display device 4. The arrangement position of the camera 2c-n which is the movement destination of the camera 2c-n is determined. Thus, by acquiring an image captured from the determined arrangement position, the image processing device 3d can acquire a background image that is blocked by a specific object and cannot be seen by the user.

<Modification>
The image processing apparatus 3d may execute the process of FIG. 17 when the specific object moves or the specific object is changed, or may execute the process of FIG. 17 when the observer 5 moves. Good.
The image acquisition unit 30 may output the image acquired from the camera 2c-n after the movement and orientation are adjusted to the cutout unit 36. When configured in this way, the arrangement unit 39 outputs a notification indicating that the movement and orientation adjustment of the camera 2c-n has been completed to the image acquisition unit 30. The notification output by the placement unit 39 includes, for example, identification information of the camera 2c-n. When the image acquisition unit 30 acquires a notification from the placement unit 39 and then acquires an image from the camera 2c-n identified by the identification information, the image acquisition unit 30 outputs the acquired image to the cutout unit 36. The cutout unit 36 inputs the image output from the image acquisition unit 30 and the background information and pasting position information input via the operation unit 31. The cutout unit 36 cuts out the background partial image from the input image. The cutout unit 36 determines the display position of the background partial image on the display screen based on the pasting position information. For example, the area specified by the pasting position information specified by the operation unit 31 is determined as the display position of the background partial image on the display screen. In this case, the arrangement determining unit 34d may not output the identification information of the camera 2c-n whose arrangement position has been determined to the cutting unit 36.

(Fifth embodiment)
In the first embodiment, the configuration in which one camera is provided in the image processing system and the camera is moved to a position where the entire background can be imaged is shown. However, in the first embodiment, even if the camera is moved to a position where the entire background can be imaged, the background image that is blocked by a specific object and cannot be seen by the user is blurred (the background is in focus). The quality of the entire image displayed on the display device 4 deteriorates. Therefore, the fifth embodiment is different from the first embodiment in that the position where the entire background can be imaged and the position where the background is not blurred is determined as the camera arrangement position. In the fifth embodiment, only differences from the first embodiment will be described.

  FIG. 18 is a diagram illustrating an example of a configuration of an image processing system 1e according to the fifth embodiment. The image processing system 1e includes a camera 2, an image processing device 3e, and a display device 4. The image processing device 3e differs from the image processing device 3a in the method of determining the arrangement position of the camera 2. Specifically, the image processing device 3 e determines a position where the background can be captured entirely and a position where the background is not blurred as the arrangement position of the camera 2. Here, the position where the background is not blurred is a position including the background within the range of the depth of field of the camera 2.

  The image processing device 3e includes a CPU, a memory, an auxiliary storage device, and the like connected by a bus, and executes an image processing program. By executing the image processing program, the image processing apparatus 3e includes an image acquisition unit 30, an operation unit 31, a variable determination unit 32, a storage unit 33, an arrangement determination unit 34e, a movement unit 35, a cutout unit 36, and an output unit 37. Functions as a device. Note that all or part of each function of the image processing device 3e may be realized using hardware such as an ASIC, PLD, or FPGA. The image processing program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. Further, the image processing program may be transmitted / received via a telecommunication line.

  The image processing device 3e is different in configuration from the image processing device 3a in that an arrangement determining unit 34e is provided instead of the arrangement determining unit 34. The image processing device 3e is the same as the image processing device 3a in other configurations. Therefore, description of the entire image processing apparatus 3e is omitted, and the arrangement determining unit 34e will be described.

  The arrangement determining unit 34 e functions as a determining device that determines the arrangement position of the camera 2. The arrangement determination unit 34e includes the image and variable information stored in the storage unit 33, the arrangement information of the camera 2 in the real space input via the operation unit 31, the position information of the observer 5, and the real space. Object arrangement information and parameter information about the camera 2 (hereinafter referred to as “camera parameter information”) are input. The camera parameter information is information on parameters used for calculating the front depth of field and the rear depth of field, and is, for example, an allowable confusion circle diameter, an aperture value, and a focal length. The arrangement determining unit 34e determines the arrangement position of the camera 2 based on the input image and variable information, the arrangement information of the camera 2, the position information of the observer 5, the arrangement information of the object, and the camera parameter information. To do. Specifically, the arrangement determining unit 34e determines the position where the background can be captured and the position where the background is not blurred as the arrangement position of the camera 2.

An arrangement example of the object and the camera 2 in the image processing system 1e is shown in FIG. FIG. 19 is an overhead view showing an example of the arrangement of the object and the camera 2. In FIG. 19, the distance from the camera 2 to the background is L1, the distance from the camera 2 to the object 6 is D1, the rear depth of field in the camera 2 is d11, and the front depth of field in the camera 2 is d21.
The forward depth of field and the backward depth of field are calculated by existing methods. For example, the front depth of field and the rear depth of field may be calculated based on Equations 1 and 2.

  The subject distance in the above formulas 1 and 2 is the distance from the camera 2 to the object 6, and the permissible circle of confusion is constant for all the cameras 2. The focal length and the aperture value are values set for each camera 2 and are different for each camera 2. The arrangement determining unit 34e determines the position where the background can be imaged and satisfies the following Expression 3 as the arrangement position of the camera 2. In Expression 3, Di (i is an integer of 1 or more) represents the distance from the camera 2 to the object 6, d1i represents the rear depth of field in the camera 2, and d2i represents the forward depth of field in the camera 2. Li represents the distance from the camera 2 to the background.

  In the example shown in FIG. 19, the position of the camera 2 is not a position where the background is not blurred. That is, the position of the camera 2 is not a position that includes the background within the range of the depth of field. As a result, even if the camera 2 captures an image with no background, the background is blurred and the image is captured. Therefore, the arrangement determining unit 34e determines the position where the background can be captured and the position where the background is not blurred as the arrangement position of the camera 2.

  FIG. 20 is a flowchart illustrating a process flow of the image processing apparatus 3e according to the fifth embodiment. The processes similar to those in FIG. 8 are denoted by the same reference numerals as those in FIG. In the description of FIG. 20, it is assumed that the processing by the variable determination unit 32 has already been performed. In FIG. 20, processing when a still image is acquired from the camera 2 will be described.

  In the process of step S102, when there is no image captured without a lack of background (step S102-NO), the arrangement determination unit 34e inputs the input image and variable information, the arrangement information of the camera 2, and the observer 5's information. Based on the position information, the object arrangement information, and the camera parameter information, the arrangement position of the camera 2 is determined (step S501). Specifically, the arrangement determining unit 34e determines a position where the background is not blurred and a position where the camera 2 is moved by a predetermined distance from the current position of the camera 2 as the arrangement position of the camera 2.

  Specifically, first, the arrangement determination unit 34e uses the camera parameter information acquired from the camera 2, the arrangement information of the camera 2, and the arrangement information of the object in the real space, based on the above formulas 1 and 2. The front depth of field and the rear depth of field of the camera 2 are calculated. Next, the arrangement determining unit 34e determines whether the camera 2 to the object 6 are obtained from the calculated front depth of field (d2i) and rear depth of field (d1i), the arrangement information of the camera 2, and the arrangement information of the object. Using the distance D and the distance L from the camera 2 to the background, a plurality of placement position candidates for the camera 2 that satisfy the above-described expression 3 are determined. Next, the arrangement determining unit 34e compares the position of the candidate for the arrangement position of the camera 2 with the position obtained by moving the camera 2 from the current position of the camera 2 by a predetermined distance. The position where the camera 2 is moved by a predetermined distance from the current position 2 is determined as the arrangement position of the camera 2. If there are a plurality of positions that coincide with the position where the camera 2 is moved by a predetermined distance from the current position of the camera 2, the arrangement determining unit 34 e is closest to the position of the observer 5 and the position of the arrangement position candidate, The position where the camera 2 is moved by a predetermined distance from the current position of the camera 2 is determined as the position where the camera 2 is arranged.

  Here, the position where the camera 2 is moved by a predetermined distance is calculated based on the arrangement information of the camera 2 by calculating an area that is imaged on the same plane as the background and moving the camera 2 in a direction closer to the calculated background area. (For example, about 10% of the distance from the initial position of the camera to the background). That is, the arrangement determining unit 34e determines a position where the camera 2 is moved by a predetermined distance in a direction approaching the background area and the background satisfying the above expression 3 is not blurred as the arrangement position of the movement destination of the camera 2. . Thereafter, the arrangement determining unit 34 e outputs the determined arrangement position information to the moving unit 35. Thereafter, the image processing device 3e executes the processing after step S104.

  On the other hand, when there is an image that is imaged without a missing background (YES in step S102), the arrangement determining unit 34e determines whether the camera 2 is installed at a position where the background is not blurred (step S502). The determination as to whether or not the camera 2 is installed at a position where the background is not blurred is performed based on Equation 3 above. Specifically, first, the arrangement determination unit 34e uses the camera parameter information acquired from the camera 2, the arrangement information of the camera 2, and the arrangement information of the object in the real space, based on the above formulas 1 and 2. The front depth of field and the rear depth of field of the camera 2 are calculated. Next, the arrangement determining unit 34e determines whether the camera 2 to the object 6 are obtained from the calculated front depth of field (d2i) and rear depth of field (d1i), the arrangement information of the camera 2, and the arrangement information of the object. Using the distance D and the distance L from the camera 2 to the background, it is determined whether or not the condition of Equation 3 is satisfied. And the arrangement | positioning determination part 34e determines with the camera 2 being installed in the position where a background is not blurred, when the arrangement position of the camera 2 satisfy | fills the conditions of said Formula 3. FIG. On the other hand, the arrangement determination unit 34e determines that the camera 2 is not installed at a position where the background is not blurred when the arrangement position of the camera 2 does not satisfy the condition of the above expression 3.

When the camera 2 is installed at a position where the background is not blurred (step S502—YES), the image processing device 3e executes the process of step S107.
On the other hand, when the camera 2 is not installed at a position where the background is not blurred (step S502-NO), the arrangement determining unit 34e determines the arrangement position of the camera 2 (step S503). Specifically, the arrangement determining unit 34e determines, as the arrangement position of the camera 2, a position where the camera 2 is moved by a predetermined distance from the current position of the camera 2 and a position that does not blur the background satisfying the above Expression 3. Here, the position where the camera 2 is moved by a predetermined distance is calculated based on the arrangement information of the camera 2 by calculating an area that is imaged on the same plane as the background and moving the camera 2 in a direction closer to the calculated background area. (For example, about 10% of the distance from the initial position of the camera to the background). The arrangement determining unit 34 e outputs the determined arrangement position information to the moving unit 35.

  The moving unit 35 moves the camera 2 to the position indicated by the determined arrangement position information based on the determined arrangement position information output from the arrangement determining unit 34e (step S504). The image acquisition unit 30 acquires an image captured by the camera 2 after movement (step S505). For example, the image acquisition unit 30 may acquire an image captured by the camera 2 as an image captured by the camera 2 after movement after receiving a notification that the movement of the camera 2 is completed from the movement unit 35. Good. The image acquisition unit 30 records the acquired image in the storage unit 33. Thereafter, the image processing device 3e executes the processing after step S102.

  When the image acquisition unit 30 acquires a moving image, differences from the processing illustrated in FIG. 20 will be described. In step S101, the image acquisition unit 30 acquires a moving image from the camera 2 and acquires each frame image from the acquired moving image. In step S <b> 102, the arrangement determining unit 34 e determines whether there is an image that has been captured without any background missing based on the first frame image of each frame image of the acquired moving image. If there is an image that is captured without any background missing, in step S107, the cutout unit 36 cuts out a background partial image for each frame image of the moving image acquired from the camera 2. Next, the cutout unit 36 determines the display position of the background partial image so as to be a common display position in each frame image. Thereafter, the output unit 37 outputs the background partial images corresponding to the number of frame images and the display position information common to the frame images to the display device 4.

On the other hand, in step S102, when there is no image captured without a missing background, in step S105, the image acquisition unit 30 acquires a moving image from the camera 2 after movement, and each frame image from the acquired moving image. To get. In step S <b> 106, the arrangement determining unit 34 e determines whether there is an image that is captured without any background missing, based on the first frame image of each frame image of the acquired moving image. If there is an image that is captured without any background missing, in step S107, the cutout unit 36 cuts out a background partial image for each frame image of the moving image acquired from the camera 2 after movement. Next, the cutout unit 36 determines the display position of the background partial image so as to be a common display position in each frame image. Thereafter, the output unit 37 outputs the background partial images corresponding to the number of frame images and the display position information common to the frame images to the display device 4.
On the other hand, in step S106, when there is no image captured without a missing background, the image processing device 3e executes the processing after step S501.

  As described above, in the image processing device 3e of the fifth embodiment, the camera 2 whose position and orientation are determined so as to image the object 6 placed in the real space is placed in the real space, and the display device 4 Arrangement of the camera 2 that outputs a background image that is observed as if the object 6 has been removed in the space of the image displayed on the screen and that is hidden by the object 6 and cannot be seen by the user A determination device for determining the position is provided. The determining apparatus of the fifth embodiment includes an arrangement determining unit 34e. The arrangement determining unit 34e determines the position where the background can be captured and the position where the background is not blurred as the arrangement position of the camera 2. Thus, by acquiring an image captured from the determined arrangement position, the image processing device 3e can acquire a background image that is blocked by a specific object and cannot be seen by the user.

<Modification>
The image processing device 3e according to the fifth embodiment may be modified in the same manner as the image processing device 3a according to the first embodiment.

(Sixth embodiment)
In the second embodiment, a configuration is shown in which one camera is selected from a plurality of cameras provided in the image processing system, and the selected camera is moved to a position where the entire background can be imaged. However, in the second embodiment, even if the camera is moved to a position where the entire background can be imaged, the background image that is blocked by a specific object and cannot be seen by the user is blurred (focus on the background). The quality of the entire image displayed on the display device 4 deteriorates. Therefore, the sixth embodiment is different from the second embodiment in that the camera arrangement position selected from a plurality of cameras is determined to be a position where all the backgrounds can be captured and the background is not blurred. Is different. In the sixth embodiment, only differences from the second embodiment will be described.

  FIG. 21 is a diagram illustrating an example of a configuration of an image processing system 1f according to the sixth embodiment. The image processing system 1f includes cameras 2-1 to 2-N, an image processing device 3f, and a display device 4. The image processing device 3e differs from the image processing device 3b in the method of selecting the camera 2 and the method of determining the arrangement position of the camera 2. Specifically, the image processing device 3f selects the camera 2-n having a high evaluation value from the plurality of cameras 2-1 to 2-N, and determines the arrangement position of the selected camera 2-n based on the background. The position is determined so that all the images can be captured and the background is not blurred.

  The image processing device 3f includes a CPU, a memory, an auxiliary storage device, and the like connected by a bus, and executes an image processing program. By executing the image processing program, the image processing apparatus 3c causes the image acquisition unit 30, the operation unit 31f, the variable determination unit 32, the storage unit 33, the arrangement determination unit 34f, the movement unit 35, the cutout unit 36, the output unit 37, and the selection. It functions as a device including the unit 38f. Note that all or part of the functions of the image processing device 3f may be realized by using hardware such as an ASIC, PLD, or FPGA. The image processing program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. Further, the image processing program may be transmitted / received via a telecommunication line.

  The image processing device 3f is different in configuration from the image processing device 3b in that an operation unit 31f, an arrangement determination unit 34f, and a selection unit 38f are provided instead of the operation unit 31, the arrangement determination unit 34b, and the selection unit 38. The image processing device 3f is the same as the image processing device 3b in other configurations. Therefore, description of the entire image processing apparatus 3f is omitted, and the operation unit 31f, the arrangement determination unit 34f, and the selection unit 38f will be described.

  The operation unit 31 f performs the same processing as the operation unit 31. The operation unit 31f acquires priority information. The priority information is information relating to a weighting coefficient of a parameter used for camera selection (hereinafter referred to as “selection parameter”). The parameters used for camera selection are the distortion (camera angle) of the object on the image, the size of the object on the image, and the amount of blur. When the distance from the camera to the object is L, the focal length of the camera is f, and the size of the object on the image is y, the size y of the object on the image is expressed by the following Expression 4.

  In addition, the operation unit 31f may acquire preset priority information, or may acquire priority information by inputting via an input device. The operation unit 31f outputs the acquired priority information to the selection unit 38f.

  The arrangement determining unit 34f functions as a determining apparatus that determines the arrangement position of the camera 2-n. The arrangement determination unit 34f includes identification information of the camera 2-n selected by the selection unit 38f, images and variable information captured by the cameras 2-1 to 2-N stored in the storage unit 33, and operations. The position information of the observer 5 input via the unit 31, the object arrangement information in the real space, the arrangement information of the cameras 2-1 to 2-N in the real space, and the camera parameter information are input. . The arrangement determining unit 34f is based on the input image and variable information, the arrangement information of each of the cameras 2-1 to 2-N, the position information of the observer 5, the arrangement information of the object, and the camera parameter information. The arrangement position of the camera 2-n is determined. Specifically, the arrangement determining unit 34f determines the position where the background can be captured and the position where the background is not blurred as the arrangement position of the camera 2-n.

  In addition, the arrangement determining unit 34f determines whether there is an image that has been captured without missing the background among the plurality of input images. When there is an image that is captured without a background missing, a notification that there is a camera 2 that is imaged without a background missing is output to the selection unit 38f. In this case, the arrangement determination unit 34f outputs, together with the notification, identification information of the camera 2 that is imaging without missing the background to the selection unit 38f. On the other hand, when there is no image captured without missing the background, a notification that there is no camera 2 that is imaged without missing the background is output to the selection unit 38f.

  In addition, the arrangement determination unit 34f determines a position moved by a predetermined distance from the current position of the camera 2-n selected by the selection unit 38f as the arrangement position of the selected camera 2-n. The arrangement determining unit 34f outputs the identification information of the selected camera 2-n and the determined arrangement position information to the moving unit 35.

  The selection unit 38f functions as a determination device that determines the camera 2-n to be selected. The selection unit 38f receives the notification output from the arrangement determination unit 34f, the image and variable information stored in the storage unit 33, and the cameras 2-1 to 2-N input via the operation unit 31f. Arrangement information, position information of the observer 5, object arrangement information in real space, priority information, and camera parameter information are input. The selection unit 38f acquires camera parameter information from each camera 2. The selection unit 38f picks up an image without missing a background based on the input image, the arrangement information of each of the cameras 2-1 to 2-N, the position information of the observer 5, and the arrangement information of the object. A camera 2-n is selected from a plurality of cameras 2-1 to 2-N. The selection unit 38f outputs the identification information of the selected camera 2-n to the arrangement determination unit 34f.

  FIG. 22 is a flowchart illustrating a processing flow of the image processing device 3f according to the sixth embodiment. The processes similar to those in FIG. 12 are denoted by the same reference numerals as those in FIG. In the description of FIG. 22, it is assumed that the process by the variable determination unit 32 has already been performed. In addition, FIG. 22 illustrates processing when still images are acquired from the plurality of cameras 2-1 to 2-N.

  In the process of step S202, if there is an image that has been captured without any background missing (step S202—YES), the selection unit 38f is identified by the identification information output from the placement determination unit 34f. It is determined whether or not there is a camera 2 installed at a position where the background is not blurred in -N (step S601). When there is a camera 2 installed at a position where the background is not blurred (step S601-YES), the selection unit 38f does not lose the background from the cameras 2 identified by the identification information output from the arrangement determination unit 34f. A camera 2-n that is imaged and is installed at a position where the background is not blurred is selected (step S602).

  Specifically, when there is one camera 2 installed at a position where the background is not blurred, the selection unit 38f selects the camera 2-n installed at a position where the background is not blurred. In addition, when there are a plurality of cameras 2 installed at positions where the background is not blurred, the selection unit 38f calculates a camera evaluation value for each of the cameras 2 installed at positions where the background is not blurred. Here, the camera evaluation value is a value used for evaluation of each camera 2, for example, each parameter (distortion of the object on the image (camera angle), the size of the object on the image, and the amount of blur) in the selection parameter. It is represented by the total value of the values weighted on the evaluation value.

  First, the selection unit 38f acquires a weighting coefficient for each parameter from the priority information input via the operation unit 31f. For example, a weighting coefficient for each parameter may be determined in advance as priority information. Next, the selection unit 38 f evaluates each parameter for each camera 2. For example, the selection unit 38f assigns a higher value to each camera 2 in order from the camera 2 with the smallest camera angle as an evaluation of the distortion of the object on the image. As a specific example, when there are five cameras 2, the selection unit 38 f sets values “5”, “4”, “3”, “2”, “1” in order from the camera 2 with the smallest camera angle. Attach to camera 2.

  Further, for example, the selection unit 38f evaluates the size of the object on the image by determining the size of the background region in the region viewed by the observer 5 and the background region on the image captured by each camera 2. A high value is assigned to each camera 2 in order from the camera 2 having the closest size. As a specific example, when there are five cameras 2, the selection unit 38 f determines the size of the background region in the region viewed by the observer 5 and the size of the background region on the image captured by each camera 2. The values “5”, “4”, “3”, “2”, and “1” are assigned to each camera 2 in order from the camera 2 that is close to the camera. The camera 2 in which the size of the background region in the region viewed by the observer 5 and the size of the background region on the image captured by each camera 2 are close, that is, the camera 2 having a close angle of view.

  For example, the selection unit 38f assigns a higher value to each camera 2 in order from the camera 2 with the longer depth of field as the evaluation of the blur amount. As a specific example, when there are five cameras 2, the selection unit 38 f selects “5”, “4”, “3”, “2”, “1” in order from the camera 2 with the long depth of field. The value “” is assigned to each camera 2.

A method for calculating the camera evaluation value will be described with a specific example.
(Weighting factor)
-Distortion of the object on the image: 0.5
-The size of the object on the image: 0.3
・ Bokeh amount: 0.2

(Evaluation of each parameter)
・ Distortion of the object on the image: 2nd out of 5 units "4"
-The size of the object on the image: 1st out of 5 units "5"
・ Bokeh amount: 3rd out of 5 units "3"

Assuming the above (weighting coefficient) and (evaluation of each parameter), the selection unit 38f calculates the camera evaluation value of the camera 2-n as follows.
0.5 × 4 + 0.3 × 3 + 3 × 0.2 = 3.5

  The selection unit 38f calculates camera evaluation values in all the cameras 2 identified by the identification information output from the arrangement determination unit 34f. Thereafter, the selection unit 38f is a camera 2-n that captures an image without losing the background, and is a camera 2-n having a camera evaluation value equal to or greater than a predetermined threshold among the cameras 2 installed at a position where the background is not blurred. Select. When there are a plurality of cameras 2-n having camera evaluation values equal to or greater than a predetermined threshold, the selection unit 38f may select the camera 2-n having the highest camera evaluation value. The selection unit 38f outputs the identification information of the selected camera 2-n to the cutout unit 36. Thereafter, the image processing device 3f executes the process of step S208.

  Further, in the process of step S202, when there is no image captured without a missing background (step S202-NO), or when there is no camera 2 installed at a position where the background is not blurred (step S601-NO). The selection unit 38f selects the camera 2-n from the plurality of cameras 2-1 to 2-N (step S603). Specifically, it is as follows.

(When there is no image captured without a missing background)
Based on the variable information for the camera 2, the selection unit 38 f selects the camera 2-n whose camera angle is equal to or smaller than a predetermined angle. When there is one camera 2-n having a predetermined angle or less, the selection unit 38f selects the camera 2-n having a predetermined angle or less. When there are a plurality of cameras 2-n having a predetermined angle or less, the selection unit 38f selects a camera 2-n having a distance difference of a predetermined value or less. The predetermined value of the distance difference is, for example, about 10% of the distance from the initial position of the camera to the background. Thereafter, the selection unit 38f outputs the identification information of the selected camera 2-n to the arrangement determination unit 34f. The selection unit 38f also includes the camera 2-n having the smallest distance difference when there are a plurality of cameras 2-n having a camera angle smaller than a predetermined angle or when there are a plurality of cameras 2-n having a distance difference equal to or smaller than a predetermined value. n may be selected.

(When there is an image captured with no background missing and there is no camera 2 installed at a position where the background is not blurred)
The selection unit 38f is a camera whose camera angle is equal to or smaller than a predetermined angle based on the variable information for the camera 2 identified by the identification information output from the arrangement determination unit 34f among the variable information stored in the storage unit 33. Select 2-n. When there is one camera 2-n having a predetermined angle or less, the selection unit 38f selects the camera 2-n having a predetermined angle or less. When there are a plurality of cameras 2-n having a predetermined angle or less, the selection unit 38f selects a camera 2-n having a distance difference of a predetermined value or less. The predetermined value of the distance difference is, for example, about 10% of the distance from the initial position of the camera to the background. Thereafter, the selection unit 38f outputs the identification information of the selected camera 2-n to the arrangement determination unit 34f. The selection unit 38f also includes the camera 2-n having the smallest distance difference when there are a plurality of cameras 2-n having a camera angle smaller than a predetermined angle or when there are a plurality of cameras 2-n having a distance difference equal to or smaller than a predetermined value. n may be selected. The selection unit 38f outputs the identification information of the selected camera 2-n to the arrangement determination unit 34f.

  The placement determination unit 34f determines the placement position of the camera 2-n identified by the identification information output from the selection unit 38f (step S604). Specifically, the arrangement determining unit 34f selects the camera 2-n from the current position of the camera 2-n identified by the position where the background satisfying Equation 3 is not blurred and the identification information output from the selection unit 38f. A position moved by a predetermined distance is determined as an arrangement position of the camera 2-n. More specifically, the arrangement determining unit 34f determines the arrangement position of the camera 2-n based on the above Equations 1 to 3, similarly to the processing performed by the arrangement determining unit 34e. Here, the position where the camera 2 is moved by a predetermined distance is calculated based on the arrangement information of the camera 2 by calculating an area that is imaged on the same plane as the background and moving the camera 2 in a direction closer to the calculated background area. (For example, about 10% of the distance from the initial position of the camera to the background). Thereafter, the arrangement determining unit 34f outputs the identification information of the selected camera 2-n and the determined arrangement position information to the moving unit 35. Thereafter, the image processing device 3f executes the processes in and after step S205.

  Differences from the processing shown in FIG. 22 will be described in the case where the image acquisition unit 30 acquires a moving image. In step S201, the image acquisition unit 30 acquires a moving image from each of the plurality of cameras 2-1 to 2-N, and acquires each frame image from each of the acquired moving images. In step S202, the arrangement determining unit 34f determines whether or not there is an image that is captured without any background missing, based on the first frame image of each frame image of each acquired moving image. If there is an image that is captured without any background missing, in step S208, the cutout unit 36 cuts out a background partial image for each frame image of the moving image acquired from the selected camera 2-n. Next, the cutout unit 36 determines the display position of the background partial image so as to be a common display position in each frame image. Thereafter, the output unit 37 outputs the background partial images corresponding to the number of frame images and the display position information common to the frame images to the display device 4.

On the other hand, in step S202, when there is no image captured without a missing background, in step S206, the image acquisition unit 30 acquires a moving image from the plurality of cameras 2-n after movement, and acquires the acquired moving image. Each frame image is acquired from. In step S207, the arrangement determining unit 34f determines whether or not there is an image that is captured without any background missing based on the first frame image of each frame image of the acquired moving image. When there is an image that is captured without any background missing, in step S208, the cutout unit 36 cuts out a background partial image for each frame image of the moving image acquired from the camera 2-n after movement. Next, the cutout unit 36 determines the display position of the background partial image so as to be a common display position in each frame image. Thereafter, the output unit 37 outputs the background partial images corresponding to the number of frame images and the display position information common to the frame images to the display device 4.
On the other hand, in step S207, if there is no image captured without a missing background, the image processing device 3f executes the processing from step S603 onward.

  As described above, in the image processing device 3f of the sixth embodiment, the camera 2 whose position and orientation are determined so as to image the object 6 placed in the real space is placed in the real space, and the display device 4 Arrangement of the camera 2 that outputs a background image that is observed as if the object 6 has been removed in the space of the image displayed on the screen and that is hidden by the object 6 and cannot be seen by the user A determination device for determining the position is provided. The determination device according to the sixth embodiment includes an arrangement determination unit 34f and a selection unit 38f. The selection unit 38f selects the camera 2-n based on the camera evaluation value among the plurality of cameras 2-1 to 2-N. The arrangement determining unit 34f can capture an image of the selected camera 2-n without missing a background that is not visible to the observer 5 with a specific object in an area viewed by the observer 5 via the display device 4. The position where the background is not blurred is determined as the arrangement position of the camera 2-n to which the camera 2-n is moved. Thus, by acquiring an image captured from the determined arrangement position, the image processing device 3f can acquire a background image that is blocked by a specific object and cannot be seen by the user.

<Modification>
The image processing device 3f according to the sixth embodiment may be modified in the same manner as the image processing device 3b according to the second embodiment.
In the present embodiment, the arrangement determining unit 34f is configured to determine the arrangement position of the camera 2-n selected by the selection unit 38f based on the camera evaluation value. However, the present invention is not limited to this. For example, the arrangement determination unit 34f may be configured to determine the arrangement position of the camera 2-n selected by the selection unit 38 in the second embodiment. When configured in this manner, the image processing device 3f includes a selection unit 38 instead of the selection unit 38f.

(Seventh embodiment)
In the third embodiment, a configuration for controlling the position and orientation of the camera movement destination so that the entire background can be imaged and the influence upon conversion to an image viewed from the direction of the observer's line of sight is reduced. showed that. However, in the third embodiment, the position and orientation of the camera destination are controlled so that the entire background can be imaged and the influence when converting to an image viewed from the observer's line of sight is reduced. Even if the background image that is blocked by a specific object and cannot be seen by the user is blurred (the background is not in focus), the quality of the entire image displayed on the display device 4 Will deteriorate. Therefore, the seventh embodiment is different from the third embodiment in that the position and orientation in which the entire background can be imaged and the position where the background is not blurred are determined as the arrangement position and orientation of the camera. In the seventh embodiment, only differences from the third embodiment will be described.

  FIG. 23 is a diagram illustrating an example of a configuration of an image processing system 1g according to the seventh embodiment. The image processing system 1g includes a camera 2c, an image processing device 3g, and a display device 4. The image processing device 3g differs from the image processing device 3c in the method of determining the arrangement position and orientation of the camera 2c. Specifically, the image processing device 3c determines the position and orientation in which the entire background can be imaged and the position where the background is not blurred as the arrangement position and orientation of the camera 2c.

  The image processing device 3g includes a CPU, a memory, an auxiliary storage device, and the like connected by a bus, and executes an image processing program. By executing the image processing program, the image processing apparatus 3c includes an image acquisition unit 30, an operation unit 31, a variable determination unit 32, a storage unit 33c, an arrangement determination unit 34g, a cutout unit 36, an output unit 37, and an arrangement unit 39. Functions as a device. Note that all or some of the functions of the image processing apparatus 3g may be realized using hardware such as an ASIC, PLD, or FPGA. The image processing program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. Further, the image processing program may be transmitted / received via a telecommunication line.

  The image processing device 3g is different from the image processing device 3c in that it includes a placement determination unit 34g instead of the placement determination unit 34. The image processing device 3g is the same as the image processing device 3c in other configurations. Therefore, the description of the entire image processing device 3g is omitted, and the arrangement determining unit 34g will be described.

  The arrangement determining unit 34g functions as a determining device that determines the arrangement position and orientation of the camera 2c. The arrangement determining unit 34g includes an image, variable information and rotation information table stored in the storage unit 33c, arrangement information of the camera 2c in the real space input via the operation unit 31, and position information of the observer 5. The arrangement information of the object in the real space and the camera parameter information are input. Based on the input image, variable information and rotation information table, the placement information of the camera 2c, the position information of the observer 5, the placement information of the object, and the camera parameter information, the placement determination unit 34g Determine the placement position and orientation. Specifically, the arrangement determining unit 34g determines the position and orientation in which the background can be captured and the position where the background is not blurred as the arrangement position and orientation of the camera 2.

  FIG. 24 is a flowchart showing the flow of processing of the image processing apparatus 3g in the seventh embodiment. The processing similar to that in FIG. 15 is denoted by the same reference numerals as those in FIG. In the description of FIG. 24, it is assumed that the processing by the variable determination unit 32 has already been performed. In addition, FIG. 24 will describe processing when still images are acquired from a plurality of cameras 2c-1 to 2c-N.

  In the process of step S102, when there is no image captured without a missing background (step S102-NO), the arrangement determination unit 34g inputs the input image, variable information and rotation information table, arrangement information of the camera 2c, Based on the position information of the observer 5, the object arrangement information, and the camera parameter information, the arrangement position and orientation of the camera 2c are determined (step S701). Specifically, the arrangement determining unit 34g determines the position where the background is not blurred and the position where the camera 2c is moved by a predetermined distance from the current position of the camera 2c as the arrangement position of the camera 2c. For example, the arrangement determining unit 34g determines the arrangement position of the camera 2c based on the above formulas 1 to 3, similarly to the processing performed by the arrangement determining unit 34e. Here, the position where the camera 2c is moved by a predetermined distance refers to the area where the camera 2c is imaged on the same plane as the background from the arrangement information of the camera 2c, and the camera 2c is determined in a direction closer to the calculated background area. (For example, about 10% of the distance from the initial position of the camera to the background).

  Further, the arrangement determining unit 34g refers to the rotation information table, and acquires the rotation angle based on the position of the camera 2c and the camera angle. The arrangement determining unit 34g determines the acquired rotation angle as the orientation of the camera 2c. The arrangement determining unit 34g outputs the camera movement destination arrangement information to the arrangement unit 39. Thereafter, the image processing device 3g executes the processing after step S302.

On the other hand, if there is an image that has been imaged with no background missing (step S102—YES), the arrangement determining unit 34g determines whether the camera 2c is installed at a position where the background is not blurred (step S702). The placement determination unit 34g determines whether or not the camera 2c is installed at a position where the background is not blurred based on Equation 3 above, similarly to the processing performed by the placement determination unit 34e.
When the camera 2c is installed at a position where the background is not blurred (step S702-YES), the image processing device 3g executes the process of step S107.

  On the other hand, when the camera 2c is not installed at a position where the background is not blurred (step S702-NO), the arrangement determining unit 34g determines the arrangement position and orientation of the camera 2c (step S703). Specifically, the arrangement determining unit 34g determines, as the arrangement position of the camera 2c, a position where the camera 2c is moved by a predetermined distance from the current position of the camera 2c and a position where the background satisfying Equation 3 is not blurred. Further, the arrangement determining unit 34g refers to the rotation information table, and acquires the rotation angle based on the position of the camera 2c and the camera angle. The arrangement determining unit 34g determines the acquired rotation angle as the orientation of the camera 2c. The arrangement determining unit 34g outputs the camera movement destination arrangement information to the arrangement unit 39.

  The arrangement unit 39 moves the camera 2c to the camera movement destination position based on the coordinates of the arrangement position included in the camera movement destination arrangement information output from the arrangement determination unit 34f (step S704). Thereafter, the arrangement unit 39 rotates the camera 2c by an angle indicated by the rotation angle based on the rotation angle included in the camera movement destination arrangement information (step S705). The image acquisition unit 30 acquires an image captured by the camera 2c after movement and rotation (step S706). For example, the image acquisition unit 30 receives the notification that the movement and rotation of the camera 2c has been completed from the placement unit 39, and then converts the image captured by the camera 2c as an image captured by the camera 2c after movement and rotation. You may get it. The image acquisition unit 30 records the acquired image in the storage unit 33c. Thereafter, the image processing device 30g executes the processing after step S102.

  When the image acquisition unit 30 acquires a moving image, differences from the processing illustrated in FIG. 24 will be described. In step S101, the image acquisition unit 30 acquires a moving image from the camera 2c, and acquires each frame image from the acquired moving image. In step S102, the arrangement determining unit 34g determines whether or not there is an image that is captured without any background missing based on the first frame image of each frame image of the acquired moving image. If there is an image that is captured with no background missing, in step S107, the cutout unit 36 cuts out a background partial image for each frame image of the moving image acquired from the camera 2c. Next, the cutout unit 36 determines the display position of the background partial image so as to be a common display position in each frame image. Thereafter, the output unit 37 outputs the background partial images corresponding to the number of frame images and the display position information common to the frame images to the display device 4.

On the other hand, in step S102, when there is no image captured without a missing background, in step S304, the image acquisition unit 30 acquires a moving image from the camera 2c after movement and rotation, and each image is acquired from the acquired moving image. Get a frame image. In step S305, the arrangement determining unit 34g determines whether or not there is an image that is captured without any background missing based on the first frame image of each frame image of the acquired moving image. When there is an image that is captured without any background missing, in step S107, the cutout unit 36 cuts out a background partial image for each frame image of the moving image acquired from the moved and rotated camera 2c. Next, the cutout unit 36 determines the display position of the background partial image so as to be a common display position in each frame image. Thereafter, the output unit 37 outputs the background partial images corresponding to the number of frame images and the display position information common to the frame images to the display device 4.
On the other hand, in step S305, when there is no image captured without a missing background, the image processing device 3g executes the processing from step S701 onward.

  As described above, in the image processing device 3g according to the seventh embodiment, the camera 2c whose position and orientation are determined so as to image the object 6 placed in the real space is placed in the real space, and the display device 4 Arrangement of a camera 2c that outputs a background image that is observed as if the object 6 has been removed in the space of the image displayed on the screen and that is hidden by the object 6 and cannot be seen by the user A determination device for determining the position is provided. The determining device of the seventh embodiment includes an arrangement determining unit 34g. The arrangement determination unit 34g determines the position where the background can be captured and the position where the background is not blurred as the arrangement position and orientation of the camera 2c. As a result, the image processing apparatus 3g is shielded by a specific object and can be viewed from the user by acquiring an image captured by the imaging apparatus arranged at the determined arrangement position and arranged in the determined orientation. It becomes possible to obtain a background image that cannot be obtained.

  The image processing device 3g according to the seventh embodiment may be modified in the same manner as the image processing device 3c according to the third embodiment.

(Eighth embodiment)
In the fourth embodiment, an image obtained by selecting one camera from a plurality of cameras provided in the image processing system and capturing the entire background of the selected camera as viewed from the observer's line of sight. A configuration is shown in which the position and orientation of the destination of the camera are controlled so that the influence of the conversion into the camera is reduced. However, in the fourth embodiment, even if the position and orientation of the destination of the camera are controlled so that the entire background of the camera can be imaged, the background image that is blocked by a specific object and cannot be seen by the user is blurred. If the image is displayed (the background is not in focus), the quality of the entire image displayed on the display device 4 deteriorates. Therefore, in the eighth embodiment, the fourth is that the arrangement position and orientation of a camera selected from a plurality of cameras are determined to be positions where the background can be captured and the background is not blurred. This is different from the embodiment. In the eighth embodiment, only differences from the fourth embodiment will be described.

  FIG. 25 is a diagram illustrating an example of a configuration of an image processing system 1h according to the eighth embodiment. The image processing system 1h includes cameras 2c-1 to 2c-N, an image processing device 3h, and a display device 4. The image processing device 3h differs from the image processing device 3d in the method of selecting the camera 2c and the method of determining the arrangement position and orientation of the camera 2c. Specifically, the image processing device 3h selects the camera 2c-n having a high evaluation value from the plurality of cameras 2c-1 to 2c-N, and determines the arrangement position and orientation of the selected camera 2c-n. The background is determined as a position where all the images can be captured and the background is not blurred.

  The image processing device 3h includes a CPU, a memory, an auxiliary storage device, and the like connected by a bus, and executes an image processing program. By executing the image processing program, the image processing apparatus 3h has an image acquisition unit 30, an operation unit 31h, a variable determination unit 32, a storage unit 33c, an arrangement determination unit 34h, a cutout unit 36, an output unit 37, a selection unit 38h, an arrangement. It functions as a device including the unit 39. Note that all or part of the functions of the image processing device 3h may be realized using hardware such as an ASIC, PLD, or FPGA. The image processing program may be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a portable medium such as a flexible disk, a magneto-optical disk, a ROM, a CD-ROM, or a storage device such as a hard disk built in the computer system. Further, the image processing program may be transmitted / received via a telecommunication line.

  The image processing device 3h is different from the image processing device 3d in that an operation unit 31h, an arrangement determination unit 34h, and a selection unit 38h are provided instead of the operation unit 31, the arrangement determination unit 34d, and the selection unit 38d. The other configuration of the image processing device 3h is the same as that of the image processing device 3d. Therefore, description of the entire image processing device 3h is omitted, and the operation unit 31h, the arrangement determination unit 34h, and the selection unit 38h will be described.

  The operation unit 31 h performs the same processing as the operation unit 31. The operation unit 31h acquires priority information. The operation unit 31h may acquire preset priority information, or may acquire priority information by being input via an input device. The operation unit 31h outputs the acquired priority information to the selection unit 38h.

  The arrangement determination unit 34h functions as a determination device that determines the arrangement position of the cameras 2c-n. The arrangement determination unit 34h includes the identification information of the camera 2c-n selected by the selection unit 38h, and the images, variable information, and rotation information captured by each of the cameras 2c-1 to 2c-N stored in the storage unit 33. Table, position information of the observer 5 input via the operation unit 31h, object arrangement information in real space, arrangement information of each of the cameras 2c-1 to 2c-N in real space, and camera parameter information Enter. The arrangement determination unit 34h includes a plurality of input images, variable information and rotation information tables, arrangement information of the cameras 2c-1 to 2c-N, position information of the observer 5, arrangement information of the object, and camera Based on the parameter information, the arrangement position and orientation of the camera 2c-n are determined. Specifically, the arrangement determining unit 34h determines the position where the background can be captured and the position where the background is not blurred as the arrangement position and orientation of the camera 2c-n.

  In addition, the arrangement determining unit 34h determines whether there is an image that is captured without any background loss among the plurality of input images. When there is an image captured without a background missing, a notification that there is a camera 2c capturing an image without a background is output to the selection unit 38h. In this case, the arrangement determination unit 34h outputs, together with the notification, identification information of the camera 2c that is imaging without missing the background to the selection unit 38h. On the other hand, when there is no image captured without missing the background, a notification that there is no camera 2c that is imaged without missing the background is output to the selection unit 38h.

  Further, the arrangement determining unit 34h determines a position moved by a predetermined distance from the current position of the camera 2c-n selected by the selecting unit 38h as the arrangement position of the selected camera 2c-n. Further, the arrangement determining unit 34h refers to the rotation information table, and acquires a corresponding rotation angle value from the arrangement position and camera angle of the selected camera 2c-n. The arrangement determining unit 34h determines the acquired rotation angle value as the orientation of the camera 2c-n. If there is no image captured without a background missing, the arrangement determining unit 34 f outputs the identification information of the selected camera 2 c-n and the camera movement destination position information to the arrangement unit 39.

  The selection unit 38h functions as a determination device that determines the selected camera 2c-n. The selection unit 38h includes the notification output from the placement determination unit 34h, the image and variable information stored in the storage unit 33c, and the cameras 2c-1 to 2c-N input via the operation unit 31h. Arrangement information, position information of the observer 5, object arrangement information in real space, priority information, and camera parameter information are input. The selection unit 38h acquires camera parameter information from each camera 2c. The selection unit 38h captures an image without missing a background based on the input image, the respective arrangement information of the cameras 2c-1 to 2c-N, the position information of the observer 5, and the arrangement information of the object. The selected camera 2c-n is selected from a plurality of cameras 2c-1 to 2c-N. The selection unit 38h outputs the identification information of the selected camera 2c-n to the arrangement determination unit 34h.

  FIG. 26 is a flowchart showing the flow of processing of the image processing apparatus 3h in the eighth embodiment. The processes similar to those in FIG. 17 are denoted by the same reference numerals as those in FIG. In the description of FIG. 26, it is assumed that the processing by the variable determination unit 32 has already been performed. In addition, FIG. 26 describes processing when still images are acquired from the plurality of cameras 2c-1 to 2c-N.

  In the process of step S202, if there is an image that has been captured without any background missing (step S202-YES), the selection unit 38h is identified by the cameras 2c-1 to 2c identified by the identification information output from the arrangement determination unit 34h. It is determined whether or not there is a camera 2c installed in a position where the background is not blurred in -N (step S801). The selection unit 38h determines whether there is a camera 2c installed at a position where the background is not blurred by a method similar to the method performed by the selection unit 38f. When there is a camera 2c installed at a position where the background is not blurred (step S801-YES), the selection unit 38h does not lose the background from the cameras 2c identified by the identification information output from the arrangement determination unit 34h. The camera 2c-n that is taking an image and is installed at a position where the background is not blurred is selected (step S802).

  Specifically, when there is one camera 2c installed at a position where the background is not blurred, the selection unit 38h selects the camera 2c-n installed at a position where the background is not blurred. Further, when there are a plurality of cameras 2c installed at a position where the background is not blurred, the selection unit 38h calculates a camera evaluation value for each of the cameras 2c installed at a position where the background is not blurred.

  The selection unit 38h calculates camera evaluation values in all the cameras 2c identified by the identification information output from the arrangement determination unit 34f. Thereafter, the selection unit 38h is a camera 2c that captures an image without losing the background, and among the cameras 2c that are installed at positions where the background is not blurred, the camera 2c-n having a camera evaluation value equal to or greater than a predetermined threshold value. Select. When there are a plurality of cameras 2c-n having camera evaluation values equal to or greater than a predetermined threshold, the selection unit 38h may select the camera 2c-n having the highest camera evaluation value. The selection unit 38h outputs the identification information of the selected camera 2c-n to the cutout unit 36. Thereafter, the image processing device 3f executes the process of step S208.

  Also, in the process of step S202, when there is no image captured without a missing background (step S202-NO), or when there is no camera 2c installed at a position where the background is not blurred (step S801-NO). The selection unit 38h selects the camera 2c-n from the plurality of cameras 2c-1 to 2c-N (step S803). Specifically, the selection unit 38h selects the camera 2c-n by a method similar to the method performed by the selection unit 38f based on the variable information. The selection unit 38h outputs the identification information of the selected camera 2c-n to the arrangement determination unit 34h.

  The arrangement determination unit 34h determines the arrangement position and orientation of the camera 2c-n identified by the identification information output from the selection unit 38h (step S804). Specifically, the arrangement determination unit 34h moves the camera 2c-n by a predetermined distance from the current position of the camera 2c-n identified by the identification information output from the selection unit 38h and the position where the background is not blurred. The position is determined as the arrangement position of the cameras 2c-n. Here, the position where the camera 2c-n is moved by a predetermined distance is calculated from an area captured on the same plane as the background from the arrangement information of the selected camera c-n. This is a position where the camera 2c-n is moved by a predetermined distance (for example, about 10% of the distance from the initial position of the camera to the background) in the approaching direction. Further, the arrangement determining unit 34h refers to the rotation information table, and acquires a corresponding rotation angle value from the arrangement position and camera angle of the selected camera 2c-n. The arrangement determining unit 34h determines the acquired rotation angle value as the orientation of the camera 2c-n. Thereafter, the arrangement determining unit 34 f outputs the identification information of the selected camera 2 c-n and the camera movement destination position information to the arrangement unit 39. Thereafter, the image processing device 3f executes the processing after step S403.

  Differences from the processing shown in FIG. 26 will be described in the case where the image acquisition unit 30 acquires a moving image. In step S201, the image acquisition unit 30 acquires a moving image from each of the plurality of cameras 2c-1 to 2c-N, and acquires each frame image from each of the acquired moving images. In step S202, the arrangement determining unit 34h determines whether or not there is an image that is captured without missing the background, based on the first frame image of each frame image of each acquired moving image. If there is an image that is captured without any background missing, in step S208, the cutout unit 36 cuts out a background partial image for each frame image of the moving image acquired from the selected camera 2c-n. Next, the cutout unit 36 determines the display position of the background partial image so as to be a common display position in each frame image. Thereafter, the output unit 37 outputs the background partial images corresponding to the number of frame images and the display position information common to the frame images to the display device 4.

On the other hand, in step S202, when there is no image captured without a missing background, in step S406, the image acquisition unit 30 acquires and acquires moving images from the plurality of cameras 2c-n after movement and rotation. Each frame image is acquired from the moving image. In step S406, the arrangement determining unit 34h determines whether or not there is an image that is captured without any background missing, based on the first frame image of each frame image of the acquired moving image. When there is an image that is captured without any background missing, in step S208, the cutout unit 36 cuts out a background partial image for each frame image of the moving image acquired from the moved and rotated camera 2c-n. Next, the cutout unit 36 determines the display position of the background partial image so as to be a common display position in each frame image. Thereafter, the output unit 37 outputs the background partial images corresponding to the number of frame images and the display position information common to the frame images to the display device 4.
On the other hand, in step S406, if there is no image captured without a missing background, the image processing device 3h executes the processing from step S803 onward.

  As described above, in the image processing device 3h according to the eighth embodiment, the camera 2c whose position and orientation are determined so as to image the object 6 placed in the real space is placed in the real space, and the display device 4 Arrangement of a camera 2c that outputs a background image that is observed as if the object 6 has been removed in the space of the image displayed on the screen and that is hidden by the object 6 and cannot be seen by the user A determination device for determining the position is provided. The determination device according to the eighth embodiment includes an arrangement determination unit 34h and a selection unit 38h. The selection unit 38h selects the camera 2c-n based on the camera evaluation value among the plurality of cameras 2c-1 to 2c-N. The arrangement determination unit 34h can capture an image of the selected camera 2c-n without missing a background that is not visible to the observer 5 with a specific object in an area viewed by the observer 5 via the display device 4. The position where the background is not blurred is determined as the arrangement position of the camera 2c-n which is the movement destination of the camera 2c-n. Thus, by acquiring an image captured from the determined arrangement position, the image processing device 3h can acquire a background image that is blocked by a specific object and cannot be seen by the user.

The image processing device 3h according to the eighth embodiment may be modified in the same manner as the image processing device 3d according to the fourth embodiment.
In the present embodiment, the arrangement determining unit 34h determines the arrangement position of the camera 2-n selected by the selection unit 38h based on the camera evaluation value. However, the present invention is not limited to this. For example, the arrangement determination unit 34h may be configured to determine the arrangement position of the camera 2c-n selected by the selection unit 38d in the fourth embodiment. When configured in this way, the image processing device 3h includes a selection unit 38d instead of the selection unit 38h.

  The determination device included in the image processing device in each of the above embodiments includes an imaging device whose position and orientation are determined so as to capture an image of a specific object placed in the real space, and is displayed on the screen of the display device. A background image that is a background still image or a moving image that is observed as if a specific object has been removed in the space of the displayed image, and is a background image that is blocked by the specific object and cannot be seen by the user This is used to determine the arrangement position of an imaging device that outputs an image including the image.

  You may make it implement | achieve at least one part of the image processing system and image processing apparatus in embodiment mentioned above with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be a program for realizing a part of the above-described functions, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system. You may implement | achieve using programmable logic devices, such as FPGA (Field Programmable Gate Array).

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  The present invention can be applied to an image processing apparatus that outputs an augmented reality image displayed on a display device such as a head-mounted display.

1a, 1b, 1c, 1d, 1e, 1f, 1g, 1h ... image processing system, 2 (2-1 to 2-N), 2c (2c-1 to 2c-N) ... camera, 3a, 3b, 3c, 3d, 3e, 3f, 3g, 3h ... image processing device, 4 ... display device, 5 ... observer, 6 ... object, 20, 20-1 to 20-3 ... optical axis, 21-24 ... camera image, 30 ... Image acquisition unit 31, 31f, 31h ... operation unit, 32 ... variable determination unit, 33, 33c ... storage unit, 34, 34b, 34c, 34d, 34e, 34f, 34g, 34h ... arrangement determination unit, 35 ... movement unit 36: Cutout section 37: Output section 38, 38d, 38f, 38h ... Selection section 39 ... Arrangement section

Claims (10)

An imaging device whose position and orientation are determined so as to capture a specific object arranged in the real space is arranged in the real space, and the specific object is removed in the space of the image displayed on the screen of the display device The arrangement position of the imaging device that outputs a background image that is a still image or a moving image of the observed background and that includes the background image that is blocked by the specific object and cannot be seen by the user A determination device for determining,
In the still image or moving image captured by the imaging device, when there is no still image or moving image captured without the background missing, the image is captured without missing the background based on the positional information of the imaging device. An arrangement determining unit that determines an arrangement position that is a destination of a possible imaging device;
A determination device comprising:   The arrangement determining unit is configured to move the imaging device so that an influence when the still image or the moving image captured by the imaging device is converted into the still image or the moving image viewed from the direction of the user's line of sight is reduced. The determining apparatus according to claim 1, wherein the determining position is determined.   The position determination unit determines a position where the image can be captured without losing the background and includes the background within the depth of field as a position where the image capturing apparatus is to be moved. 2. The determination device according to 1.   In the case where there is a still image or a moving image that is captured without missing the background, and the position of the imaging device is not a position that includes the background within the range of the depth of field, The determination device according to claim 1, wherein a position including the background within a range of depth of field is determined as an arrangement position of a moving destination of the imaging device. When there are a plurality of imaging devices, the image processing device further includes a selection unit that selects an imaging device from the plurality of imaging devices based on an angle formed by the user's line-of-sight direction and the optical axis of the imaging device,
The determination device according to claim 1, wherein the arrangement determination unit determines an arrangement position of the selected imaging device.   The selection unit is configured such that an angle formed by the user's line-of-sight direction and the optical axis of the imaging apparatus is equal to or smaller than a predetermined angle, and a distance from the user position to a specific position on the background and the position of the imaging apparatus The determination apparatus according to claim 5, wherein an image pickup apparatus having a difference from a distance from a specific position of the background to a specific position of the background is equal to or less than a predetermined value is selected. When there are a plurality of imaging devices, an evaluation value for a plurality of parameters in the image is calculated for each imaging device, and a total evaluation value for each imaging device is calculated by weighting the evaluation value according to priority. And a selection unit that selects an imaging device having the calculated total evaluation value equal to or greater than a threshold,
The determination device according to claim 1, wherein the arrangement determination unit determines an arrangement position of the selected imaging device. A determination device according to any one of claims 1 to 7,
An image acquisition unit that acquires a still image or a moving image captured by the imaging device;
A cutout unit that cuts out the background image from the acquired still image or moving image;
An output unit for outputting the clipped background image to the display device;
An image processing apparatus comprising: An imaging device whose position and orientation are determined so as to capture a specific object arranged in the real space is arranged in the real space, and the specific object is removed in the space of the image displayed on the screen of the display device The arrangement position of the imaging device that outputs a background image that is a still image or a moving image of the observed background and that includes the background image that is blocked by the specific object and cannot be seen by the user A determination method performed by a determination device that determines,
In the still image or moving image captured by the imaging device, when there is no still image or moving image captured without the background missing, the image is captured without missing the background based on the positional information of the imaging device. An arrangement determining step for determining an arrangement position that is a destination to which the imaging device can be moved;
Determining method. An imaging device whose position and orientation are determined so as to capture a specific object arranged in the real space is arranged in the real space, and the specific object is removed in the space of the image displayed on the screen of the display device The arrangement position of the imaging device that outputs a background image that is a still image or a moving image of the observed background and that includes the background image that is blocked by the specific object and cannot be seen by the user To the computer as a decision device to decide,
In the still image or moving image captured by the imaging device, when there is no still image or moving image captured without the background missing, the image is captured without missing the background based on the positional information of the imaging device. An arrangement determining step for determining an arrangement position that is a destination to which the imaging device can be moved;
Decision program to execute.

Images