JP2018129784A - Determination device, image processing apparatus, determination method, and determination program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a determination device, an image processing apparatus, a determination method, and a determination program that can prevent the apparent size of an image of an object observed when assumed that the object is moved in a real space, from being different from the apparent size of an image of the object virtually moved in a space of an image displayed on a display screen.SOLUTION: A determination device selects, from a plurality of cameras arranged in a real space the positions and directions of which are determined to pick up an image of an object arranged in the real space, a camera that outputs an image including an object image that is an image of the object observed to be moved in a space of a displayed image. The determination device comprises a selection part that selects a camera on the basis of at least one of the direction of a line connecting from an object actual position that is an actual position of the object to an object temporary position that is a temporary position of the object, the direction of a line connecting from a camera movement source position that is the position of the source of camera movement to a camera movement destination position that is the position of the destination of camera movement, and magnification.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a determination device, an image processing device, a determination method, and a determination program.

  Augmented Reality (AR) technology is a technology that expands and expresses reality by adding information to an image of an object in real space. Mixed reality (MR) technology means that the device changes the image of an object that exists in real space, and the device combines the image of the changed object and the image of the object that exists in virtual space. It is a technology that expresses. To virtually move an image of an object that is stationary in real space in the space of the image displayed on the display screen, depending on the position or orientation of the object in the space of the image displayed on the display screen Thus, a plurality of cameras need to be arranged in the real space. A method has been proposed in which a plurality of cameras are arranged in real space so as to surround an object, and the direction of the optical axis of the camera is determined so that the object is imaged in the center of the camera image. (See Non-Patent Document 1).

Kenji Ikeya, 3 others, “Multi-viewpoint robot camera system that can follow-up the movement of the subject”, 2011 Winter Conference of the Institute of Image Information and Television Engineers

  However, the position and orientation of an object that is stationary in the real space, the position or direction of the object that is virtually moved in the space of the image displayed on the display screen, and the positions of a plurality of cameras arranged in the real space Or, it may not always match the direction. For this reason, conventionally, the apparent size of the image of the object observed when the object is temporarily moved in the real space and the image of the object virtually moved in the space of the image displayed on the display screen There was a problem that the apparent size of the case may be different.

  In view of the above circumstances, the present invention is virtually moved in the apparent size of the image of the object observed when the object is temporarily moved in the real space and in the space of the image displayed on the display screen. It is an object of the present invention to provide a determination device, an image processing device, a determination method, and a determination program that can prevent the apparent size of an object image from differing.

  In one embodiment of the present invention, a plurality of cameras whose positions and orientations are determined so as to capture an image of an object arranged in the real space are arranged in the real space and moved in the space of the image displayed on the screen of the display device. A determination device that selects, from a plurality of cameras, the camera that outputs an image including an object image that is a still image or a moving image of the object to be observed, and is an object that is an actual position of the object The direction of the line connecting from the actual position to the temporary object position, which is the temporary position of the object, and the camera movement, which is the camera movement destination position, from the camera movement source position, which is the movement source position of the camera that images the object The determination apparatus includes a selection unit that selects the camera from a plurality of the cameras based on at least one of a direction of a line connecting to the previous position and a predetermined magnification.

  One aspect of the present invention is the above-described determination device, wherein the selection unit connects a direction of a line connecting the actual object position to the temporary object position and the camera movement source position to the camera movement destination position. The camera is selected with priority as the camera is closer to the line direction.

  One aspect of the present invention is the determination device described above, wherein the selection unit moves the camera from an observer temporary position that is a predetermined temporary position of an observer who observes the object image displayed on the screen. The camera with a short distance to the destination position is selected with priority.

  One aspect of the present invention is the above-described determination device, wherein the selection unit is configured to detect the object actual from an observer temporary position that is a predetermined temporary position of an observer who observes the object image displayed on the screen. The camera having a short difference between the distance to the position and the distance from the camera destination position to the actual object position is selected with priority.

  One aspect of the present invention is the above-described determination device, wherein when the selection unit selects the camera based on the magnification, a distance from the camera to the object is a reciprocal of the magnification. The camera at a distance determined according to the selection is selected.

  One aspect of the present invention is the determination apparatus described above, wherein the coordinate system of the first space, which is a real space in which an observer who observes the object image displayed on the screen is present, and the real space in which the object is present And a coordinate processing unit for determining an observer temporary position that is a predetermined temporary position of the observer in the second space based on a result of associating with a coordinate system of the second space.

  One aspect of the present invention provides the determination apparatus, a moving unit that moves the selected camera, an image acquisition unit that acquires an image including the object image from the moved camera, and the object image. An image processing apparatus comprising: a cutout unit that cuts out the object image from an included image; and an output unit that outputs the cut out object image to a display device.

  One aspect of the present invention is the image processing apparatus described above, further including a correction unit that corrects at least one of a direction and a size of the cut-out object image, and the output unit includes a direction of the object image. And the object image in which at least one of the sizes is corrected is output to the display device.

  In one embodiment of the present invention, a plurality of cameras whose positions and orientations are determined so as to capture an image of an object arranged in the real space are arranged in the real space and moved in the space of the image displayed on the screen of the display device. A determination method executed by a determination device that selects, from a plurality of cameras, the camera that outputs an image including an object image that is a still image or a moving image of the object observed as described above, The direction of the line connecting from the actual object position, which is the position, to the temporary object position, which is the temporary position of the object, and the position of the camera movement destination from the camera movement source position, which is the movement source position of the camera that captures the object This is a determination method including a step of selecting the camera from a plurality of the cameras based on at least one of the direction of a line connecting to the camera movement destination position and a predetermined magnification.

  One embodiment of the present invention is a determination program for causing a computer to function as the determination device.

  According to the present invention, the apparent size of the image of the object observed when the object is temporarily moved in the real space and the appearance of the image of the object virtually moved in the space of the image displayed on the display screen It becomes possible not to differ from the above size.

It is a figure which shows the example of a structure of the image processing system in 1st Embodiment. It is a figure which shows the example of the arrangement information of an object and a camera in 1st Embodiment. It is an overhead view which shows the example of arrangement | positioning of an object and a camera in 1st Embodiment. It is a figure which shows the example of the variable regarding the arrangement | positioning of the camera in 1st Embodiment. It is a figure which shows the 1st example of the display image in 1st Embodiment. It is a figure which shows the 1st example of the camera image in 1st Embodiment. It is a figure which shows the 2nd example of the camera image in 1st Embodiment. It is a figure which shows the 3rd example of the camera image in 1st Embodiment. It is a figure which shows the 2nd example of the display image in 1st Embodiment. 3 is a flowchart illustrating an example of an operation of the image processing apparatus in the first embodiment. It is a figure which shows the example of a structure of the image processing system in 2nd Embodiment. It is a figure which shows the example of the display image in 2nd Embodiment. 12 is a flowchart illustrating an example of an operation of the image processing apparatus in the second embodiment. It is a figure which shows the example of a structure of the image processing system in 3rd Embodiment. It is an overhead view which shows the example of arrangement | positioning of the observer in 1st space in 3rd Embodiment. It is a figure which shows the example of the display image in 3rd Embodiment. It is an overhead view which shows the example of arrangement | positioning of the object and camera in 2nd space in 3rd Embodiment. 10 is a flowchart illustrating an example of an operation of an image processing apparatus according to a third embodiment. It is a figure which shows the example of a structure of the image processing system in 4th Embodiment. 14 is a flowchart illustrating an example of an operation of an image processing apparatus according to a fourth embodiment.

Embodiments of the present invention will be described in detail with reference to the drawings.
(First embodiment)
FIG. 1 is a diagram illustrating an example of the configuration of the image processing system 1a. The image processing system 1a is a system that performs image processing on an image. The image may be either a still image or a moving image. The image processing system 1a uses the apparent size of the image of the object observed when the object is temporarily moved in the real space and the image of the object virtually moved in the space of the image displayed on the display screen. The apparent size of can be made different. The image processing system 1a reproduces an image of an object that is stationary in real space as faithfully as possible in the space of the image displayed on the display screen (virtual space). Can be moved. The image processing system 1a includes cameras 2-1 to 2-N (N is an integer of 2 or more), an image processing device 3a, and a display device 4. In the following, N is 3 as an example. Hereinafter, items common to the cameras 2-1 to 2-3 are denoted by “camera 2” while omitting a part of the reference numerals.

  The camera 2 is arranged at a predetermined position in a predetermined direction in the real space where the object is arranged. The camera 2 captures an image of an object placed in the real space where the camera 2 is placed through an optical system that is a lens. The optical system may be a wide-angle lens, a fish-eye lens, or an omnidirectional lens. Hereinafter, a still image or a moving image of an object to be imaged is referred to as an “object image”. The camera 2 sends an image (camera image) including a background still image or moving image (hereinafter referred to as “background image”) behind the object to be imaged for each camera and an object image to the image processing apparatus 3a. Output.

  The image processing device 3a is an information processing device such as a personal computer device, a server device, a smartphone terminal, or a tablet terminal. The image processing device 3a and the display device 4 may be separate or integrated. The image processing apparatus 3a selects the camera 2-n (1 ≦ n ≦ N) from the plurality of cameras 2. The image processing device 3a moves the selected camera 2 to a predetermined position. The image processing apparatus 3a may move the selected camera 2 while maintaining the optical axis direction of the selected camera 2. The image processing device 3a performs image processing on the image output by the selected camera 2. For example, the image processing device 3a may execute image processing at an image acquisition period (frame rate) of the camera. The image processing device 3 a outputs an image including the object image on which image processing has been performed to the display device 4. The image processing device 3a may execute image output processing at, for example, the display cycle (refresh rate) of the display device. That is, the image processing device 3a may perform moving image processing on a moving image including a plurality of frame images.

  The display device 4 is a display device such as a head mounted display. The screen of the display device 4 transmits light that reaches the screen of the display device 4 from an object outside the display device 4. For example, the screen of the display device 4 is a liquid crystal display having light transparency. Hereinafter, an optical image of an object by light transmitted through the screen of the display device 4 is referred to as an “object optical image”. The screen of the display device 4 displays the object image output by the image processing device 3a so as to hide the object optical image on the screen. That is, on the screen of the display device 4, the object image output by the image processing device 3a is pasted on the object optical image on the screen. Thereby, the screen of the display device 4 can display the object image output by the image processing device 3a instead of displaying the object optical image. Hereinafter, an image displayed on the screen of the display device 4 is referred to as a “display image”.

  Hereinafter, the actual position of the observer in the real space is referred to as “observer actual position”. The predetermined temporary position of the observer is referred to as “observer temporary position”. The observer temporary position is determined as, for example, the position of the observer's movement destination when it is assumed that the observer has moved. Hereinafter, the actual position of the object in the real space is referred to as “the actual object position”. Hereinafter, the predetermined temporary position of the object is referred to as “object temporary position”. The temporary object position is determined as, for example, the position to which the object is moved when it is assumed that the object has moved. Hereinafter, the position of the camera movement source in the real space is referred to as “camera movement source position”. Hereinafter, the position of the camera movement destination in the real space is referred to as “camera movement destination position”.

  FIG. 2 is a diagram illustrating an example of the arrangement information of the object and the camera 2. The image processing device 3a acquires predetermined arrangement information. For example, the arrangement information of the observer actual position and the observer temporary position is expressed using coordinates. In the following, the origin of coordinates is determined at the observer's actual position. In the following, the coordinates are expressed in meters as an example. The arrangement information of the actual object position and the temporary object position is expressed using coordinates, for example. The arrangement information of the camera 2 is represented using, for example, coordinates and an optical axis direction. The optical axis direction is expressed by, for example, an angle with reference to the x-axis of coordinates. In FIG. 2, as an example, B1 = about 108 degrees, B2 = 90 degrees, and B3 = about 108 degrees.

  FIG. 3 is an overhead view showing an example of the arrangement of the object 6 and the camera 2. The object 6 (object) is, for example, a box or a pillar. The shape of the object 6 is, for example, a cube. In FIG. 3, the letter “A” is drawn on the upper surface of the object 6 as an example. The object 6 is arranged at a predetermined actual object position in the room 100 existing in the real space. When the object 6 is disposed outdoors, the wall of the room 100 may be a virtual wall. The object 6 may be stationary in real space.

  The observer 5 observes the object optical image of the object 6 arranged in the room 100 from the observer's actual position through the screen of the display device 4 with the light transmitted through the screen of the display device 4. The observer 5 does not move from a predetermined observer actual position. The observer 5 observes the object image of the object 6 moved in the space (virtual space) of the image displayed on the display screen from the observer's actual position by the display image on the screen of the display device 4. In FIG. 3, the observer 5 moves the object 6 from the actual object position to the temporary object position 60 in the space of the image displayed on the display screen, and the distance from the temporary object position 60 to the actual observer position is a predetermined distance. The object image that looks like this is observed from the observer's actual position by the display image on the screen of the display device 4.

  For example, the observer 5 observes the object image of the object 6 captured by the camera 2 arranged near the observer temporary position 50 with the display image on the screen of the display device 4, so that the object 6 has moved. Reality (AR) and mixed reality (MR) can be obtained. The observer 5 can obtain augmented reality or mixed reality because the object image viewed by the observer 5 when the object 6 moves from the actual object position to the temporary object position 60 and the actual object position. This is because the object image seen by the observer 5 when the observer 5 observes the object 6 viewed from the observer temporary position 50 is equivalent.

  In other words, when the observer 5 views the object 6 moving from the actual object position to the temporary object position 60 from the actual observer position, the size of the object 6 is changed from the actual observer position to the temporary observer position 50. This is the same as how the object 6 is seen when the observer 5 sees the object 6 at the actual position of the object while the observer 5 moves. Therefore, the movement of the object 6 from the actual object position to the temporary object position 60 and the movement of the observer 5 from the actual observer position to the temporary observer position 50 are equivalent in terms of how the object 6 appears in the display image. It is.

  The camera 2 is arranged at a position based on the arrangement information of the camera 2 in a direction based on the arrangement information of the camera 2. The optical axis direction of the camera 2 is determined in advance so as to include the object 6 at the angle of view of the camera 2. The camera 2 is movable in a predetermined direction. The vector 21 is a vector connecting from the camera movement source position to the camera movement destination position. That is, the vector 21 represents a movement vector of the camera 2 that moves from the camera movement source position to the camera movement destination position. In FIG. 3, the vector 21-3 is parallel to the vector 51 and the vector 61.

  A straight line 22 is a line connecting the camera movement destination position to the actual object position. The optical axis 23 is the optical axis of the camera 2 at the camera movement destination position. The camera 2 may move to the camera movement destination position along the vector 21 while maintaining the direction of the optical axis 20 of the camera 2 at the camera movement source position. In FIG. 3, the optical axis 20 of the camera 2 at the camera movement source position and the optical axis 23 of the camera 2 at the camera movement destination position are parallel. In FIG. 3, the optical axis 20-2 of the camera 2-2 at the camera movement source position and the optical axis of the camera 2-2 at the camera movement destination position match.

  The vector 51 is a vector that connects from the actual observer position to the temporary observer position 50. That is, the vector 51 represents a movement vector when the observer 5 is assumed to move. A straight line 52 is a line connecting the observer actual position and the object actual position. A straight line 52 represents the line-of-sight direction of the observer 5 at the observer's actual position. A straight line 53 is a line connecting the actual observer position and the temporary object position 60. The straight line 53 and the straight line 55 are parallel. Since the straight line 53 and the straight line 55 are parallel, the angle formed by the straight line 55 and the straight line 22-1 is equal to the angle formed by the straight line 53 and the straight line 22-1.

  The visual field boundary 54 is a line representing the boundary of the visual field of the observer 5 at the actual position of the observer. A straight line 55 is a line connecting the temporary observer position 50 and the actual object position. A straight line 56 represents the line of sight of the observer 5 at the temporary observer position 50. The straight line 56 and the straight line 52 are parallel. A straight line 57 is a line parallel to the y-axis at the camera movement destination position. Therefore, the straight line 52, the straight line 56, and the straight line 57 are parallel to each other. The vector 61 is a vector that connects the actual object position to the temporary object position 60. The vector 61 represents a movement vector when it is assumed that the object 6 moves. The vector 61 is an inverse vector of the vector 51. The observer 5 can see an image of the object 6 moving along the vector 61 on the screen of the display device 4.

  In FIG. 3, the camera 2-2 is a camera arranged at a position closest to the observer temporary position 50. Therefore, the camera 2-2 outputs an image including an object image having a size closest to the size of the object optical image observed by the observer 5 to the image processing device 3a. The size of the object image is the area of the object image with respect to the area of the screen of the display device 4. The area of the object image with respect to the area of the screen may be expressed by, for example, the number of pixels of the object image with respect to the total number of pixels of the screen of the display device 4. The size of the object image may be a ratio of the area of the object image to the area of the camera image output by the camera 2.

  Hereinafter, the difference between the predetermined distance L0 from the actual object position to the temporary observer position 50 and the distance from the camera movement destination position of the camera 2-1 to the actual object position is denoted as “D1”. In FIG. 3, the predetermined distance L0 is about 8.1 m as an example. Hereinafter, the difference between the predetermined distance L0 from the actual object position to the temporary observer position 50 and the distance from the camera movement destination position of the camera 2-2 to the actual object position is denoted as “D2”. Hereinafter, the difference between the predetermined distance L0 from the actual object position to the observer temporary position 50 and the distance from the camera movement destination position of the camera 2-3 to the actual object position is denoted as “D3”. Hereinafter, the difference between the distance from the observer temporary position 50 to the actual object position and the distance from the camera destination position to the actual object position is referred to as a “distance difference”.

  Hereinafter, an angle formed by a line in the y-axis direction and the vector 21-1 is denoted as “A1”. In FIG. 3, the angle A1 formed by the vector 21-1 connecting the camera 2-1 from the camera movement destination position to the actual object position and the straight line 57-1 is 0 degrees. That is, the vector 21-1, the straight line 57-1 and the y-axis are parallel to each other.

  Hereinafter, an angle formed by a line in the y-axis direction and the vector 21-2 is denoted as “A2”. In FIG. 3, the angle A2 formed by the vector 21-2 connecting the camera 2-2 from the camera movement destination position to the actual object position and the straight line 57-2 is 0 degrees. That is, the vector 21-2, the straight line 57-2, and the y axis are parallel to each other.

  Hereinafter, the angle formed by the line in the y-axis direction and the vector 21-3 is denoted as “A3”. In FIG. 3, the angle A3 formed by the vector 21-3 connecting the camera 2-3 from the camera destination position to the actual object position and the straight line 57-3 is about 14 degrees. The straight line 57-3 and the y-axis are parallel to each other. In FIG. 3, the angle formed by the vector 61 connecting the actual object position to the temporary object position 60 and the straight line 52 is A3. That is, the angle formed by the vector 61 and the y axis is A3. Accordingly, the camera 2-3 outputs an image including an object image in which the apparent change rate of the object 6 is reproduced to the image processing device 3a even while the camera 2-3 is moving.

Next, returning to FIG. 1, an example of the configuration of the image processing apparatus 3a will be described.
The image processing apparatus 3 a includes an image acquisition unit 30, an operation unit 31, a variable determination unit 32, a storage unit 33, a selection unit 34, a movement unit 35, a cutout unit 36, and an output unit 37. . Some or all of the image acquisition unit 30, the operation unit 31, the variable determination unit 32, the selection unit 34, the movement unit 35, the cutout unit 36, and the output unit 37 are, for example, a processor such as a CPU (Central Processing Unit). However, it may be realized by executing a program stored in the storage unit, or may be realized by using hardware such as LSI (Large Scale Integration) or ASIC (Application Specific Integrated Circuit).

  The image acquisition unit 30 acquires an image including a background image and an object image from each of the cameras 2 selected by the selection unit 34. The image acquisition unit 30 records the image including the background image and the object image and the identification information of the camera 2 in the storage unit 33. The operation unit 31 is an operation device having operation keys and the like. For example, the operation unit 31 is a keyboard, a mouse, or a touch panel. The operation unit 31 acquires the arrangement information of the camera 2 in the real space by receiving an operation on the operation unit 31. The operation unit 31 receives the operation on the operation unit 31 to obtain the arrangement information of the actual object position and the temporary object position as the arrangement information of the object 6. The arrangement information of the actual object position and the temporary object position is represented using, for example, the coordinates and orientation of the object 6. The operation unit 31 receives the operation on the operation unit 31 to acquire the arrangement information of the observer actual position and the observer temporary position as the arrangement information of the observer 5. The arrangement information of the observer actual position and the observer temporary position is represented using, for example, the coordinates and the line-of-sight direction of the observer 5. The operation unit 31 includes information indicating the position of the area cut out from the camera image (hereinafter referred to as “cutout position information”) and information indicating the position where the area cut out from the camera image is pasted on the display image (hereinafter referred to as “pasting position information”). Acquisition position information ”). The cutout position information is information indicating the position of the area cut out from the camera image. The cutout position information may be represented by, for example, information on the upper left coordinates, width, and height in the object optical image. The upper left coordinate represents the upper left coordinate of the region to be cut out in the object optical image. The width represents the width of a region to be cut out in the object optical image. The height represents the height of a region to be cut out in the object optical image. The pasting position information is information indicating a position where an area cut out from the camera image is pasted on the display image. The pasting position information may be represented by, for example, information on the upper left coordinates, width, and height in the display image. Further, for example, the pasting position information may be information such as “center”, “right”, and “left”, which are areas defined in advance in association with the coordinates of the display image.

  The variable determination unit 32 acquires the arrangement information of the actual object position and the temporary object position, the arrangement information of the actual observer position and the temporary observer position, and the respective arrangement information of the camera 2 from the operation unit 31. The variable determination unit 32 determines each placement of the camera 2 based on the placement information of the actual object position and the temporary object position, the placement information of the actual observer position and the temporary observer position, and the placement information of the camera 2. Is determined for each camera 2. The variables relating to the arrangement of the camera 2 include, for example, an angle formed by a line in a predetermined direction and a line connecting the camera movement source position to the camera movement destination position (hereinafter referred to as “camera movement destination position angle”) and observation. The distance from the temporary person position 50 to the camera destination position (hereinafter referred to as “camera destination position distance”) and the distance difference. The predetermined direction is, for example, the y-axis direction. The variable determination unit 32 may determine a variable related to the placement of the camera 2 for each camera 2 based on the placement information of the camera 2 and the trigonometric function.

  FIG. 4 is a diagram illustrating an example of variables related to the arrangement of the camera 2. The variable determination unit 32 records variable information regarding the arrangement of the cameras 2 in the storage unit 33 for each camera 2. For example, the variable determination unit 32 records the camera movement destination position angle in the storage unit 33 for each camera 2. For example, the variable determination unit 32 records the camera movement destination position distance in the storage unit 33 for each camera 2. For example, the variable determination unit 32 records the distance difference for each camera 2 in the storage unit 33. In FIG. 4, the camera movement destination position distance L150 of the camera 2-1 is 3 m as an example. The camera movement destination position distance L250 of the camera 2-2 is 1 m as an example. The camera movement destination position distance L350 of the camera 2-3 is 4 m as an example.

  Returning to FIG. 1, the description of the configuration example of the image processing apparatus 3a will be continued. The storage unit 33 is a storage device having a nonvolatile recording medium (non-temporary recording medium) such as a magnetic hard disk device or a semiconductor storage device. The storage unit 33 stores an image including a background image and an object image, and variable information regarding each arrangement of the camera 2. The storage unit 33 may store arrangement information (coordinates, optical axis direction) of each camera 2. The storage unit 33 may store arrangement information of the actual object position and the temporary object position, and arrangement information of the actual observer position and the temporary observer position.

  The selection unit 34 functions as a determination device that determines the camera 2 to be selected. The selection unit 34 acquires variable information related to the placement of the camera 2 from the storage unit 33. The selection unit 34 selects the camera 2-n based on the camera movement destination position angle. The selection unit 34 selects a camera 2-n whose camera destination position angle is equal to or smaller than a predetermined angle. The predetermined angle is a predetermined angle, for example, 30 degrees. In the example of the arrangement of the camera 2 shown in FIG. 3, the selection unit 34 sets the direction of the vector 61 that connects the actual object position to the temporary object position 60 and the vector 21-3 that connects the camera movement source position to the camera movement destination position. The camera 2-3 is selected giving priority to the camera 2 whose direction is nearly parallel. In the example of the arrangement of the cameras 2 shown in FIG. 3, since the angle formed by the vector 61 and the y axis is A3, the selection unit 34 selects the camera 2-3 whose camera movement destination position angle is A3.

  When there are a plurality of cameras 2 in which the direction of the vector 61 and the direction of the vector 21 are nearly parallel, the selection unit 34 may select the camera 2-n based on the camera movement destination position distance or the distance difference. The distance difference is a predetermined distance difference, for example, a distance corresponding to 10% of the distance from the temporary observer position to the actual object position. The selection unit 34 may preferentially select the camera 2 whose camera movement destination position is close to the temporary observer position 50. For example, the selection unit 34 may preferentially select the camera 2 with a shorter camera movement destination position distance. For example, the selection unit 34 may preferentially select the camera 2 having a shorter distance difference.

  The moving unit 35 functions as a determination device that determines the position and orientation of the selected camera 2 by moving the selected camera 2. The moving unit 35 moves the camera 2 to a predetermined camera movement destination position. The moving unit 35 may arrange the camera 2 in a direction in which the optical axis direction at the camera movement source position and the optical axis direction at the camera movement destination position are the same direction. For example, the moving unit 35 moves and arranges the camera 2 by driving an actuator such as a motor. The moving unit 35 may detect the position information of the arranged camera 2 based on the detection result of the position sensor. The moving unit 35 may detect the orientation (posture) of the camera 2 based on the detection result of the acceleration sensor or the azimuth sensor.

  The cutout unit 36 acquires an image including a background image and an object image output from the selected camera 2 from the storage unit 33. The cutout unit 36 cuts out an object image from the image acquired from the storage unit 33 based on the cutout position information. The cutout unit 36 determines the display position of the object image on the display screen based on the pasting position information. For example, when the pasting position information represents “center”, the cutout unit 36 determines the display position of the object image on the display screen as the center of the screen of the display device 4.

  The cutout unit 36 may determine the display position of the object image on the display screen based on the selected direction of the camera 2 and the viewing direction of the observer 5. For example, based on the selected placement information of the camera 2, the placement information of the observer 5, and the placement information of the object 6, the clipping unit 36 moves the object 6 at the temporary object position from the observer 5 at the observer's actual position. The position and range of the object optical image on the display screen when observed are calculated. The cutout unit 36 may determine the position and range of the object optical image on the display screen as the display position of the object image on the display screen. The output unit 37 outputs the clipped object image and display position information to the display device 4.

  FIG. 5 is a diagram illustrating a first example of a display image. The screen of the display device 4 transmits light that reaches the screen of the display device 4 from the object 6. The screen of the display device 4 displays a display image 40 including an object optical image of the object 6. When the distance L65 from the actual object position to the actual observer position is, for example, 12 m, the observer 5 at the actual observer position displays the object optical image of the object 6 that is 12 m away from the actual observer position as the display image 40. Can be seen in the middle of.

  FIG. 6 is a diagram illustrating a first example of a camera image, which is a camera image 24 output from the camera 2-1 to the image processing device 3a. The camera image 24 includes the object image of the object 6 that is separated from the camera 2-1 at the camera movement destination position by the distance L61 in the right region from the center of the camera image 24. The distance L61 is about 8.2 m, for example. In the camera image 24, the surface of the object 6 facing the observer 5 is facing right in the camera image 24. In the camera image 24, the surface of the object 6 facing the observer 5 is distorted in a rhombus.

  FIG. 7 is a diagram illustrating a second example of a camera image, which is a camera image 25 output from the camera 2-2 to the image processing device 3a. The camera image 25 includes the object image of the object 6 that is separated from the camera 2-2 at the camera movement destination position by the distance L62 in the left region from the center of the camera image 25. The distance L62 is about 8.2 m, for example. In the camera image 25, the surface of the object 6 facing the observer 5 in the real space is facing left in the camera image 25. In the camera image 25, the surface of the object 6 that faces the observer 5 in real space is distorted in a rhombus.

  FIG. 8 is a diagram illustrating a third example of a camera image, which is a camera image 26 output from the camera 2-3 to the image processing device 3a. Since the vector 51 and the vector 21-3 shown in FIG. 3 are parallel, the size of the object image of the object 6 in the camera image 26 is from the observer actual position to the observer temporary position 50. When the object 6 is viewed while moving, the object 6 changes at the same change rate as the change rate of the apparent size. The camera image 26 includes an object image of the object 6 that is a distance L63 away from the camera 2-3 at the camera movement destination position in the left region from the center of the camera image 26. The distance L63 is about 9.4 m, for example. In the camera image 26, the surface of the object 6 facing the observer 5 in the real space is facing left in the camera image 26. In the camera image 26, the surface of the object 6 directly facing the observer 5 in the real space is greatly distorted into a rhombus.

  FIG. 9 is a diagram illustrating a second example of the display image. When the selection unit 34 selects the camera 2-3, the cutout unit 36 includes a region 110 including the object image of the object 6 from the camera image 26 including the background image and the object image output by the selected camera 2-3. Cut out. The cutout unit 36 determines the display position of the object image on the display screen according to, for example, the angle formed by the straight line 52 and the straight line 53 shown in FIG. For example, the cutout unit 36 determines the display position of the object image on the display screen to the left of the center of the screen of the display device 4. The output unit 37 outputs the area 110 including the clipped object image and the display position information “left of the center” to the display device 4. The screen of the display device 4 displays a region 110 including the cut object image so as to hide the object optical image at the center of the screen. Thereby, the screen of the display device 4 can display the area 110 including the clipped object image instead of displaying the object optical image of the object 6.

Next, an example of the operation of the image processing device 3a will be described.
FIG. 10 is a flowchart illustrating an example of the operation of the image processing apparatus 3a. The operation unit 31 acquires camera arrangement information (step S101). The variable determination unit 32 determines a variable related to the camera arrangement based on the camera arrangement information (step S102). The selection unit 34 selects the camera 2-n based on the determined variable (step S103). The moving unit 35 moves the selected camera 2 to the camera movement destination position (step S104). The image acquisition unit 30 acquires a still image from the selected camera 2 (step S105). The cutout unit 36 cuts out an object image from the still image acquired from the selected camera 2 (step S106). The cutout unit 36 determines the display position of the object image (step S107). The output unit 37 outputs the clipped object image and display position information to the display device 4 (step S108). The image processing device 3a may execute the operation shown in the flowchart of FIG. 10 at, for example, an image acquisition period (frame rate) of the camera.

  The difference between the case where the image acquisition unit 30 acquires a moving image from the selected camera 2 and the case where the image acquisition unit 30 acquires a still image from the selected camera 2 will be described. When the image acquisition unit 30 acquires a moving image from the selected camera 2, the image acquisition unit 30 acquires a moving image from the selected camera 2 in step S <b> 105. In step S <b> 106, the cutout unit 36 acquires each frame image from the moving image acquired from the selected camera 2. In step S106, the cutout unit 36 cuts out an object image from each frame image. In step S <b> 107, the cutout unit 36 determines the display position of the object image so that it becomes a common display position in each frame image. In step S <b> 108, the output unit 37 outputs the number of object images corresponding to the number of frame images and display position information common to the frame images to the display device 4.

  As described above, the determination device included in the image processing device 3a of the first embodiment is a determination device that selects the camera 2-n from the plurality of cameras 2 that capture an image of the object from a plurality of positions and directions. 34 and a moving unit 35. Based on the direction of the vector 61 that connects the actual object position to the temporary object position 60 and the direction of the vector 21 that connects the camera movement source position to the camera movement destination position, the selection unit 34 sets the camera 2 to the camera 2- Select n.

  As a result, the image processing apparatus 3a according to the first embodiment uses the apparent size of the image of the object observed when the object is temporarily moved in the real space and the space of the image displayed on the display screen. Therefore, it is possible to prevent the apparent size of the image of the moved object from being different. The image processing apparatus 3a according to the first embodiment faithfully displays a specific image of the object 6 stationary in the real space in the space of the image displayed on the display screen as much as possible. Can be reproduced and moved. When the temporary observer position 50 and the camera movement destination position are sufficiently close, the image processing apparatus 3a according to the first embodiment does not correct the size of the object image and displays the camera image as it is. An object image with no sense of incongruity in the upper size can be displayed on the screen of the display device 4. The image processing apparatus 3a according to the first embodiment includes the apparent size or orientation of the image of the object 6 that is observed when the object 6 is temporarily moved in the real space, and the space of the image displayed on the display screen. Thus, it is possible to prevent the apparent size or orientation of the object image of the object 6 that has been virtually moved.

(Second Embodiment)
The second embodiment is different from the first embodiment in that the image processing apparatus corrects at least one of the orientation and size of the object image. In the second embodiment, only differences from the first embodiment will be described.

  FIG. 11 is a diagram illustrating an example of the configuration of the image processing system 1b. The image processing system 1b includes cameras 2-1 to 2-N, an image processing device 3b, and a display device 4. The image processing device 3b is an information processing device such as a personal computer device, a server device, a smartphone terminal, or a tablet terminal. The image processing device 3b and the display device 4 may be separate or integrated. The image processing device 3b selects the camera 2-n from the plurality of cameras 2. The image processing device 3b moves the selected camera 2. The image processing device 3b performs image processing on the image output by the selected camera 2. The image processing device 3b includes an image acquisition unit 30, an operation unit 31, a variable determination unit 32, a storage unit 33, a selection unit 34, a movement unit 35, a cutout unit 36, an output unit 37, and a correction unit. Part 38.

  The correction unit 38 corrects at least one of the orientation and size of the cut out object image. When correcting at least one of the orientation and size of the clipped object image, the correction unit 38 performs, for example, affine transformation processing using an affine transformation matrix on the object image and the region clipped from the camera image. The affine transformation matrix may include information indicating a region to be subjected to processing such as rotation, enlargement, or reduction. The correction unit 38 corrects at least one of the orientation and size of the cut out object image.

  The correction unit 38 is configured so that the apparent direction of the object 6 when the observer 5 views the object 6 at the actual object position from the temporary position of the observer matches the apparent direction of the object image on the display screen. Rotate the object image. For example, the correction unit 38 rotates the object image based on the camera movement destination position angle. That is, the correction unit 38 corrects the distortion of the object image.

  The correction unit 38 is configured so that the apparent size of the object 6 when the observer 5 sees the object 6 at the actual object position from the temporary position of the object matches the apparent size of the object image on the display screen. , Enlarge or reduce the object image. For example, the correction unit 38 enlarges or reduces the object image based on the ratio of the distance from the temporary object position 60 to the actual observer position and the distance from the actual object position to the actual observer position. The output unit 37 outputs an object image in which at least one of the direction and size of the object image is corrected to the display device 4.

  FIG. 12 is a diagram illustrating an example of a display image. The screen of the display device 4 displays an area including an object image in which at least one of orientation and size is corrected so as to hide the object optical image on the left side of the center of the screen based on the display position information. In FIG. 12, the orientation of the surface of the object 6 seen on the left side when viewed from the observer 5 at the observer temporary position 50 in the real space and the orientation of the corresponding surface in the display image 40 are the same. In FIG. 12, the apparent size of the object 6 when the observer 5 views the object 6 at the actual object position from the observer temporary position and the apparent size of the object image in the display image 40 are equal to each other. I'm doing it.

  Thereby, the screen of the display device 4 can display an object image in which at least one of the orientation and the size is corrected instead of displaying the object optical image of the object 6. When the observer 5 looks at the display image 40 shown in FIG. 12, the augmented reality (AR) or the mixed reality (MR) that the stationary object 6 has moved to the temporary object position 60 in the real space. Can be obtained.

Next, an example of the operation of the image processing device 3b will be described.
FIG. 13 is a flowchart illustrating an example of the operation of the image processing apparatus 3b. The operation unit 31 acquires camera arrangement information (step S201). The variable determination unit 32 determines a variable related to the camera arrangement based on the camera arrangement information (step S202). The selection unit 34 selects the camera 2-n based on the determined variable (step S203). The moving unit 35 moves the selected camera 2 to the camera movement destination position (step S204). The image acquisition unit 30 acquires an image from the selected camera 2 (step S205). The cutout unit 36 cuts out an object image from the image acquired from the selected camera 2 (step S206). The correcting unit 38 corrects at least one of the direction and size of the object image (step S207). The cutout unit 36 determines the display position of the object image (step S208). The output unit 37 outputs the clipped object image and display position information to the display device 4 (step S209). The image processing device 3b may perform the operation shown in the flowchart of FIG. 13 at, for example, an image acquisition cycle (frame rate) of the camera.

  The difference between the case where the image acquisition unit 30 acquires a moving image from the selected camera 2 and the case where the image acquisition unit 30 acquires a still image from the selected camera 2 will be described. When the image acquisition unit 30 acquires a moving image from the selected camera 2, the image acquisition unit 30 acquires a moving image from the selected camera 2 in step S <b> 205. In step S <b> 206, the cutout unit 36 acquires each frame image from the moving image acquired from the selected camera 2. In step S206, the cutout unit 36 cuts out an object image from each frame image. In step S207, the correction unit 38 corrects at least one of the orientation and size of the object images corresponding to the number of frame images. In step S208, the cutout unit 36 determines the display position of the object image so that the display position is common to the frame images. In step S <b> 209, the output unit 37 outputs the number of frame images of object images and display position information common to the frame images to the display device 4.

  As described above, the image processing device 3b according to the second embodiment includes the determination device according to the first embodiment, the cutout unit 36, the correction unit 38, and the output unit 37. The cutout unit 36 cuts out an object image from an image including the object image. The correction unit 38 corrects at least one of the orientation and size of the cut out object image. The output unit 37 outputs an object image in which at least one of the direction and size of the object image is corrected to the display device 4. As a result, the image processing apparatus 3b according to the second embodiment can perform virtual processing based on the apparent size of the object image observed when the object is temporarily moved in the real space and the space of the image displayed on the display screen. Therefore, it is possible to prevent the apparent size of the image of the moved object from being different. For example, the image processing apparatus 3b according to the second embodiment displays an image of a specific appropriate size of the object 6 that is stationary in the real space in the image space displayed on the display screen. It is possible to reproduce and move the change rate as faithfully as possible.

(Third embodiment)
The third embodiment is different from the first embodiment in that the observer 5 and the object 6 exist in different real spaces. In the third embodiment, only differences from the first embodiment will be described.

  Hereinafter, the real space in which the observer 5 exists is referred to as “first space”. Hereinafter, the real space in which the object 6 exists is referred to as “second space”. The first space and the second space may be different real spaces that are adjacent to each other, or may be different real spaces that are separated from each other.

  The first space is, for example, a small model (miniature) of a baseball field in real space. A small model of a baseball field is placed on a table in real space, for example. When the first space is a small model of a baseball field, for example, the second space is a full-scale baseball field in real space. A baseball game may be held at a full-scale baseball field. When the first space is a small model of a baseball field and the second space is a full-scale baseball field, the virtual space is, for example, the first space and the second space such that the coordinate systems of the first space and the second space coincide. This is a space on the screen in which at least one distance between the space and the second space is scale-corrected, and an object image of an object existing in the second space is superimposed and displayed in the first space. The object image of the object 6 existing in the full-size baseball field in the second space is corrected on the basis of the distance in the first space and the distance in the second space, so that the object 6 is displayed in the small model of the baseball field. It is displayed on the screen of the display device 4 as an image observed so as to move virtually.

  The first space is, for example, a full-scale first baseball field in real space. In the case where the first space is a full-size first baseball field, the second space is, for example, a full-size second baseball field in real space. The image acquisition unit 30 may acquire a moving image of a baseball game that has been played in the past at the full-scale second baseball field. When the first space is a full-size first baseball field and the second space is a full-size baseball field, for example, the virtual space is such that the coordinate systems of the first space and the second space match each other. In addition, the first space and the second space are on the screen in which the object image of the object existing in the second space is superimposed and displayed on the first space while the original space is the full size. The object image of the object 6 existing in the full-size baseball field in the second space is displayed on the screen of the display device 4 as an image observed so that the object 6 moves virtually in the full-size baseball field in the first space. Is displayed.

  FIG. 14 is a diagram illustrating an example of the configuration of the image processing system 1c. The image processing system 1c includes cameras 2-1 to 2-N, an image processing device 3c, and a display device 4. The image processing device 3c is an information processing device such as a personal computer device, a server device, a smartphone terminal, or a tablet terminal. The image processing device 3c and the display device 4 may be separate or integrated. The image processing device 3c performs image processing on the image output from the camera 2-n. The image processing apparatus 3 c includes an image acquisition unit 30, an operation unit 31, a variable determination unit 32, a storage unit 33, a selection unit 34, a cutout unit 36, an output unit 37, and a coordinate processing unit 41. Prepare.

  The image processing device 3c may further include a camera capable of capturing the first space. The storage unit 33 may store information representing the arrangement of a predetermined object in the first space or the second space. For example, the storage unit 33 may store a design drawing of a baseball field in the first space or the second space.

  FIG. 15 is an overhead view showing an example of the arrangement of the observers 5 in the first space 101. Hereinafter, the types of objects that exist in common in the first space and the second space are referred to as “common objects”. For example, when the first space is a small baseball field model and the second space is a full-scale baseball field, the common object is a base such as a home base.

  In FIG. 15, the first space 101 is a small model of a baseball field. The first common object 7-1 is a home base small model. The second common object 8-1 is a 1 塁 -based small model. The third common object 9-1 is a small model based on 2 塁. The fourth common object 10-1 is a 3 塁 -based small model. Each common object from the first common object 7-1 to the fourth common object 10-1 may not move in the first space 101. The common object may be a part of the object. The number of common objects may not be four. For example, when the optical axis direction of the camera 2 is determined, the number of common objects may be three or less. The observer 5 observes the first space 101 as an object optical image with the light transmitted through the screen of the display device 4.

  FIG. 16 is a diagram illustrating an example of the display image 42. The display image 42 is an image displayed on the screen of the display device 4 when the observer 5 is observing the first space 101. In FIG. 16, the 1st space is a small model of a baseball field like the 1st space 101 shown in FIG. 15 as an example. The display device 4 displays the fine adjustment key image 15, the enter key image 16, and the menu key image 17 on the screen as operation key images of the operation unit 31.

  The observer 5 determines the temporary object position 60-1 in the first space 101 by operating the operation unit 31. The temporary object position 60-1 in the first space 101 may be determined by a person other than the observer 5. For example, the temporary object position 60-1 in the first space 101 may be determined by an external device.

  The observer 5 designates the position of the origin of the first space (hereinafter referred to as “first origin”) on the screen of the display device 4 by operating the operation unit 31. For example, the observer 5 designates the position of a common object having a common point in the arrangement in each space between the object existing in the first space 101 and the object existing in the second space 102. The arrangement of the common object is, for example, a relative arrangement of a home base, a 1-base, a 2-base, and a 3-base in a baseball field. In the display image 42, the observer 5 touches the image of the common object with his / her finger in the order of home base, first base, second base, and third base, and designates the position of each common object. The first origin of the first space is, for example, the position of the first common object 7-1 touched first. The observer 5 may specify the position of the temporary object position 60-1 by touching the position of the temporary object position 60-1 with a finger in the display image 42.

  The display device 4 displays the arrow images 11 to 14 on the display image 42 when the observer 5 presses the menu key image 17. The observer 5 moves the arrow image 11 in the display image 42 by pressing the fine adjustment key image 15, and finely adjusts the exact designated position of the first common object 7-1 in the display image 42 (calibration). ) The observer 5 moves the arrow image 12 in the display image 42 by pressing the fine adjustment key image 15 to finely adjust the exact designated position of the second common object 8-1 in the display image 42. The observer 5 moves the arrow image 13 in the display image 42 by pressing the fine adjustment key image 15 to finely adjust the exact designated position of the third common object 9-1 in the display image 42. The observer 5 moves the arrow image 14-1 to the position of the arrow image 14-2 in the display image 42 by depressing the fine adjustment key image 15, and the fourth common object 10-1 in the display image 42 is moved. Fine-tune the exact specified position. The observer 5 presses the determination key image 16 when determining the exact designated position of each common object in the display image 42.

  When the operation unit 31 receives an operation of pressing the enter key image 16, the operation unit 31 outputs position information (designated position information) of each common object in the display image 42 to the coordinate processing unit 41. When the operation unit 31 receives an operation of pressing the enter key image 16, the operation unit 31 outputs position information of the temporary object position 60-1 in the display image 42 to the coordinate processing unit 41.

  FIG. 17 is an overhead view showing an example of the arrangement of the object 6 and the camera 2 in the second space 102. In FIG. 17, the second space 102 is a full-scale baseball field. The first common object 7-2 is a full-size home base. The second common object 8-2 is a full-scale 1-inch base. The third common object 9-2 is a full-size 2cm base. The fourth common object 10-2 is a full-scale 3cm base. Each common object from the first common object 7-2 to the fourth common object 10-2 does not move in the second space 102. The common object may be a part of the object. Each camera 2 images the second space 102.

  Returning to FIG. 14, the description of the configuration example of the image processing apparatus 3 c is continued. The coordinate processing unit 41 acquires position information of each common object in the display image 42 from the operation unit 31. For example, the coordinate processing unit 41 acquires the position information of the first common object 7-1 in the display image 42 as the position information of the first origin. The coordinate processing unit 41 acquires position information of the temporary object position 60-1 in the display image 42 from the operation unit 31.

  The coordinate processing unit 41 detects the position of the first common object 7-1 (first origin) in the first space 101 based on the position information of each common object in the display image 42. The coordinate processing unit 41 detects the distance between the common objects in the first space 101 based on the position information of each common object in the display image 42. For example, the coordinate processing unit 41 detects the distance from the first common object 7-1 to the second common object 8-1 in the first space 101. For example, the coordinate processing unit 41 may further detect the distance from the first common object 7-1 to the third common object 9-1 in the first space 101. For example, the coordinate processing unit 41 may further detect the distance from the first common object 7-1 to the fourth common object 10-1 in the first space 101.

  When the observer 5 presses the enter key image 16, the coordinate processing unit 41 selects the first origin of the first space 101, the direction of the first space 101, the distance between the common objects, and the actual position of the observer And the depression angle of the position of each common object with reference to. When the first space 101 is a small model of a baseball field, the coordinate processing unit 41 determines the position of the first origin in the first space 101 based on, for example, the result of the space recognition process on the object optical image of the first space 101. Then, the orientation of the first space 101, the distance between the common objects in the first space 101, and the depression angle of the position of each common object with respect to the actual position of the observer in the first space 101 are detected.

  For example, the coordinate processing unit 41 may execute the space recognition process using the learning result of the neural network. The coordinate processing unit 41 may learn spatial recognition using, for example, a home-based teacher image.

  When the first space 101 is a full-scale baseball field, the design drawing of the baseball field stored in the storage unit 33 shows the result of measuring the positions of the position tracking devices arranged in advance in the first space 101. It may include information to represent. When the first space 101 is a full-scale baseball field, the coordinate processing unit 41, for example, based on the result of measuring the position of the position tracking device, the position of the first origin in the first space 101 and the first space The direction of 101, the distance between the common objects in the first space 101, and the depression angle of the position of each common object with respect to the actual position of the observer in the first space 101 are detected.

  The coordinate processing unit 41 detects the distance from the first common object 7-1 to the temporary object position 60-1 in the first space 101. For example, the coordinate processing unit 41 detects the distance from the first common object 7-1 to the temporary object position 60-1 in the first space 101. For example, the coordinate processing unit 41 may further detect the distance from the second common object 8-1 to the temporary object position 60-1 in the first space 101. For example, the coordinate processing unit 41 may further detect the distance from the third common object 9-1 to the temporary object position 60-1 in the first space 101. For example, the coordinate processing unit 41 may further detect the distance from the fourth common object 10-1 to the temporary object position 60-1 in the first space 101.

  The coordinate processing unit 41 acquires the position information of each common object in the second space 102 from the storage unit 33. For example, the coordinate processing unit 41 acquires position information of each common object in the second space 102 as a design drawing of the second space 102 that is a full-scale baseball field or the like.

  The coordinate processing unit 41 detects the position of the origin of the coordinate system of the second space 102 (hereinafter referred to as “second origin”) based on the position information of each common object in the second space 102. The coordinate processing unit 41 detects the distance between the common objects in the second space 102 based on the position information of each common object in the second space 102. For example, the coordinate processing unit 41 detects the distance from the first common object 7-2 to the second common object 8-2 in the second space 102. For example, the coordinate processing unit 41 may further detect the distance from the first common object 7-2 to the third common object 9-2 in the second space 102. For example, the coordinate processing unit 41 may further detect the distance from the first common object 7-2 to the fourth common object 10-2 in the second space 102.

  When the observer 5 presses the enter key image 16, the coordinate processing unit 41 uses the second origin of the second space 102, the direction of the second space 102, the distance between the common objects, and a predetermined position as a reference. And the depression angle of the position of each common object. When the second space 102 is a full-scale baseball field, the design drawing of the baseball field stored in the storage unit 33 shows the result of measuring the positions of the position tracking devices arranged in advance in various places in the second space 102. It may include information to represent. When the second space 102 is a full-scale baseball field, the coordinate processing unit 41, for example, based on the result of measuring the position of the position tracking device, the position of the second origin in the second space 102, and the second space The direction of 102, the distance between the common objects in the second space 102, and the depression angle of the position of each common object with respect to a predetermined position in the second space 102 are detected.

  The coordinate processing unit 41 matches the second origin with the first origin on the screen. The coordinate processing unit 41 matches the direction of the coordinate system in the second space with the direction of the coordinate system in the first space. Thus, when the scale of the coordinate system of the first space 101 and the scale of the coordinate system of the second space 102 are the same, the coordinate processing unit 41 has the coordinate system of the first space 101 and the coordinate system of the second space 102. Can be converted to each other.

  Furthermore, when the scale of the coordinate system of the first space 101 and the scale of the coordinate system of the second space 102 are different, the coordinate processing unit 41 determines the distance in the coordinate system of the second space 102 and the coordinate system of the first space 101. A ratio to the distance (distance ratio) is detected for each set of common objects. That is, the coordinate processing unit 41 detects the scale of the coordinate system of the first space 101 with respect to the coordinate system of the second space 102 for each set of common objects. For example, the coordinate processing unit 41 has the distance from the first common object 7-1 to the second common object 8-1 and the first common object 7-2 to the second common object 8 in the horizontal x-axis direction of the real space. The ratio to the distance to -2. For example, for the horizontal y-axis direction of the real space, the coordinate processing unit 41 and the distance from the second common object 8-1 to the third common object 9-1 and the second common object 8-2 to the third common object 9 The ratio to the distance to -2 may be detected. The coordinate processing unit 41 determines the distance from the xy plane of the first space 101 to a predetermined seat in the first space 101 and the xy plane of the second space 102 with respect to the z-axis direction (vertical upward direction) of each real space. A ratio of the distance to a predetermined seat in the two spaces 102 may be detected. The coordinate processing unit 41 may detect the scale of the coordinate system of the second space 102 with respect to the coordinate system of the first space 101. The distance ratio (scale) may be common to the three axes xyz of each real space, or may be different for each xyz axis of each real space. As a result, even when the scale of the coordinate system of the first space 101 is different from the scale of the coordinate system of the second space 102, the coordinate processing unit 41 has the coordinate system of the first space 101 and that of the second space 102. Can be converted to each other.

  The coordinate processing unit 41 determines the temporary observer position 50 in the second space 102 based on the result of associating the coordinate system of the first space 101 with the coordinate system of the second space 102. For example, the coordinate processing unit 41 determines the temporary object position 60-2 in the second space 102 based on the distance and the distance ratio from the first common object 7-1 to the temporary object position 60-1 in the first space 101. decide. The coordinate processing unit 41 determines the temporary observer position 50 in the second space 102 based on the temporary object position 60-2 in the second space.

  The coordinate processing unit 41 outputs the temporary observer position 50 in the second space 102 to the variable determination unit 32. The variable determination unit 32 updates the variables related to the arrangement of the camera 2 illustrated in FIG. 4 based on the temporary observer position 50 in the second space.

  The observer 5 observes the object image of the object 6 captured by the camera 2-2 that has been moved near the observer temporary position 50 in the second space 102 by the display image on the screen of the display device 4, and thus the object 6 Augmented reality and mixed reality can be obtained. The observer 5 can obtain augmented reality or mixed reality because the image viewed by the observer 5 when the object 6 moves from the actual object position to the temporary object position 60-2 and the actual object position. This is because the image seen by the observer 5 when the observer 5 observes the object 6 being viewed from the observer temporary position 50 is equivalent.

  FIG. 18 is a flowchart illustrating an example of the operation of the image processing device 3c. The coordinate processing unit 41 acquires position information of each common object in the display image 42 (step S301). The coordinate processing unit 41 acquires position information of the temporary object position in the display image 42 (step S302). The coordinate processing unit 41 detects the distance between the common objects in the first space 101 (step S303). The coordinate processing unit 41 detects the distance from each common object to the temporary object position 60-1 in the first space 101 (step S304).

  The coordinate processing unit 41 acquires position information of each common object in the second space 102 (step S305). The coordinate processing unit 41 detects the distance between the common objects in the second space 102 (step S306). The coordinate processing unit 41 determines the ratio (distance ratio) between the distance between the common objects in the first space 101 and the distance between the common objects in the second space 102 for each set of common objects (step S307). The coordinate processing unit 41 determines the temporary object position 60 in the second space 102 based on the distance and the distance ratio from the first common object 7-1 (first origin) to the temporary object position 60-1 in the first space 101. -2 is determined (step S308). The coordinate processing unit 41 determines the temporary observer position 50 in the second space 102 based on the temporary object position 60-2 in the second space 102 (step S309).

  Note that the first space may not be a small model of a baseball field, for example, a small model of a soccer field. The first space may not be a full size baseball field, for example, a full size soccer field. Similarly, the second space may be a full-scale soccer field. When the first space is a small model of a soccer field, the display image 42 shown in FIG. 16 may be an overhead view when the small model of a soccer field with goal posts arranged on the left and right is looked down vertically. . When the first space is a full size soccer field, the display image 42 shown in FIG. 16 may be an overhead view when the full size soccer field where the goal posts are arranged on the left and right is looked down vertically. .

  When the first space is a full size soccer field, the first common object 7-1 may be the lower left corner portion of the goal line in the overhead view when the full size soccer field is looked down vertically. The second common object 8-1 may be a lower right corner portion of the goal line. The third common object 9-1 may be the upper left corner of the goal line. The fourth common object 10-1 may be a part of the frame having the highest position on the goal post.

  The coordinate processing unit 41 has a distance from the first common object 7-1 to the second common object 8-1 and a first common object 7-2 to a second common object 8-2 in the horizontal x-axis direction of the real space. The ratio to the distance to may be determined. The coordinate processing unit 41 has a distance from the second common object 8-1 to the third common object 9-1 and the second common object 8-2 to the third common object 9-2 in the horizontal y-axis direction of the real space. The ratio to the distance to may be detected. The coordinate processing unit 41 is configured to calculate the distance from the xy plane to the fourth common object 10-1 and the distance from the xy plane to the fourth common object 10-2 with respect to the z-axis direction (vertical upward direction) of each real space. The ratio may be detected.

  As described above, the image processing device 3 c according to the third embodiment includes the coordinate processing unit 41. The coordinate processing unit 41 determines the temporary observer position 50 in the second space 102 based on the result of associating the coordinate system of the first space 101 with the coordinate system of the second space 102.

  As a result, the image processing apparatus 3c according to the third embodiment allows the image of the object to be observed when the object 5 is temporarily moved in the real space even when the observer 5 and the object 6 exist in different real spaces. It is possible to prevent the apparent size or orientation from differing from the apparent size or orientation of the object image of the object virtually moved in the space of the image displayed on the display screen.

  In the first embodiment to the second embodiment, the image processing apparatuses 3a to 3b may further include a coordinate processing unit 41.

(Fourth embodiment)
The fourth embodiment is different from the first embodiment in that the image processing apparatus selects or moves the camera according to a magnification that determines the size of the object image to be displayed on the display screen. In the fourth embodiment, only differences from the first embodiment will be described.

  FIG. 19 is a diagram illustrating an example of the configuration of the image processing system 1d. The image processing system 1d includes cameras 2-1 to 2-N, an image processing device 3d, and a display device 4. The image processing device 3d is an information processing device such as a personal computer device, a server device, a smartphone terminal, or a tablet terminal. The image processing device 3d and the display device 4 may be separate or integrated. The image processing device 3d performs image processing on the image output from the camera 2-n. The image processing device 3d includes an image acquisition unit 30, an operation unit 31, a variable determination unit 32, a storage unit 33, a selection unit 34d, a cutout unit 36, and an output unit 37. The image processing apparatus 3d may further include an arrangement unit 39.

The relationship between the length W ₀ in the predetermined direction of the object 6 and the length W _i in the predetermined direction of the object image on the screen of the display device 4 is expressed as Expression (1). The predetermined direction is, for example, the x-axis direction, the y-axis direction, or the z-axis direction.

W _i = f / d × W ₀ (1)

Here, f represents the focal length of the lens provided in the camera 2. The focal length f is constant. d represents the distance from the camera 2 to the object 6 (subject). W ₀ represents the length of the object 6 in a predetermined direction. _Wi is the length of the object image on the screen of the display device 4 in a predetermined direction. The area of the object image may be expressed as a ratio to the area of the screen or may be expressed as the number of pixels. Note that “W ₀ ” in Expression (1) may be further multiplied by a coefficient other than (f / d).

The selection unit 34d detects the distance d from the camera 2 selected from the cameras 2-1 to 2-N to the object 6. Selecting unit 34d is a command signal indicating that the predetermined direction of the object image in the screen length W _i to X times, acquires from the operation unit 31. Selecting unit 34d, when acquiring a command signal indicating that the length W _i to X times the distance to the object 6 to select the camera 2 is (d / X). X represents a magnification.

For example, when the distance from the observer actual position to the object temporary position 60 is (1 / X) times the distance from the observer actual position to the object actual position, the selection unit 34d selects the object image in the predetermined direction on the screen. the length W _i may be determined as to X times. Selecting unit 34d is in a predetermined direction of the object image in the screen length W _i when determining to the X times, the distance to the object 6 to select the camera 2 is (d / X). The selection unit 34d may select the camera 2 whose distance to the object 6 is close to (d / X) based on the priority information. For example, when the priority information indicates that the direction of the object image in the display image is given priority over the size of the object image, the selection unit 34d selects the camera 2 whose distance to the object 6 is close to (d / X). May be.

  When the image processing device 3d includes the arrangement unit 39, the selection unit 34d may arrange the camera 2 at a position where the distance to the object 6 is (d / X). The selection unit 34d arranges the camera 2 in a direction that includes the object 6 in the angle of view.

FIG. 22 is a flowchart illustrating an example of the operation of the image processing apparatus 3d. The selection unit 34d detects the distance d from the camera 2 selected from the cameras 2-1 to 2-N to the object 6 (step S401). Selecting unit 34d is a command signal indicating that the predetermined direction of the object image in the screen length W _i to X times, acquires from the operation unit 31 (step S402). Selecting unit 34d, when acquiring a command signal indicating that the length W _i to X times the distance to the object 6 to select the camera 2 is (d / X) (step S403).

  As described above, the image processing device 3d according to the fourth embodiment includes the selection unit 34d. When the selection unit 34d selects the camera 2 based on the magnification, the selection unit 34d has a distance (d / X) where the distance from the camera 2 to the object 6 is determined according to the reciprocal of the magnification (X). 2 is selected.

  As a result, the image processing apparatus 3d according to the fourth embodiment can be used in real space even when the image processing apparatus selects or moves the camera according to the magnification that determines the size of the object image to be displayed on the display screen. The apparent size or orientation of the object image observed when the object is temporarily moved, and the apparent size of the object image of the object virtually moved in the space of the image displayed on the display screen It is possible to prevent the height or direction from being different.

  In the first embodiment, the selection unit 34 of the image processing apparatus 3a may further execute the same process as the selection unit 34d. In the second embodiment, the selection unit 34 of the image processing device 3b may further execute the same process as the selection unit 34d. In the third embodiment, the selection unit 34 of the image processing device 3c may further execute the same process as the selection unit 34d.

  You may make it implement | achieve at least one part of the image processing system in the embodiment mentioned above, the determination apparatus, and the image processing apparatus with a computer. In that case, a program for realizing this function may be recorded on a computer-readable recording medium, and the program recorded on this recording medium may be read into a computer system and executed. Here, the “computer system” includes an OS and hardware such as peripheral devices. The “computer-readable recording medium” refers to a storage device such as a flexible medium, a magneto-optical disk, a portable medium such as a ROM and a CD-ROM, and a hard disk incorporated in a computer system. Furthermore, the “computer-readable recording medium” dynamically holds a program for a short time like a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line. In this case, a volatile memory inside a computer system serving as a server or a client in that case may be included and a program held for a certain period of time. Further, the program may be a program for realizing a part of the above-described functions, and may be a program capable of realizing the functions described above in combination with a program already recorded in a computer system. You may implement | achieve using programmable logic devices, such as FPGA (Field Programmable Gate Array).

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes designs and the like that do not depart from the gist of the present invention.

  The present invention is applicable to a determination apparatus that determines the selection, position, and orientation of a camera that outputs an image that gives an observer an augmented reality (AR) or mixed reality (MR).

  DESCRIPTION OF SYMBOLS 1a-1d ... Image processing system, 2 ... Camera, 3a-3d ... Image processing apparatus, 4 ... Display apparatus, 5 ... Observer, 6 ... Object, 7 ... 1st common object, 8 ... 2nd common object, 9 ... 3rd common object, 10 ... 4th common object, 11 ... arrow image, 12 ... arrow image, 13 ... arrow image, 14 ... arrow image, 15 ... fine adjustment key image, 16 ... decision key image, 17 ... menu key image 20 ... optical axis, 21 ... vector, 22 ... straight line, 23 ... optical axis, 24 ... camera image, 25 ... camera image, 26 ... camera image, 30 ... image acquisition unit, 31 ... operation unit, 32 ... variable determination unit 33 ... Storage unit, 34, 34d ... Selection unit, 35 ... Moving unit, 36 ... Extraction unit, 37 ... Output unit, 38 ... Correction unit, 40 ... Display image, 41 ... Coordinate processing unit, 42 ... Display image, 50 ... Temporary observer position, 51 ... Vector, 52 ... Straight line, 53 ... Straight line, 4 ... field boundaries, 55 ... straight line, 56 ... straight line, 57 ... straight line, 60 ... object temporary position, 61 ... vector, 100 ... room 101 ... first space, 102: second space, 110 ... area

Claims (10)

A plurality of cameras whose positions and orientations are determined so as to image an object placed in the real space are arranged in the real space and observed as if they were moved in the space of the image displayed on the screen of the display device A determination device that selects the camera that outputs an image including an object image that is a still image or a moving image of an object from a plurality of the cameras,
The direction of a line connecting from the actual object position, which is the actual position of the object, to the temporary object position, which is the temporary position of the object, and the camera movement source position, which is the movement source position of the camera that captures the object, And a selection unit that selects the camera from the plurality of cameras based on at least one of a direction of a line connecting to the camera destination position, which is a destination position of the camera, and a predetermined magnification .   The selection unit gives priority to the camera in which the direction of the line connecting the actual object position to the temporary object position and the direction of the line connecting the camera movement source position to the camera movement destination position are nearly parallel. The determination device according to claim 1, wherein the determination device selects a camera.   The selection unit preferentially selects the camera having a short distance from the observer temporary position, which is a predetermined temporary position of the observer observing the object image displayed on the screen, to the camera movement destination position. The determination apparatus according to claim 2.   The selection unit includes a distance from an observer temporary position, which is a predetermined temporary position of an observer observing the object image displayed on the screen, to the object actual position and the object actual position from the camera destination position. The determination apparatus according to claim 2, wherein the camera having a short difference from the distance to the camera is selected with priority.   The selection unit, when the selection unit selects the camera based on the magnification, selects the camera in which the distance from the camera to the object is a position determined according to the reciprocal of the magnification. Item 2. The determination device according to Item 1. As a result of associating the coordinate system of the first space, which is the real space where the observer who observes the object image displayed on the screen exists, with the coordinate system of the second space, which is the real space where the object exists. The determination apparatus according to claim 1, further comprising: a coordinate processing unit that determines an observer temporary position that is a predetermined temporary position of the observer in the second space. A determination device according to any one of claims 1 to 6,
A moving unit for moving the selected camera;
An image acquisition unit for acquiring an image including the object image from the moved camera;
A cutout unit that cuts out the object image from the image including the object image;
An image processing apparatus comprising: an output unit that outputs the cut-out object image to a display device. A correction unit that corrects at least one of the orientation and size of the clipped object image;
The image processing apparatus according to claim 7, wherein the output unit outputs the object image in which at least one of a direction and a size of the object image is corrected to the display device. A plurality of cameras whose positions and orientations are determined so as to image an object placed in the real space are arranged in the real space and observed as if they were moved in the space of the image displayed on the screen of the display device A determination method executed by a determination apparatus that selects the camera that outputs an image including an object image that is a still image or a moving image of an object from a plurality of the cameras,
The direction of the line connecting the actual position of the object, which is the actual position of the object, to the temporary position of the object, which is the temporary position of the object, and the camera movement source position, which is the position of the movement source of the camera that captures the object, A determination method including a step of selecting the camera from a plurality of the cameras based on at least one of a direction of a line connecting to a camera movement destination position which is a movement destination position and a predetermined magnification.   The determination program for functioning a computer as a determination apparatus as described in any one of Claims 1-6.

Images