JP2016145733A - Information processing device and method thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To generate characteristic arrangement information suitable for estimating a position attitude of an imaging device in a space.SOLUTION: An input unit 111 inputs camera information on an internal parameter of an imaging device, scene shape information for drawing a scene, and sight line information indicating a position attitude of a sight line in observing a virtual object in the scene. An imaging scene setting unit 150 sets an imaging scene that is a real space imaged by the imaging device in the position attitude indicated by the sight line information, and a depth distance of the imaging scene, on the basis of the scene shape information, the camera information and the sight line information. An arrangement information generation unit 170 generates characteristic arrangement information arranged in a scene, on the basis of the scene shape information, the imaging scene and the depth distance.SELECTED DRAWING: Figure 4

Description

  The present invention relates to information processing for generating arrangement information of features to be arranged in an actual scene in order to estimate the position and orientation of an imaging device in space.

  As a method for estimating the position and orientation of the imaging device in space, there is a method that uses an image feature reflected in a captured image of the imaging device. The technology disclosed in Patent Document 1 measures the position and orientation of an imaging device (a camera fixed to a head mounted display (HMD)) with a magnetic sensor. Then, an error caused by magnetic noise is corrected using image features (color markers arranged in a table), and a position and orientation that is consistent with the captured image is estimated.

  Non-Patent Document 1 discloses image features (square markers and circular markers) having a square outer shape and identification information inside, and by performing image processing and recognizing the markers reflected in the captured image, the imaging device itself A method for estimating the position and orientation of the is described.

  In the techniques of Patent Document 1 and Non-Patent Document 1, a system environment builder needs to arrange image features in a space in advance in consideration of an area imaged by an imaging device. Patent Document 1 describes that the density distribution of color markers is changed according to the distance from the camera (hereinafter referred to as the depth distance) to suppress the number of markers reflected in the captured image. However, although Patent Literature 1 assumes an air hockey game, it does not mention an appropriate density distribution in a case where a wearer of an HMD walks freely in space.

  Non-Patent Document 1 discloses a method in which a plurality of markers (a square marker and a circular marker) are arranged at equal intervals on a paper surface, and the position and orientation of a camera can be estimated even in a situation where an experienced person looks around. However, Non-Patent Document 1 uses only a square marker of a certain size, and for cases where the user walks in a large space and moves, the placement information such as the appropriate marker size and position is provided. The mechanism to present is not disclosed.

  Non-Patent Document 2 discloses a method of measuring the position and orientation of the HMD itself by measuring a plurality of self-luminous markers attached to the HMD to be measured with a measurement camera arranged in a space. Further, Non-Patent Document 2 discloses a method for simulating which location of the HMD should be attached in consideration of the light emission range of the self-light-emitting marker and the hiding due to the shape of the HMD. However, the simulation is only performed in such a way that the measurement camera surrounds the HMD to be imaged in a spherical shape at equal intervals, and it is possible to effectively image the distance between the measurement camera and the marker and an important viewpoint position. A method of arranging the features is not disclosed in Non-Patent Document 2.

  Patent Document 1 and Non-Patent Documents 1 and 2 do not consider the case where there is a shielding object between the imaging device and the image feature, and do not disclose the arrangement and size of the image feature considering the shielding object. Therefore, a technique for providing an arrangement and size of image features suitable for estimating the position and orientation of the imaging device in space is desired.

Japanese Patent No. 3450704

Hirokazu Kato, Mark Billinghurst, Ivan Poupyrev, Shinji Tetsuya, Keihachiro Tachibana "Tangible Interface Using Augmented Reality Technology" Journal of Art and Science, Vol. 1, No. 2, pp. 97-104, 2002 Larry Davis, Eric Clarkson, Janniek P. Rolland `` Predicting Accuracy in Pose Estimation for Marker-based Tracking '' Proceeding of the Second IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR '03), 2003

  An object of the present invention is to generate arrangement information of features suitable for estimating the position and orientation of an imaging device in space.

  The present invention has the following configuration as one means for achieving the above object.

  The information processing according to the present invention draws the camera information regarding the internal parameters of the imaging apparatus and the scene when generating the arrangement information of the characteristics to be arranged in the actual scene in order to estimate the position and orientation of the imaging apparatus in space. Scene shape information and gaze information indicating the position and orientation of the gaze when observing a virtual object in the scene are input, and the gaze information is based on the scene shape information, the camera information, and the gaze information. An imaging scene that is an actual space imaged by the imaging apparatus at the position and orientation shown, and a depth distance of the imaging scene are set, and the scene is arranged in the scene based on the scene shape information, the imaging scene, and the depth distance Generate placement information for.

  According to the present invention, it is possible to generate feature arrangement information suitable for estimating the position and orientation of an imaging device in space.

The figure which shows an example of the environment which measures the position and orientation of an imaging device. FIG. 2 is a diagram showing an example of marker arrangement in the environment shown in FIG. The block diagram which shows the structural example of the information processing apparatus of an Example. The block diagram which shows the function structural example of the information processing apparatus which performs the presentation process of marker arrangement | positioning in an Example. The figure explaining the imaging scene in the position and orientation which three eyes | visual_axis information shows. The figure which shows the example of an area | region division. The figure which shows an example of UI. The flowchart explaining the example of a procedure of the presentation process of marker arrangement | positioning. The flowchart explaining the process of an imaging scene setting part and an area division part. The flowchart explaining the process of an arrangement | positioning information generation part. The figure which shows an example of a marker table. The figure explaining arrangement | positioning of a marker. The figure explaining arrangement | positioning of a marker. The figure which shows the example of a final marker arrangement | positioning. The figure which shows the example of final marker arrangement | positioning in the case of performing the mixed arrangement | positioning of the marker from which size differs. The figure which shows the example which combines the camera for observation, and the camera for position and orientation estimation.

  Hereinafter, an information processing apparatus and an information processing method according to an embodiment of the present invention will be described in detail with reference to the drawings. In addition, an Example does not limit this invention concerning a claim, and all the combinations of the structure demonstrated in an Example are not necessarily essential for the solution means of this invention.

[Measurement of position and orientation of imaging device]
An outline of a method for estimating the position and orientation of the imaging device in space will be described. FIG. 1 shows an example of an environment for measuring the position and orientation of the imaging apparatus. Figure 1 shows the case of observing a three-dimensional computer graphics model (hereinafter referred to as CG model) 330 indoors, which is a real scene. Is assumed.

  Note that the environment shown in FIG. 1 includes wall surfaces 301 and 302 and a floor surface 303 on which image features (hereinafter referred to as markers), which are features for estimating the position and orientation of the imaging device in space, are arranged. In addition, it is assumed that a fixture 310 such as a desk or table is disposed in the environment illustrated in FIG. 1 and a structure 320 such as a pillar is present. Reference numerals 100a to 100c denote the positions and orientations of the camera 100, which is an imaging device fixed to the HMD.

  FIG. 2 shows an example of marker arrangement in the environment shown in FIG. The marker arrangement shown in FIG. 2 is an example in which markers of the same size are arranged at equal intervals on the wall surfaces 301 and 302 and the floor surface 303 according to the method of Non-Patent Document 1. Some of the markers are shielded by the structure 320 and the fixture 310, and the markers may not be captured by the camera 100 in some cases.

  Originally, even if there is a marker shielded by a fixture or a structure, the position and orientation of the camera 100 can be estimated if at least one marker not shielded from the captured image can be detected. However, when the camera 100 moves in a wide range, there is a high possibility that there is a place where the unshielded marker cannot be photographed, and it is difficult to estimate the position and orientation of the camera 100 in such a place. When there is a place where it is difficult to estimate the position and orientation of the camera 100, the observer cannot observe the CG model 330 smoothly.

  In order to prevent the above problems, it is necessary to construct an environment that adjusts the arrangement and size of the markers so that the observer will not be blocked at a position where the observer observes the CG model 330 in advance. . In other words, it takes trial and error to place the marker, such as moving the marker on the site, printing again to a marker size that fits within the angle of view of the camera 100, and attaching the marker again.

  In the following, an information processing apparatus according to an embodiment for generating marker placement information to be placed in a real scene (hereinafter referred to as an MR environment) as shown in FIG. 1, which is an environment for an observer to experience augmented reality (MR), and An information processing method will be described.

[Device configuration]
The block diagram of FIG. 3 shows a configuration example of the information processing apparatus of the embodiment. The CPU 201 uses the RAM 202 as a work memory, executes an OS and various programs stored in the ROM 203 and the storage unit 204, and controls a configuration to be described later via the system bus 205.

  The CPU 201 inputs a user instruction from the operation unit 207 such as a keyboard and a mouse via a general-purpose interface 206 such as USB. The CPU 201 executes a program stored in the storage unit 204 according to a user instruction, and displays a user interface (UI), a process progress, and a process result on the monitor 209 via the video card (VC) 208 according to the program. . In addition to the operation unit 207, a memory card reader or the like can be connected to the general-purpose interface 206, and the CPU 201 can read various data from the memory card or the like.

  The storage unit 204 is a hard disk drive (HDD) or a solid state drive (SSD). In the program stored in the storage unit 204, a marker disclosed in Non-Patent Document 1 is placed in the indoor environment shown in FIG. 1, and the CG model 330 is added to the HMD 211 using the measured position and orientation of the camera 100. Contains programs to display. That is, the observer wearing the HMD 211 can experience augmented reality (MR) in which the object represented by the CG model 330 is arranged in the environment shown in FIG.

  When executing a program for MR, the CPU 201 transmits image data to the HMD 211 by wired or wireless connection via the interface 210 and receives captured image data from the camera 100 fixed to the HMD 211. Note that the CPU 201 may use a GPU installed in the VC 208 to generate image data to be transmitted to the HMD 211, but details thereof are omitted.

  Note that the CPU 201 can also send and receive programs, data, and arithmetic processing results to and from computer devices and server devices on a wired or wireless network via a network I / F (not shown) connected to the system bus 205. it can. Further, the monitor 209 and the operation unit 207 may be a touch panel in which they are overlapped. In that case, the information processing apparatus may be a computer device such as a tablet device or a smartphone.

  In addition, the program stored in the storage unit 204 includes a program that realizes a process for presenting a marker placement method to be placed in an MR environment that allows an observer to experience MR, and the CPU 201 executes a marker placement presentation process described later. To do.

[Function configuration]
FIG. 4 is a block diagram illustrating an example of a functional configuration of an information processing apparatus that executes a marker placement presentation process in the embodiment. Note that the functional configuration shown in FIG. 4 is realized by the CPU 201 executing a marker placement presentation program.

  The scene shape input unit 110 inputs information representing the MR environment (hereinafter, scene shape information). For example, a three-dimensional model (hereinafter referred to as a 3D model) indicating the arrangement and shape of the wall surfaces 301 and 302 and the floor surface 303 and the fixture 310 and the structure 320 of the environment shown in FIG. 1 is input. As the 3D model, for example, a 3D polygon model created by CAD can be used. Alternatively, a polygon model may be generated from a point cloud acquired by a measuring instrument that can acquire the three-dimensional shape of the MR environment at once, and the polygon model may be input as scene shape information.

  The fixture 310 and the structure 320 are objects (shields) that exist between the marker placed in the scene and the camera 100 and may shield the marker from the camera 100. Shields are not limited to desks and pillars, and include pot plants, banners, banners, stand-type lighting, chairs including sofas, partitions, and the like.

  The model shape input unit 120 inputs information (hereinafter, object shape information) indicating the 3D shape of an object (hereinafter referred to as a virtual object) to be observed by the experience person. For example, a CG model 330 representing the vehicle shown in FIG. 1 is input. The input scene shape information is not limited to the room size, and the input object shape information is not limited to the CG model representing the vehicle.

  The camera information input unit 130 inputs camera internal parameters of the camera 100 necessary for determining the marker size, position and orientation as camera information. The camera internal parameters include, for example, the angle of view, the resolution of the captured image, and the principal point position. When the camera 100 having a large lens distortion is used, the lens distortion correction center position and the distortion correction parameter are added to the camera internal parameters.

  The line-of-sight information input unit 140 inputs the position and orientation that the camera 100 can take as line-of-sight information. For example, assuming that the experiencer observes a virtual object placed in the scene, the position / orientation 100a-103c of the camera 100 shown in FIG. 1 is input. For example, the position / posture 100a is a position / posture from which the virtual object is viewed from the front, the position / posture 100b is a position / posture for observing the rear part of the virtual object (for example, a rear curve of the vehicle), and the position / posture 100c is an upper part of the virtual object (for example, It is a position and orientation for observing the top curve of the vehicle.

  The scene shape input unit 110, the model shape input unit 120, the camera information input unit 130, and the line-of-sight information input unit 140 constitute an input unit 111 in the marker arrangement presentation process. The scene shape information, object shape information, camera information, and line-of-sight information input by the input unit 111 are stored in a predetermined area of the storage unit 204.

  Based on the scene shape information, camera information, and line-of-sight information stored in the storage unit 204, the imaging scene setting unit 150 sets a scene (hereinafter, an imaging scene) that the camera 100 captures at the position and orientation indicated by each line-of-sight information. The imaging scene is a real space photographed by the camera 100. An imaging scene in the position and orientation indicated by the three line-of-sight information will be described with reference to FIG.

  FIG. 5A shows an imaging scene 610 in the position / orientation 100a. Similarly, FIGS. 5B and 5C show the imaging scenes 620 and 630 in the positions and orientations 100b and 100c. The acquisition method of the imaging scene is the same, and the acquisition method will be described with the imaging scene 610 as a representative.

  For example, the imaging scene 610 can be acquired by back projecting the value of the depth buffer when the 3D polygon model of the scene shape information is drawn from the position and orientation 100a as the viewpoint onto the 3D polygon model of the scene shape information. At this time, when the depth buffer is 8 bits and the pixel value is initialized to 255, a pixel area having a value other than 255 is an area corresponding to the imaging scene 610 in the position and orientation 100a. The captured scene 610 is stored as a texture of a 3D polygon model.

  Note that acquisition and storage of an imaging scene are not limited to acquisition by reprojection using a depth buffer or storage as a texture. Any method that can identify the area captured by the camera 100 based on the three-dimensional information of the scene or that can store information in association with the three-dimensional area of the scene can be applied to acquisition and storage of the captured scene. it can.

  As shown in FIG. 5 (a), when the imaging scene 610 is back-projected onto the 3D polygon model, it can be seen that there is an area (hereinafter referred to as an occluded area) 615 that is shielded by the fixture 310 or the structure 320 and is not imaged. Since the depth buffer value is not reprojected in the shielding area 615, it does not become a marker arrangement area, which will be described later. In other words, waste of attaching a marker to an area that cannot be seen from the camera 100 can be eliminated.

  Next, the area dividing unit 160 divides the imaging scenes 610, 620, and 630 into areas based on the distances indicated by the scene shape information and the line-of-sight information. For example, the area is divided every 1 m from the camera 100 (depth distance). FIG. 6 shows an example of area division. FIG. 6A shows a divided area obtained by dividing the area of the imaging scene 610 for each depth distance of 1 m. For example, the area 710 corresponds to an area divided in a range of a depth distance of 1 m to 2 m. Similarly, the region 720 corresponds to a region greater than 2 m and less than 3 m, the region 730 corresponds to a region greater than 3 m and less than 4 m, and the region 740 corresponds to a region greater than 4 m and less than 5 m. FIGS. 6 (b) and 6 (c) show divided areas obtained by dividing the areas of the imaging scenes 620 and 630 for each 1m depth distance.

  The interval of the depth distance of the area division is not limited to 1 m, and the interval can be increased or decreased as necessary. For example, if the maximum distance between the camera 100 and the target wall or floor to which the marker is to be attached is half of the environment shown in FIG. 1, the interval between the depth distances of the divided regions may be half, 50 cm. Moreover, you may change the space | interval of the depth distance of a division area according to depth distance. For example, a depth distance of 1 m may be 50 cm intervals, a depth distance of 1 m to 3 m may be 1 m intervals, and a depth distance of 3 m or more may be 2 m intervals.

  The arrangement information generation unit 170 generates arrangement information indicating the size and position of the marker to be arranged in the scene, and stores the arrangement information in the storage unit 204, as will be described in detail later, based on the divided areas divided by the area division unit 160. To do.

  Based on at least the arrangement information and scene shape information stored in the storage unit 204, the image generation unit 180 generates an image indicating the size and position of the marker to be arranged in the scene (hereinafter referred to as a marker arrangement image) and displays it on the UI. To do.

FIG. 7 shows an example of a UI displayed on the monitor 209 by the marker arrangement presentation program. The marker arrangement image may include a CG model 330 drawn from the object shape information and a line-of-sight object indicating the position and orientation of the camera 100 drawn from the line-of-sight information. The UI display unit 500 includes, for example, the following 3D A marker arrangement image composed of objects is displayed.
Scene (including wall surfaces 301, 302, floor surface 303, fixture 310, structure 320),
Virtual object (CG model 330),
Line-of-sight objects 101-103 (each corresponding to position and orientation 100a-100c),
Marker object 507 (indicates marker size and position / posture),
Coordinate axis 501 (indicates the coordinate system of the scene).

  Note that it is not essential to display the marker arrangement image having the configuration shown in FIG. It is sufficient that at least marker arrangement information can be presented to the builder (hereinafter referred to as user) of the MR environment.

  When the user presses or touches the “scene input” button 510 of the UI, the image generation unit 180 displays a file selection dialog (not shown) on the UI. The user can select a file indicating scene shape information stored in advance in the storage unit 204 or the like by operating the dialog. Similarly, when the user presses or touches the “model input” button 520, the image generation unit 180 displays a file selection dialog (not shown) on the UI. The user can select the file indicating the object shape information stored in advance in the storage unit 204 or the like by operating the dialog.

  Thus, the UI functions as the scene shape input unit 110 and the model shape input unit 120. Similarly, the UI input unit 530 functions as the camera information input unit 130, and the UI input unit 540 functions as the line-of-sight information input unit 140. Further, the user can change the position and orientation of the line-of-sight object 101-103, the CG model 330, the fixture 310, and the like with the cursor 505, for example.

  When the user presses the “save” button 550 after the change of the position and orientation, the imaging scene setting unit 150, the region dividing unit 160, and the arrangement information generating unit 170 generate arrangement information corresponding to the changed position and orientation. The arrangement information is stored in the storage unit 204, or the arrangement information previously stored in the storage unit 204 is updated by the arrangement information. When the user presses the “end” button 560, the CPU 201 ends the marker arrangement presentation process.

  When the user selects any of the line-of-sight objects, the line-of-sight object is highlighted with, for example, a thick line, and the input unit 540 indicates the position and posture (rotation axis and rotation angle) as position / posture information of the line-of-sight object. A numerical value is displayed. The user can change the numerical value indicating the position and orientation displayed on the input unit 540. As the user changes the numerical value of the position and orientation, the display of the line-of-sight object is also changed.

[Marker placement presentation processing]
A procedure example of marker arrangement presentation processing will be described with reference to the flowchart of FIG. Note that the processing procedure shown in FIG. 8 is realized by the CPU 201 executing a marker placement presentation program. When the marker arrangement presentation process is started, the CPU 201 causes the image generation unit 180 to display a default UI on the monitor 209 (S101).

  Next, the CPU 201 determines a user operation on the UI, and determines whether the scene shape information, object shape information, camera information, or line-of-sight information has been updated (S102). If at least one of these pieces of information has been updated, the CPU 201 moves the process to step S103, and executes a marker arrangement information update process.

  On the other hand, when none of the scene shape information, object shape information, camera information, and line-of-sight information has been updated, the CPU 201 determines whether or not the end button 560 has been pressed (S107). If the end button 560 is pressed, the marker arrangement presentation process is terminated, and if the end button 560 is not pressed, the process returns to step S102.

  In the marker arrangement information update process, the CPU 201 draws an imaging scene corresponding to the line-of-sight information by the imaging scene setting unit 150 (S103), and sets the above-described divided regions by the region dividing unit 160 (S104). Then, the arrangement information generation unit 170 generates marker arrangement information based on the divided areas (S105). Details of these processes will be described later. Then, the CPU 201 generates a marker arrangement image using the image generation unit 180, displays the marker arrangement image on the UI, and advances the process to step S107.

Imaging Scene Setting Unit and Region Dividing Unit The processing of the imaging scene setting unit 150 (S103) and the processing of the region dividing unit 160 (S104) will be described with reference to the flowchart of FIG. Based on the scene shape information, camera information, and line-of-sight information, the imaging scene setting unit 150 draws a scene (hereinafter, an imaging scene) that the camera 100 captures at the position and orientation indicated by the line-of-sight information (S111). Then, the 3D polygon model representing the imaging scene is stored in the storage unit 204 or the like (S112). If there is a plurality of line-of-sight information, one of the plurality of line-of-sight information is selected to draw an imaging scene.

  When drawing a scene, generally, the 3D vertex information of the polygons that make up the drawing scene is projected onto the projection surface of the virtual camera, and the interior of the polygon is filled in pixel units while taking into account the depth distance from the camera. It is. As a drawing result, an image of a scene viewed from the position and orientation of the line-of-sight information and a distance image are generated. The bit depth of the distance image is often 8 bits, but in the subsequent processing, it is necessary to refer to the actual distance between the viewpoint and the polygon, so the bit depth of the distance image is expanded to 24 bits, for example. . Alternatively, a table indicating the correspondence between the bit depth of 8 bits and the actual distance may be prepared.

  Next, the imaging scene setting unit 150 re-projects the generated distance image as a texture onto a 3D polygon model representing the imaging scene (S113). For example, the distance image is re-projected to the 3D polygon model by performing processing for all pixels in the distance image to find the point where the straight line connecting the center point of the pixels constituting the distance image and the viewpoint position intersects the 3D polygon model. can do.

  The above is processing corresponding to step S103, and the subsequent processing is processing corresponding to step S104.

  Next, the region dividing unit 160 sets a distance classification based on the pixel value of the corresponding distance image as the texture pixel value (S114), and stores the texture in which the pixel value is set in the storage unit 204 or the like (S115). . For example, “0” is set as the pixel value in an area having a depth distance of 1 m or less. Similarly, "1" for the pixel value in the area above 1m and 2m, "2" for the pixel value in the area above 2m and 3m, "3" for the pixel value in the area above 3m and 4m and below 5m and below 4m Set the pixel value of the area to “4”.

  In addition, instead of setting the distance category for the pixel value, the depth value of the distance image may be set to the pixel value. When the bit depth of the distance image is 8 bits, the 8-bit depth value is used as the texture pixel value. Should be stored.

  Next, the area dividing unit 160 determines whether or not the processing of steps S111 to S115 has been completed for all the line-of-sight information (S116). If there is line-of-sight information that has not been processed, the process returns to step S111, and the processes of steps S111 to S115 are repeated.

Arrangement Information Generation Unit The processing (S104) of the arrangement information generation unit 170 will be described with reference to the flowchart of FIG. The arrangement information generation unit 170 reads a 3D polygon model representing an imaging scene corresponding to one of the line-of-sight information from the storage unit 204 (S121), groups polygons that can be regarded as the same plane, and presents a marker arrangement ( Hereinafter, the marker placement presentation surface) is extracted (S122).

  The grouping is processing for handling the polygons as a group on the same plane when the wall surfaces 301 and 302 and the floor surface 303 are composed of a plurality of polygons. If the length of the short side of the grouped plane is less than a predetermined value (for example, less than 10 cm), the subsequent processing of the group is skipped.

  That is, a location corresponding to a narrow surface that does not match the shape to which the marker is pasted, such as the curved surface of the structure 320 included in the scene shape information, is extracted so that the marker arrangement is not presented. For example, in the case of the scene shape information shown in FIG. 1, the wall surfaces 301 and 302, the floor surface 303, and the fixture 310 are listed as marker placement presentation surfaces, and the placement information generation unit 170 performs the following for each marker placement presentation surface. Execute the process.

  The arrangement information generation unit 170 reads the texture corresponding to the imaging scene from the storage unit 204, and acquires the distance category from the texture pixel value (S123). Then, referring to the marker table stored in the storage unit 204 or the like, the size and arrangement interval of the marker corresponding to the distance section are acquired (S124). FIG. 11 shows an example of the marker table. In the marker table, the marker size (side length) and the arrangement interval corresponding to the distance classification are recorded. In addition, when the depth value is stored in the pixel value of the texture, the marker size and the arrangement interval may be acquired by referring to the distance column of the marker table.

  Next, the arrangement information generation unit 170 arranges a marker for each presentation surface of the marker arrangement (S125). The arrangement of the markers will be described with reference to FIGS. FIG. 12 shows a presentation surface corresponding to the divided area 740 obtained by the area division shown in FIG. 6 (a), and a presentation corresponding to the divided area 730 obtained by the area division shown in FIG. 6 (b). Show the surface. A region 615 is a shielding region by the structure 320 detected by back projection of the imaging scene 610 shown in FIG. Hereinafter, the presentation surface corresponding to the divided region 740 is referred to as a presentation surface 740, and the presentation surface corresponding to the divided region 730 is referred to as a presentation surface 730.

  As shown in FIG. 13 (a), the arrangement information generation unit 170 arranges square markers with a side length of 45 cm, for example, at intervals of 75 cm on the presentation surface 740 based on the marker table. Note that a rectangle including the presentation surface 740 is set, and markers are arranged at the size and interval indicated by the marker table with the upper left vertex of the rectangular parallelepiped as a reference.

  Next, the arrangement information generation unit 170 moves, in parallel with at least one of the vertical direction and the horizontal direction, the marker that protrudes from the presentation surface 740 among the markers shown in FIG. Adjust its position to fit. When the moving marker overlaps with an adjacent marker in an area ratio of 40% or more, for example, the moving marker is deleted. When the superposition is less than 40%, the moving marker is moved in the reverse direction to the position where the superposition is canceled. FIG. 13 (b) shows the arrangement of the markers after the arrangement adjustment.

  The arrangement information generation unit 170 performs the same processing on the presentation surface 730, and arranges, for example, 35 cm markers at 60 cm intervals as shown in FIG. 13 (d). For example, the marker 980 is a marker that has been moved upward so that the overlap with the marker adjacent in the downward direction is 40% or more and the overlap is eliminated. The adjustment of the marker arrangement is not limited to the movement of the marker arranged in the rectangle including the divided area, and the marker may be rotated or the marker size may be changed.

  Next, the arrangement information generation unit 170 deletes or moves the marker arranged for each presentation surface so as not to overlap the shielding area 615 (S126). The arrangement information generation unit 170 lists markers that overlap the shielding area 615 from the markers arranged on the presentation surfaces 730 and 740. In the example of FIG. 13 (b), the markers 910, 920, 930, and 940 overlap with the shielding area 615, and in the example of FIG. 13 (d), the markers 950, 960, 970, 990, and 995 overlap with the shielding area 615. The arrangement information generation unit 170 deletes a marker whose area overlaps with the shielding region 615, for example, 40% or more, from among the listed markers, and a marker whose overlay is less than 40% moves in the vertical direction or the horizontal direction so as to cancel the overlap. To do.

  In FIG. 13 (b), markers 910 and 920 are deleted from the markers overlapping with the shielding region 615, and the markers 930 and 940 are moved to the right so as not to overlap with the shielding region 615 (FIG. 13 (c)). . Similar processing is performed on the presentation surface 730, and the marker is arranged so as not to overlap the shielding area 615.

  Next, the arrangement information generation unit 170 integrates the marker arrangement for each presentation surface, determines the final marker arrangement for the attention surface, generates the arrangement information (S127), and stores the generated arrangement information in the storage unit 204. (S128). FIG. 14 shows a final marker arrangement example. Fig. 14 integrates the marker arrangement on the presentation surface 740 shown in Fig. 13 (c) and the marker arrangement obtained by deleting the markers 650, 960, 970, 990, 995 from the marker arrangement on the presentation surface 730 shown in Fig. 13 (d). Shows how it was done.

  That is, the marker arrangement shown in FIG. 13 (d) is superimposed on the marker arrangement shown in FIG. 13 (c). When markers on different presentation surfaces partially overlap, a small marker (hereinafter referred to as a small marker) is left. The small marker is detected as a marker even when the distance between the camera 100 and the marker is long, and the position and orientation of the camera 100 can be measured. On the other hand, it is considered that a marker having a large size (hereinafter referred to as a large marker) does not fall within the angle of view of the camera 100, and the number of scenes in which the position and orientation of the camera 100 cannot be estimated increases. Therefore, a small marker is left.

  Next, the arrangement information generation unit 170 determines whether or not marker arrangement information has been generated for all of the marker arrangement presentation surfaces (S129). Return to S123. When the marker placement information is generated for all the marker placement presentation surfaces, it is determined whether or not the marker placement information has been generated for the imaging scene corresponding to all of the line-of-sight information (S130). If there is, the process returns to step S121.

  As described above, based on the scene shape information, camera information, and line-of-sight information, marker placement information suitable for estimating the position and orientation of the camera 100 in the MR environment can be presented. In particular, even when the camera 100 moves in a wide range in the MR environment and there is a shield between the camera 100 and the marker in some areas, it is possible to present marker placement information suitable for estimating the position and orientation of the camera 100 become. Also, by inputting the viewpoint position indicating the observation position and orientation of the observer in the MR environment, it is possible to present in advance the marker placement information appropriate for the MR environment, and the marker of the marker by the user who is the MR environment builder Time and cost required for creation and placement can be reduced.

  In addition, although 40% was cited as an example of the criteria for determining the overlapping area ratio between adjacent markers and the area ratio at which the marker overlaps the shielding area, it is not limited to this, and the criteria can be changed as necessary. May be.

[Mixed arrangement of markers of different sizes]
In the above description, the example in which the small marker is preferentially arranged when the marker arrangement for each divided region is integrated has been described. However, the present invention is not limited to the priority arrangement of small markers, and markers of different sizes may be mixed and arranged on the surface in consideration of the stability of the process for estimating the position and orientation of the camera 100.

  In general, in order to estimate the position and orientation of the camera 100 with high accuracy, it is desirable that a marker is photographed as large as possible and markers dispersed on the image are photographed. In order to carry out the mixed arrangement of the marker sizes, the process of step S127 by the arrangement information generation unit 170 is changed as follows.

  The arrangement information generation unit 170 detects the superposition of the markers by superimposing the marker arrangements for each presentation surface. When markers of different sizes are superimposed, the distance between adjacent large markers is acquired, and if the distance is equal to or greater than a threshold, the large marker is preferentially arranged.

  For example, when the arrangement of the large marker shown in FIG. 13 (c) and the arrangement of the small marker shown in FIG. 13 (d) are overlapped, the large marker 945 and the small marker 955 are overlapped. When the distance between the large marker 945 and the large marker adjacent to the large marker 945 is 2 m or more, for example, the large marker 945 is disposed instead of the small marker 955. FIG. 15 shows a final marker arrangement example in the case of performing mixed arrangement of markers having different sizes. Compared with the example of the priority arrangement of the small markers shown in FIG. 14, the marker arranged in the lower left is changed from the small marker 955 to the large marker 945.

  In this way, not only the priority placement of small markers is always performed, but also placement information is generated in consideration of the distribution of markers of the same size. In other words, marker arrangements are integrated based on the distribution of markers of the same size, and mixed arrangement of markers having different sizes is performed. By considering the distribution of markers of the same size, the position and orientation estimation accuracy of the camera 100 can be improved even when the distance between the camera 100 and the wall is long.

[Add gaze information]
In the above, an example of generating marker placement information using the input viewpoint information has been described. However, the line of sight when observing the CG model 330 is predicted, and the predicted line of sight information is added to generate the placement information. You can also According to the prediction of the line-of-sight information, it is possible to present marker arrangement information that can estimate the position and orientation of the camera 100 when the CG model 330 is observed from a position and orientation that is not assumed by the system builder who is the user. . On the other hand, for the observer, the CG model 330 can be observed from a wider range.

  The line-of-sight information prediction unit 190 predicts line-of-sight information to be added from the line-of-sight information input to the line-of-sight information input unit 140, and stores the predicted line-of-sight information in the storage unit 204. The line-of-sight information prediction unit 190, for example, combines line-of-sight information input or added by the user two by two, adds line-of-sight information corresponding to an intermediate position indicated by the line-of-sight information, An intermediate posture is set by combining the postures of the two line-of-sight information.

  In addition to predicting and adding new line-of-sight information from the line-of-sight information input or added by the user, the line-of-sight information when a similar scene has been experienced in the past is sampled, and new line-of-sight information is obtained from the line-of-sight information. Gaze information can be predicted and added. For example, the position and orientation of the camera 100 when an observer observes the same CG model 330 in another environment is expected to appear with high probability even in an environment to be constructed.

  A “Gaze information reading” button 570 is added to the UI shown in FIG. When the “Gaze information reading” button 570 is pressed or touched, the gaze information input unit 140 reads gaze information sampled in the past from the storage unit 204 or the like. The line-of-sight information prediction unit 190 predicts new line-of-sight information from the read line-of-sight information, and stores the predicted line-of-sight information in the storage unit 204.

[Separate display of CG model when entering line-of-sight information]
The example of the input unit 540 illustrated in FIG. 7 has been described above as an example of the line-of-sight information input unit 140. When the user inputs the position and orientation of the line-of-sight information, the user adjusts the positional relationship between the CG model 330 and the camera 100 with reference to the overhead view image displayed on the display unit 500. However, the overhead viewpoint image displayed on the display unit 500 is drawn based on the scene shape information of the MR environment, and is not an image drawn based on the line-of-sight information. Therefore, the user needs to set the line-of-sight information by imagining the image of the CG model 330 observed from the viewpoint of the camera 100 based on the image of the overhead viewpoint, and trial and error is necessary for setting the line-of-sight information There is a case.

  In order to provide an image of the CG model 330 observed from the viewpoint of the camera 100, the image generation unit 180 displays the image of the CG model 330 corresponding to the line-of-sight information as a separate screen on the monitor 209 when inputting the position and orientation of the line-of-sight information. indicate. That is, the image generation unit 180 draws the CG model 330 based on the position and orientation of the line-of-sight information set by the user and the camera information, and displays the drawing result on another screen.

  Note that the CG model 330 placed in the MR environment may be drawn including scene information in the drawing for another screen, and the display of the image of the CG model 330 corresponding to the line-of-sight information is not limited to another screen. This may be done by switching the image of unit 500.

[Gaze information position and orientation sensor input]
In the above, an example in which the position and orientation of the line-of-sight information is numerically input has been described. However, the position and orientation can be input using a position sensor, a posture sensor, and a position and posture sensor connected to the general-purpose interface 206 or the interface 210. it can. As the attitude sensor, for example, InertiaCube3 (trademark) manufactured by Thales Visionix may be used.

[Marker placement for different cameras in different directions]
In the above description, the system environment builder who is a user uses the camera 100 as an observation camera of the CG model 330 and uses the marker reflected in the image to estimate the position and orientation of the camera 100. However, the present invention is not limited to using one imaging device for observation and position / orientation estimation simultaneously, and a plurality of cameras may be prepared, one for observation and one for position / orientation estimation.

  FIG. 16 shows an example in which an observation camera 100 and a position / orientation estimation camera 1610 are combined. Note that the camera 100 and the camera 1610 are fixed to each other and moved. The video from the camera 1610 is input to the information processing apparatus via the interface 210 in the same manner as the camera 100. In addition, camera information of the camera 1610 is input to the camera information input unit 130, and the imaging scene setting unit 150 calculates the distance of the camera 1610 from an imaging scene, a wall, and the like.

  For example, when the camera 1610 is attached in the downward direction with respect to the camera 100, the area of the imaging scene area of the camera 1610 is suppressed relative to the area of the imaging scene area corresponding to the position and orientation of the line of sight during observation. Can do. For example, if the posture is to face the CG model 330, the downward facing camera 1610 captures almost the floor surface 303, and the maximum distance between the imaging device and the wall or floor tends to decrease. If the area of the imaging scene is reduced, the time for attaching the marker to the MR environment is reduced, and the cost for printing the marker can be reduced.

[Other Examples]
The present invention supplies a program that realizes one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and one or more processors in a computer of the system or apparatus read and execute the program It can also be realized by processing. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

  111… Input unit, 150… Imaging scene setting unit, 170… Arrangement information generation unit

Claims (13)

An information processing apparatus that generates arrangement information of features to be arranged in an actual scene in order to estimate a position and orientation of an imaging apparatus in space,
Input means for inputting camera information related to internal parameters of the imaging apparatus, scene shape information for drawing the scene, and line-of-sight information indicating the position and orientation of the line of sight when observing a virtual object in the scene;
Based on the scene shape information, the camera information, and the line-of-sight information, an imaging scene that is an actual space imaged by the imaging apparatus in the position and orientation indicated by the line-of-sight information, and a setting unit that sets a depth distance of the imaging scene ,
An information processing apparatus comprising: generation means for generating arrangement information of features to be arranged in the scene based on the scene shape information, the imaging scene, and the depth distance.   2. The information processing apparatus according to claim 1, further comprising an image generation unit configured to generate an image indicating the arrangement of the features in the scene based on the scene shape information and the arrangement information. The input means further inputs object shape information for drawing the virtual object,
3. The information processing apparatus according to claim 2, wherein the image generation unit generates an image in which the feature and the virtual object are arranged in the scene.   4. The information processing apparatus according to claim 3, wherein the image generation unit further generates an image for displaying the virtual object observed in the position and orientation indicated by the line-of-sight information.   The scene shape information includes information for drawing a wall surface and a floor surface of the scene, and a feature arranged in the scene and the imaging device, and shields the feature from the imaging device. 5. The information processing apparatus according to any one of claims 1 to 4, wherein information for drawing an object that is likely to be displayed is included. The generation unit extracts a presentation surface as a surface for presenting a marker arrangement from the imaging scene,
Refer to a predetermined table based on the depth distance corresponding to the imaging scene, obtain the size and arrangement interval of the feature to be arranged on the presentation surface,
6. The information processing apparatus according to claim 5, wherein the arrangement information is generated by arranging the feature on the presentation surface based on the size and an arrangement interval, and deleting or moving the feature shielded by the object.   7. The information processing apparatus according to claim 6, wherein when there are a plurality of presentation surfaces, the generation unit generates the arrangement information by integrating the arrangement of the features on each presentation surface.   8. The information processing apparatus according to claim 7, wherein the generation unit preferentially arranges features having a small size in the integration of the arrangement of the features.   8. The information processing apparatus according to claim 7, wherein the generation unit integrates the arrangement of the features based on a distribution of features of the same size.   10. The information according to claim 1, wherein the input unit inputs a plurality of line-of-sight information, and the setting unit sets the imaging scene and the depth distance for each of the plurality of line-of-sight information. Processing equipment.   11. The information processing apparatus according to claim 10, further comprising prediction means for predicting additional line-of-sight information based on the plurality of line-of-sight information. An information processing method for generating arrangement information of features to be arranged in an actual scene in order to estimate the position and orientation of an imaging device in space,
The input means inputs camera information related to internal parameters of the imaging device, scene shape information for drawing the scene, and line-of-sight information indicating the position and orientation of the line of sight when observing a virtual object in the scene,
Based on the scene shape information, the camera information, and the line-of-sight information, a setting unit sets an imaging scene that is an actual space captured by the imaging apparatus at a position and orientation indicated by the line-of-sight information, and a depth distance between the imaging scenes And
An information processing method in which a generation unit generates arrangement information of features to be arranged in the scene based on the scene shape information, the imaging scene, and the depth distance.   12. A program for causing a computer to function as each unit of the information processing apparatus according to any one of claims 1 to 11.

Images