JP2021144472A - Information processing system, information processing method, and program - Google Patents

Abstract

To provide an information processing system, information processing method and program, which allow for generating images all around an object along a contour thereof.SOLUTION: An information processing system provided herein comprises an estimation unit and an image generation unit. The estimation unit estimates a three-dimensional shape of an object based on an image of the object on a screen. The image generation unit estimates a space occupied by the object on the screen based on the three-dimensional shape of the object and generates an image to be projected on the screen avoiding the space.SELECTED DRAWING: Figure 1

Description

The present invention relates to information processing systems, information processing methods and programs.

In projection mapping, the projected image is calculated in consideration of the three-dimensional positional relationship between the projector and the screen and the unevenness of the screen. When an object is inserted on the screen, the image calculated to be projected on the screen is projected on the object, and a distorted image is displayed on the object. Therefore, it is considered to project an image while avoiding an object. As a technique for projecting an image while avoiding an object, for example, the techniques of Patent Documents 1 and 2 are known.

Japanese Unexamined Patent Publication No. 2018-09714 Japanese Unexamined Patent Publication No. 2016-130663

In the above prior art, the non-projection region is set based only on the two-dimensional shape of the object photographed by the camera. The shape of the back part of the object that becomes the blind spot of the camera cannot be grasped from the image of the camera. Therefore, it is difficult to project an image along the contour of the object on the area on the screen that becomes the blind spot of the camera due to the object.

Therefore, the present disclosure proposes an information processing system, an information processing method, and a program capable of generating an image along the contour of an object over the entire circumference of the object.

According to the present disclosure, an estimation unit that estimates the three-dimensional shape of the object based on an image of the object on the screen, and an estimation unit that estimates the space occupied by the object on the screen based on the three-dimensional shape of the object. An information processing system including an image generation unit that generates an image projected on the screen while avoiding the space is provided. Further, according to the present disclosure, there is provided an information processing method in which the information processing of the information processing system is executed by a computer, and a program for realizing the information processing of the information processing system in the computer.

It is the schematic of the information processing system of 1st Embodiment. It is a figure which shows the correspondence relationship between a local feature defined by an analysis model, and an outline. It is a figure which shows the relationship between a pseudo object and an image. It is a flowchart which shows the information processing method. It is a figure which shows an example of the determination method of the outline in a modification. It is a figure which shows an example of the determination method of the outline in a modification. It is a figure which shows an example of the image projected along the low-likelihood contour region. It is the schematic of the information processing system of 2nd Embodiment. It is a flowchart which shows the information processing method. It is the schematic of the information processing system of 3rd Embodiment. It is a figure which shows an example of the object model defined in the object model information. It is a figure which shows an example of the object model defined in the object model information. It is a flowchart which shows the information processing method.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In each of the following embodiments, the same parts are designated by the same reference numerals, so that duplicate description will be omitted.

The explanation will be given in the following order.
[1. First Embodiment]
[1-1. Information processing system configuration]
[1-2. Information processing method]
[1-3. effect]
[1-4. Modification 1]
[1-5. Modification 2]
[1-6. Modification 3]
[2. Second Embodiment]
[2-1. Information processing system configuration]
[2-2. Information processing method]
[2-3. effect]
[3. Third Embodiment]
[3-1. Information processing system configuration]
[3-2. Information processing method]
[3-3. effect]

[1. First Embodiment]
[1-1. Information processing system configuration]
FIG. 1 is a schematic view of the information processing system 1 of the first embodiment.

The information processing system 1 includes, for example, a processing device 10, a storage device 20, one or more projectors 30, and a camera 40.

The processing device 10 is a device that maps image IMs from one or more projectors 30 on the screen SCR. The screen SCR is, for example, a building or furniture. The processing device 10 detects the object OB inserted on the screen SCR, for example, based on the image taken by the camera 40. The processing device 10 avoids the object OB and generates a video IM projected on the screen SCR.

The camera 40 is a camera capable of detecting the depth information of the subject. The camera 40 is installed above the screen SCR, for example. The camera 40 captures the entire screen SCR from one location above the screen SCR. The processing device 10 detects the depth information of the object OB inserted on the screen SCR based on the image taken by the camera 40.

As a method for detecting the depth information, for example, a passive stereo method, an active stereo method, and a TOF method (Time of Flight) are adopted. The passive stereo method is a method of detecting the depth information of a subject by matching between the images of two cameras. The active stereo method is a method of detecting the depth information of a subject by matching between an image of a pattern projected from a projector and an image taken by a camera. The TOF method is a method of detecting depth information of a subject by irradiating light from a light source and examining the time difference between reflection by an object and return.

One or more projectors 30 are installed above the screen SCR. In the example of FIG. 1, a plurality of projectors 30 including the first projector 30A and the second projector 30B are installed on the screen SCR. The first projector 30A and the second projector 30B are installed at positions facing each other with the center of the screen SCR, for example. The second projector 30B is installed in the vicinity of the camera 40, for example. The optical axis of the camera 40 and the optical axis of the second projector 30B are substantially the same.

The processing device 10 includes, for example, an image acquisition unit 11, a coordinate information extraction unit 12, a rough shape determination unit 13, an estimation unit 14, and a video generation unit 15.

The image acquisition unit 11 acquires data of an image taken by the camera 40. The image acquisition unit 11 outputs the image data to the coordinate information extraction unit 12.

The coordinate information extraction unit 12 extracts the coordinate information of the object OB in the three-dimensional space from the image taken by the camera 40. The coordinate information extraction unit 12 converts, for example, the depth information of the object OB into the coordinate information in the three-dimensional space. For example, the coordinate information extraction unit 12 uses the coordinate information of a part or all of the point cloud of the front portion of the object OB that is not the blind spot of the camera 40 as the coordinate information of the object OB among the point clouds constituting the object OB. Extract.

The outline determination unit 13 determines the outline of the object OB based on the coordinate information of the object OB. For example, the outline determination unit 13 applies the local features of the object OB extracted from the coordinate information of the object OB to the analysis model 23 to determine the outline of the object OB.

The analysis model 23 defines, for example, the correspondence between the local feature of the object OB and the outline shape of the object OB. The analytical model 23 defines, for example, information about the cross-sectional shape and area ratio of a plurality of objects OBs having different heights as local features. The outline is defined as a type of simple object such as a sphere, a cylinder, a prism, or a cone. The analysis model 23 defines information on the outlines of a plurality of simple objects.

The outline determination unit 13 extracts, for example, information on the ratio of the cross-sectional shapes and areas of a plurality of objects OB having different heights from the coordinate information of the object OB. The shape of the object OB on the back surface, which is the blind spot of the camera 40, cannot be grasped from the image of the camera 40. Therefore, the outline determination unit 13 estimates the cross-sectional shape of the object OB based on the shape of the object OB in the front portion of the camera 40 that is not a blind spot. For example, the outline determination unit 13 sets a bounding box that surrounds a point cloud that constitutes the contour of the cross section. If the coordinates of the point cloud do not exist at the corners of the bounding box, the outline determination unit 13 determines that the cross-sectional shape is a circle. When the coordinates of the point cloud are present at the corners of the bounding box, the outline determination unit 13 determines that the cross-sectional shape is a square. The outline determination unit 13 determines the type of the simple object associated with the extracted information as the outline of the object OB. The outline determination unit 13 outputs information on the outline of the object OB to the estimation unit 14.

FIG. 2 is a diagram showing the correspondence between the local features defined by the analysis model 23 and the outline.

In the example of FIG. 2, a first cross section CT1, a second cross section CT2, and a third cross section CT3 are used as a plurality of cross sections that serve as criteria for determining the outline. The first cross section CT1 is, for example, the lowest surface of the object OB (the surface facing the screen SCR). The second cross section CT2 is, for example, the highest surface of the object OB (the surface farthest from the screen SCR). The third cross section CT3 is, for example, a cross section having a height intermediate between the first cross section CT1 and the second cross section CT2.

Of the first cross section CT1 and the second cross section CT2, the area of the larger cross section is A, and the area of the smaller cross section is B. Let C be the area of the third cross section. For example, when the B / A is equal to or higher than the first threshold value and the C / A is equal to or lower than the second threshold value, the general shape of the object OB is determined to be columnar. If the coordinates of the point cloud do not exist at the corners of the bounding box of the cross section having the area A, it is determined to be a cylinder, and if the coordinates of the point cloud exist, it is determined to be a prism.

For example, when the C / A is larger than the third threshold value, it is determined that the approximate shape of the object OB is a sphere. When the B / A is less than the first threshold value and the C / A is less than or equal to the third threshold value, it is determined that the approximate shape of the object OB is a cone. If the coordinates of the point cloud do not exist at the corners of the bounding box of the cross section having the area A, it is determined to be a cone, and if the coordinates of the point cloud exist, it is determined to be a quadrangular pyramid.

Returning to FIG. 1, the estimation unit 14 estimates the three-dimensional shape of the object OB based on the image of the object OB on the screen SCR. The estimation unit 14 generates, for example, a three-dimensional model of a pseudo-object POB that simplifies the object OB based on the outline shape and coordinate information of the object OB. The three-dimensional model of the pseudo-object POB is generated, for example, by combining the information on the outline of the object OB with the dimensional information of the pseudo-object POB (information such as the diameter and height of the cross section of the pseudo-object POB). The dimensional information of the pseudo object POB is estimated based on, for example, the coordinate information of the object OB. The estimation unit 14 estimates the three-dimensional shape of the pseudo-object POB as the three-dimensional shape of the object OB.

The image generation unit 15 estimates the space on the screen SCR occupied by the object OB based on the three-dimensional shape of the object OB estimated by the estimation unit 14. The image generation unit 15 avoids this space and generates an image IM projected on the screen SCR.

FIG. 3 is a diagram showing the relationship between the pseudo-object POB and the video IM.

Image IMs from one or more projectors 30 are projected around the pseudo-object POB. The video IM includes, for example, a first video IM1 projected from the first projector 30A and a second video IM2 projected from the second projector 30B.

A region SA that becomes a blind spot of the camera 40 is formed on the screen SCR by the object OB. The contour SL of the region SA is a projection of the contour POL of the front portion of the pseudo-object POB on the screen SCR. Region SA extends along axis AX. The axis AX is an axis obtained by projecting the optical axis of the camera 40 onto the screen SCR.

The image generation unit 15 generates an image projected on the region SA based on the three-dimensional shape of the pseudo object POB. The generated image is projected onto the area SA from the first projector 30A installed at a position different from that of the camera 40. The first image IM1 and the second image IM2 generated based on the three-dimensional shape of the pseudo object POB are projected on the region NSA on the screen SCR that does not become a blind spot of the camera 40. The first video IM1 and the second video IM2 are generated as images along the contour of the bottom surface of the pseudo-object POB. The pseudo-object POB has a shape that imitates the object OB. Therefore, the image IM along the contour of the object OB is displayed on the screen SCR over the entire circumference of the object OB.

Returning to FIG. 1, the storage device 20 stores the screen model 21, the PJ parameter 22, the analysis model 23, and the program 24.

The screen model 21 is a three-dimensional model of the screen SCR. The screen model 21 includes, for example, coordinate information on the surface of the screen SCR on which the image IM is projected. The PJ parameter 22 includes, for example, information such as an internal parameter, an installation position, and a posture at the installation position for each projector 30.

The video generation unit 15 uses the screen model 21 and the PJ parameter 22 to generate a video IM of geometrically corrected display content for each projector 30. The image generation unit 15 sets, for example, an area on the screen SCR that does not project the image IM for each projector 30 based on the three-dimensional shape of the object OB estimated by the estimation unit 14 and the PJ parameter 22.

The program 24 is a program that causes a computer to execute information processing according to the present embodiment. The processing device 10 performs various processes according to the program 24 stored in the storage device 20. The storage device 20 may be used as a work area for temporarily storing the processing result of the processing device 10. The storage device 20 includes any non-transient storage medium such as, for example, a semiconductor storage medium and a magnetic storage medium. The storage device 20 includes, for example, an optical disk, a magneto-optical disk, or a flash memory. The program 24 is stored, for example, in a non-transient storage medium that can be read by a computer.

The processing device 10 is, for example, a computer composed of a processor and a memory. The memory of the processing device 10 includes a RAM (Random Access Memory) and a ROM (Read Only Memory). By executing the program 24, the processing device 10 functions as an image acquisition unit 11, a coordinate information extraction unit 12, a rough shape determination unit 13, an estimation unit 14, and a video generation unit 15.

[1-2. Information processing method]
FIG. 4 is a flowchart showing the information processing method of the present embodiment.

In step S1, the image acquisition unit 11 acquires an image of the screen SCR taken by the camera 40. The image includes depth information of the object OB inserted on the screen SCR.

In step S2, the coordinate information extraction unit 12 extracts the coordinate information of the object OB in the three-dimensional space based on the depth information of the object OB. The extracted coordinate information is the coordinate information of the front portion of the object OB that does not become a blind spot of the camera 40. The coordinate information of the back surface portion of the object OB, which is the blind spot of the camera 40, is not acquired.

In step S3, the outline determination unit 13 determines the outline of the object OB based on the coordinate information of the object OB. The outline determination unit 13 determines the type of a simple object including the characteristics of the object OB as the outline of the object OB, based on, for example, the ratio of the shapes and areas of a plurality of cross sections at a specific height of the object OB.

In step S4, the estimation unit 14 generates a three-dimensional model of the pseudo-object POB which is a simplification of the object OB in the image. For example, the estimation unit 14 combines the rough shape information determined by the rough shape determination unit 13 with the dimensional information of the pseudo object POB estimated from the coordinate information of the object OB to generate a three-dimensional model of the pseudo object POB. do.

In step S5, the estimation unit 14 estimates the three-dimensional shape of the pseudo-object POB as the three-dimensional shape of the object OB.

In step S6, the image generation unit 15 estimates the space on the screen SCR occupied by the object OB based on the three-dimensional shape of the object OB estimated by the estimation unit 14. The image generation unit 15 avoids this space and generates an image IM projected on the screen SCR.

[1-3. effect]
The information processing system 1 has an estimation unit 14 and a video generation unit 15. The estimation unit 14 estimates the three-dimensional shape of the object OB based on the image of the object OB on the screen SCR. The image generation unit 15 estimates the space on the screen SCR occupied by the object OB based on the three-dimensional shape of the object OB, and generates the image IM projected on the screen SCR while avoiding this space. In the information processing method of the present embodiment, the information processing of the information processing system 1 described above is executed by the computer. The program 24 of the present embodiment makes the computer realize the information processing of the information processing system 1 described above.

According to this configuration, the non-projection region of the image IM is set based on the three-dimensional shape of the object OB. Therefore, it is possible to generate a video IM along the contour of the object OB over the entire circumference of the object OB.

The image generation unit 15 generates, for example, an image IM projected by the object OB onto an area on the screen SCR that becomes a blind spot of the camera 40 that captured the image.

According to this configuration, the image IM along the contour of the object OB is also projected on the area that becomes the blind spot of the camera 40.

The information processing system 1 has, for example, a coordinate information extraction unit 12 and a rough shape determination unit 13. The coordinate information extraction unit 12 extracts, for example, the coordinate information of the object OB from the image. The outline determination unit 13 determines the outline of the object OB based on the coordinate information of the object OB, for example. The estimation unit 14 generates, for example, a three-dimensional model of a pseudo-object POB that simplifies the object OB based on the outline shape and coordinate information of the object OB. The estimation unit 14 estimates, for example, the three-dimensional shape of the pseudo-object POB as the three-dimensional shape of the object OB.

According to this configuration, the three-dimensional shape of the object OB is estimated by a simple calculation. There may be an error between the three-dimensional shape of the actual object OB and the three-dimensional shape of the pseudo-object POB. However, the image IM is not projected in the space occupied by the object OB. Even if there is an error between the three-dimensional shape of the pseudo-object POB and the three-dimensional shape of the actual object OB, the area where the image IM is not projected only increases or decreases. Therefore, the display quality is not significantly impaired.

[1-4. Modification 1]
In the present embodiment, the three-dimensional model of the pseudo-object POB is generated by combining the outline information output from the outline determination unit 13 with the dimensional information of the pseudo-object POB. However, the method of generating a three-dimensional model of the pseudo-object POB is not limited to this. For example, the estimation unit 14 may generate a three-dimensional model of the pseudo-object POB by connecting the coordinates of a plurality of point clouds of the object OB extracted from the image of the camera 40 with a smooth curve. For example, a plurality of cross sections having different heights of an object OB can be connected by NURBS (Non-Uniform Regional B-Spline) to generate a three-dimensional model of a pseudo object POB.

[1-5. Modification 2]
In the present embodiment, the outline determination unit 13 determines the outline of the object OB based on the information regarding the shape and the area ratio of the cross sections of the plurality of objects OB having different heights. However, the method for determining the outline is not limited to this. For example, the outline determination unit 13 can determine the outline of the object OB based on the contour of the object OB captured in the image of the camera 40.

5 and 6 are diagrams showing an example of a method for determining the outline shape in this modified example.

FIG. 5 shows an image of the object OB taken by the camera 40. The camera 40 is installed, for example, above the center of the screen SCR. The outline determination unit 13 extracts, for example, the contour OL of the object OB from the two-dimensional shape of the object OB shown in the image. The outline determination unit 13 detects, for example, the contour obtained by projecting the contour OL on the screen SCR along the optical axis of the camera 40 that captured the image as a pseudo contour PC. FIG. 6 is a diagram showing the relationship between the contour OL and the pseudo contour PC. The outline determination unit 13 determines, for example, a columnar body having the pseudo-contour PC as the outline of the lowest surface as the outline of the object OB.

The estimation unit 14 generates a three-dimensional model of the columnar body based on the outline shape and coordinate information of the object OB. The three-dimensional model of the pseudo-object POB, which is a columnar body, is generated by combining the information on the shape and dimensions of the pseudo-contour PC with the information on the height of the object OB. The height of the object OB is calculated as the difference between the coordinates of the contour LOL of the lowest surface of the object OB and the contour HOL of the highest surface in the height direction. The estimation unit 14 estimates the three-dimensional shape of the pseudo-object POB as the three-dimensional shape of the object OB.

The image generation unit 15 estimates the space on the screen SCR occupied by the object OB based on the three-dimensional shape of the object OB estimated by the estimation unit 14. The image generation unit 15 avoids this space and generates an image IM projected on the screen SCR. The pseudo-object POB is a columnar body having a pseudo-contour PC as the lowest surface. Therefore, the video generation unit 15 generates, for example, a video IM along the pseudo-contour PC.

For example, the image generation unit 15 estimates that the region of the contour OL where the depth information of the subject changes discontinuously between the inside (object OB side) and the outside (screen SCR side) of the contour OL is the step region LL. The step region LL has a low likelihood as the lowest surface of the object OB. Therefore, the image generation unit 15 estimates that the region of the pseudo contour PC corresponding to the step region LL is the low likelihood contour region LA. The image generation unit 15 blurs the image IM displayed along the low-likelihood contour region LA. As a method of blurring the image, a known method such as Gaussian blur is adopted.

The image generation unit 15 makes the amount of blur different depending on the likelihood, for example. The image generation unit 15 determines, for example, the likelihood of the low-likelihood contour region LA based on the amount of change in the depth information that occurs in the step region LL corresponding to the low-likelihood contour region LA. The image generation unit 15 determines, for example, that the lower the likelihood contour region LA corresponding to the step region LL in which the amount of change in depth information is large, the smaller the likelihood. The image generation unit 15 blurs the image IM along the low-likelihood contour region LA, which has a small likelihood, as much as possible.

For example, the image generation unit 15 estimates that the region of the contour OL where the depth information of the subject continuously changes between the inside and the outside of the contour OL is the non-step region HL. The non-step region HL has a high likelihood as the lowest surface of the object OB. Therefore, the image generation unit 15 estimates that the region of the pseudo contour PC corresponding to the non-step region HL is the high likelihood contour region HA. The image generation unit 15 does not blur the image IM displayed along the high-likelihood contour region HA.

FIG. 7 is a diagram showing an example of an image projected along the low-likelihood contour region LA. In FIG. 7, the image IM is blurred by adding a gradient to the brightness of the dot pattern. The upper part of FIG. 7 shows the positional relationship between the video IM and the pseudo-object POB. The lower part of FIG. 7 shows the positional relationship between the image IM and the object OB.

The back portion of the object OB facing the low-likelihood contour region LA is a blind spot of the camera. The shape of the object OB on the back surface, which is the blind spot of the camera 40, cannot be grasped from the image of the camera 40. When the image IM is projected along the pseudo-contour PC, if a deviation δ occurs between the estimated three-dimensional shape of the object OB and the three-dimensional shape of the actual object OB, the contour of the actual object OB is different. A boundary of the image IM is formed at the position (the position of the pseudo-contour PC). However, if the image is blurred, a clear boundary line cannot be formed. Therefore, the display is less likely to cause discomfort. By making the amount of blur different according to the likelihood, the feeling of discomfort is reduced.

[1-6. Modification 3]
In the present embodiment, the camera 40 for detecting the depth information of the subject is provided at one place above the screen SCR, but the number of cameras 40 is not limited to one. A plurality of cameras 40 may be installed above the screen SCR.

[2. Second Embodiment]
[2-1. Information processing system configuration]
FIG. 8 is a schematic view of the information processing system 2 of the second embodiment.

The points different from the first embodiment in this embodiment are that the class to which the object OB belongs is determined by using the object recognition technique, and the information of one or more object models defined for each class and the coordinates of the object OB. This is a point where the three-dimensional shape of the object OB is estimated by collating with the information. Hereinafter, the differences from the first embodiment will be mainly described.

The processing device 50 has, for example, a class determination unit 51 and a collation unit 52. The storage device 60 stores, for example, the object model information 61 and the analysis model 62.

The analysis model 62 is a machine learning model in which the image of the subject is used as an input layer and the class to which the subject belongs is used as an output layer. The analysis model 62 classifies subjects into classes such as PET bottles, watches, smartphones, chairs, and desks by using general object recognition technology based on deep learning.

The object model information 61 defines, for example, the characteristics of one or more object models belonging to a class for each class. For example, in the PET bottle class, a plurality of PET bottles having different three-dimensional shapes are registered as a plurality of object models. In the object model information 61, a plurality of PET bottles and the characteristics of the three-dimensional shape of each PET bottle are defined in association with each other in the PET bottle class.

The image acquisition unit 11 outputs the image acquired from the camera 40 to the class determination unit 51 and the coordinate information extraction unit 12. For example, the class determination unit 51 applies the image output from the image acquisition unit 11 to the analysis model 62, and determines the class of the object OB based on the image of the object OB. When the image may contain a plurality of objects or environmental objects that become noise, for example, the class determination unit 51 uses semantic segmentation to separate the target objects OB and classify the objects OB. conduct.

For example, the collating unit 52 collates the features of the object OB extracted from the coordinate information of the object OB with the features of one or more object models belonging to the class of the object OB, and determines the object model including the features of the object OB. .. The collation is performed using, for example, known three-dimensional features such as SHOT (Signature of Histograms of Origin States), PFH (Point Fature Histograms), and PPF (Point Pair Histogram).

The estimation unit 53 estimates the three-dimensional shape of the object OB based on, for example, the feature information of the object model including the feature of the object OB and the coordinate information of the object OB. The three-dimensional shape of the object OB is estimated by combining the dimensional information of the object OB with the shape defined based on the characteristics of the object model, for example. The dimensional information of the object OB is estimated based on, for example, the coordinate information of the object OB.

The program 63 is a program that causes a computer to execute information processing according to the present embodiment. The processing device 50 performs various processes according to the program 63 stored in the storage device 60. By executing the program 63, the processing device 50 functions as an image acquisition unit 11, a coordinate information extraction unit 12, a class determination unit 51, a collation unit 52, an estimation unit 53, and a video generation unit 15.

[2-2. Information processing method]
FIG. 9 is a flowchart showing the information processing method of the present embodiment.

In step S11, the image acquisition unit 11 acquires an image of the screen SCR taken by the camera 40. The image includes depth information of the object OB inserted on the screen SCR.

In step S12, the coordinate information extraction unit 12 extracts the coordinate information of the object OB based on the depth information of the object OB. The coordinate information extraction unit 12 extracts, for example, the coordinate information of the point cloud of the object OB as the coordinate information from the image.

In step S13, the class determination unit 51 applies the image output from the image acquisition unit 11 to the analysis model 62, and determines the class of the object OB based on the image of the object OB.

In step S14, the collating unit 52 collates the features of the object OB extracted from the coordinate information of the object OB with the features of one or more object models belonging to the class of the object OB, and creates an object model including the features of the object OB. judge.

In step S15, the estimation unit 53 estimates the three-dimensional shape of the object OB based on the feature information of the object model including the feature of the object OB and the coordinate information of the object OB.

In step S16, the image generation unit 15 estimates the space on the screen SCR occupied by the object OB based on the three-dimensional shape of the object OB estimated by the estimation unit 53. The image generation unit 15 avoids this space and generates an image IM projected on the screen SCR.

[2-3. effect]
The information processing system 2 has a class determination unit 51 and a collation unit 52. The class determination unit 51 determines the class of the object OB based on the image of the object OB inserted on the screen SCR. The storage device 60 stores the object model information 61 that defines the characteristics of one or more object models belonging to the class for each class. The collation unit 52 collates the features of the object OB extracted from the coordinate information of the object OB with the features of one or more object models belonging to the class of the object OB, and determines the object model including the features of the object OB. The estimation unit 53 estimates the three-dimensional shape of the object OB based on the feature information of the object model including the feature of the object OB and the coordinate information of the object OB.

According to this configuration, it is possible to accurately estimate the three-dimensional shape of various objects.

[3. Third Embodiment]
[3-1. Information processing system configuration]
FIG. 10 is a schematic view of the information processing system 3 of the third embodiment.

The difference from the first embodiment in this embodiment is that the three-dimensional shape of the object OB is estimated by collating the information of one or more object models with the coordinate information of the object OB. The information processing system 3 is applied to an environment in which the types of objects OB are limited, such as a play facility in which available toys are specified. Therefore, the classification using the object recognition technique as in the second embodiment is not performed. Hereinafter, the differences from the first embodiment will be mainly described.

The storage device 80 stores, for example, the object model information 81. The object model information 81 defines, for example, the features of one or more object models. For example, when a toy such as a building block is provided in a play facility, the object model information 81 defines one or more toys that can be used and the three-dimensional shape characteristics of each toy in association with each other. NS.

The processing device 70 has, for example, a collating unit 71. For example, the collating unit 71 collates the features of the object OB extracted from the coordinate information of the object OB with the features of the one or more object models, and determines one or more object models including the features of the object OB. The estimation unit 72 estimates the three-dimensional shape of the object OB based on, for example, the three-dimensional shape of one or more object models having the characteristics of the object OB.

11 and 12 are diagrams showing an example of the object model OMB defined in the object model information 81.

FIG. 11 shows a toy T in which a plurality of cube-shaped unit structure TUs are combined. The amusement facility is provided with a plurality of types of toys T according to the combination of the unit structure TY. In the object model information 81, the unit structure TU is defined as the object model OBM.

The estimation unit 72 estimates, for example, one object model including the features of the object OB as the object model OBM of the unit structure YU. The estimation unit 72 estimates, for example, the three-dimensional shape of the structure (toy T) in which a plurality of unit structures TUs are combined as the three-dimensional shape of the object OB. The number of unit structures YU and the connection structure are estimated based on, for example, the coordinate information of the object OB.

FIG. 12 shows a plurality of types of building blocks registered as an object model OBM. In the example of FIG. 12, five types of building blocks are registered as the object model OBM. For example, the object model OBM1 is a plate-shaped building block having a square plate surface. The object model OBM2 is a columnar building block. The object model OBM3 is a cube-shaped building block. The object model OBM4 is a plate-shaped building block having a rectangular plate surface. The object model OBM5 is a triangular columnar building block. In the object model information 81, five types of object model OBMs and the features of the three-dimensional shapes of the respective object model OBMs are defined in association with each other.

The estimation unit 72 identifies, for example, one or more object model OBMs having the characteristics of the object OB. In FIG. 12, for example, an object OB obtained by stacking an object model OBM3, an object model OBM2, and an object model OBM1 from the bottom in this order is shown. The estimation unit 72 determines, for example, the relative position of one or more object model OBMs based on the coordinate information of the object OB. The estimation unit 72 estimates the three-dimensional shape of the object OB based on, for example, the three-dimensional shape and the relative position of each object model OBM.

The program 82 is a program that causes a computer to execute information processing according to the present embodiment. The processing device 70 performs various processes according to the program 82 stored in the storage device 80. By executing the program 82, the processing device 70 functions as an image acquisition unit 11, a coordinate information extraction unit 12, a collation unit 71, an estimation unit 72, and a video generation unit 15.

[3-2. Information processing method]
FIG. 13 is a flowchart showing the information processing method of the present embodiment.

In step S21, the image acquisition unit 11 acquires an image of the screen SCR taken by the camera 40. The image includes depth information of the object OB inserted on the screen SCR.

In step S22, the coordinate information extraction unit 12 extracts the coordinate information of the object OB based on the depth information of the object OB. The coordinate information extraction unit 12 extracts, for example, the coordinate information of the point cloud of the object OB as the coordinate information from the image.

In step S23, the collating unit 71 collates the features of the object OB extracted from the coordinate information of the object OB with the features of the one or more object model OBM, and determines one or more object model OBM including the features of the object OB. ..

In step S24, the estimation unit 72 estimates the three-dimensional shape of the object OB based on the three-dimensional shape of one or more object model OBMs having the characteristics of the object OB.

In step S25, the image generation unit 15 estimates the space on the screen SCR occupied by the object OB based on the three-dimensional shape of the object OB estimated by the estimation unit 72. The image generation unit 15 avoids this space and generates an image IM projected on the screen SCR.

[3-3. effect]
The information processing system 3 has a collation unit 71. The storage device 80 stores the object model information 81 that defines the characteristics of one or more object model OBMs. The collation unit 71 collates the features of the object OB extracted from the coordinate information of the object OB with the features of the one or more object model OBM, and determines one or more object model OBM including the features of the object OB. The estimation unit 72 estimates the three-dimensional shape of the object OB based on the three-dimensional shape of one or more object model OBMs having the characteristics of the object OB.

According to this configuration, the three-dimensional shape of the object OB is accurately estimated based on the object model information 81.

For example, the estimation unit 72 estimates one object model OBM including the features of the object OB as the object model OBM of the unit structure TU, and three-dimensionally of the structure (toy T) in which a plurality of unit structure TUs are combined. The shape is estimated to be the three-dimensional shape of the object OB.

According to this configuration, the three-dimensional shape of a structure in which a plurality of unit structures TUs are combined can be accurately estimated. The number of object model OBMs registered in the object model information 81 can also be reduced.

It should be noted that the effects described in the present specification are merely examples and are not limited, and other effects may be obtained.

The present technology can also have the following configurations.

(1)
An estimation unit that estimates the three-dimensional shape of the object based on the image of the object on the screen,
An image generation unit that estimates the space on the screen occupied by the object based on the three-dimensional shape of the object and generates an image projected on the screen while avoiding the space.
Information processing system with.
(2)
The information processing system according to (1), wherein the image generation unit generates an image projected by the object onto an area on the screen that becomes a blind spot of a camera that has taken the image.
(3)
A coordinate information extraction unit that extracts coordinate information of the object from the image,
An outline determination unit that determines the outline of the object based on the coordinate information of the object,
Have,
The estimation unit generates a three-dimensional model of a pseudo-object that simplifies the object based on the outline shape of the object and the coordinate information, and estimates the three-dimensional shape of the pseudo-object as the three-dimensional shape of the object. The information processing system according to (1) or (2) above.
(4)
The rough shape determination unit extracts the contour of the object from the two-dimensional shape of the object reflected in the image, and projects the contour onto the screen along the optical axis of the camera that captured the image. The information processing system according to (3) above, wherein the obtained contour is detected as a pseudo contour, and a columnar body having the pseudo contour as the contour of the lowest surface is determined as a rough shape of the object.
(5)
The image contains depth information of the subject and contains
The image generation unit estimates that the contour region where the depth information of the subject changes discontinuously inside and outside the contour as a step region, and the pseudo contour region corresponding to the step region. Is estimated as a low-likelihood contour region, and the image along the low-likelihood contour region is blurred. The information processing system according to (4).
(6)
The video generation unit determines the likelihood of the low likelihood contour region based on the amount of change in the depth information that occurs in the step region corresponding to the low likelihood contour region.
The image generation unit determines that the lower likelihood contour region corresponding to the step region where the amount of change in the depth information is larger has a smaller likelihood.
The information processing system according to (5), wherein the image generation unit blurs the image along the low-likelihood contour region having a small likelihood.
(7)
A class determination unit that determines the class of the object based on the image,
A storage device that stores object model information that defines the characteristics of one or more object models belonging to the class for each class.
A coordinate information extraction unit that extracts coordinate information of the object from the image,
A collating unit that collates the features of the object extracted from the coordinate information of the object with the features of the one or more object models belonging to the class of the object and determines the object model including the features of the object.
Have,
The information processing system according to (1) or (2) above, wherein the estimation unit estimates a three-dimensional shape of the object based on the feature information of the object model including the feature of the object and the coordinate information of the object. ..
(8)
A storage device that stores object model information that defines the characteristics of one or more object models, and
A coordinate information extraction unit that extracts coordinate information of the object from the image,
A collating unit that collates the features of the object extracted from the coordinate information of the object with the features of the one or more object models and determines one or more object models including the features of the object.
Have,
The information processing system according to (1) or (2) above, wherein the estimation unit estimates the three-dimensional shape of the object based on the three-dimensional shape of one or more object models having the characteristics of the object.
(9)
The estimation unit estimates one object model including the features of the object as an object model of the unit structure, and sets the three-dimensional shape of the structure in which a plurality of the unit structures are combined as the three-dimensional shape of the object. The information processing system according to (8) above.
(10)
The three-dimensional shape of the object is estimated based on the image of the object on the screen.
Based on the three-dimensional shape of the object, the space on the screen occupied by the object is estimated, and an image projected on the screen is generated while avoiding the space.
A method of information processing performed by a computer that has.
(11)
The three-dimensional shape of the object is estimated based on the image of the object on the screen.
Based on the three-dimensional shape of the object, the space on the screen occupied by the object is estimated, and an image projected on the screen is generated while avoiding the space.
A program that makes a computer realize that.

1,2,3 Information processing system 12 Coordinate information extraction unit 13 Approximate shape determination unit 14,53,72 Estimating unit 15 Image generation unit 40 Camera 51 Class determination unit 52,71 Collation unit 60,80 Storage device 61,81 Object model Information LA Low likelihood contour area IM Video LL Step area OB Object OBM Object model OL Contour POB Pseudo object PC Pseudo contour SCR Screen TU Unit structure

Claims (11)

An estimation unit that estimates the three-dimensional shape of the object based on the image of the object on the screen,
An image generation unit that estimates the space on the screen occupied by the object based on the three-dimensional shape of the object and generates an image projected on the screen while avoiding the space.
Information processing system with. The information processing system according to claim 1, wherein the image generation unit generates an image projected by the object onto an area on the screen that becomes a blind spot of a camera that has taken the image. A coordinate information extraction unit that extracts coordinate information of the object from the image,
An outline determination unit that determines the outline of the object based on the coordinate information of the object,
Have,
The estimation unit generates a three-dimensional model of a pseudo-object that simplifies the object based on the outline shape of the object and the coordinate information, and estimates the three-dimensional shape of the pseudo-object as the three-dimensional shape of the object. The information processing system according to claim 1. The rough shape determination unit extracts the contour of the object from the two-dimensional shape of the object reflected in the image, and projects the contour onto the screen along the optical axis of the camera that captured the image. The information processing system according to claim 3, wherein the obtained contour is detected as a pseudo contour, and a columnar body having the pseudo contour as the contour of the lowest surface is determined as an outline of the object. The image contains depth information of the subject and contains
The image generation unit estimates that the contour region in which the depth information of the subject changes discontinuously between the inside and the outside of the contour as a step region, and the pseudo contour region corresponding to the step region. The information processing system according to claim 4, wherein is estimated to be a low-likelihood contour region, and the image projected along the low-likelihood contour region is blurred. The video generation unit determines the likelihood of the low likelihood contour region based on the amount of change in the depth information that occurs in the step region corresponding to the low likelihood contour region.
The image generation unit determines that the lower likelihood contour region corresponding to the step region where the amount of change in the depth information is larger has a smaller likelihood.
The information processing system according to claim 5, wherein the image generation unit blurs the image along the low-likelihood contour region having a small likelihood. A class determination unit that determines the class of the object based on the image,
A storage device that stores object model information that defines the characteristics of one or more object models belonging to the class for each class.
A coordinate information extraction unit that extracts coordinate information of the object from the image,
A collating unit that collates the features of the object extracted from the coordinate information of the object with the features of the one or more object models belonging to the class of the object and determines the object model including the features of the object.
Have,
The information processing system according to claim 1, wherein the estimation unit estimates a three-dimensional shape of the object based on the feature information of the object model including the feature of the object and the coordinate information of the object. A storage device that stores object model information that defines the characteristics of one or more object models, and
A coordinate information extraction unit that extracts coordinate information of the object from the image,
A collating unit that collates the features of the object extracted from the coordinate information of the object with the features of the one or more object models and determines one or more object models including the features of the object.
Have,
The information processing system according to claim 1, wherein the estimation unit estimates the three-dimensional shape of the object based on the three-dimensional shape of one or more object models having the characteristics of the object. The estimation unit estimates one object model including the features of the object as an object model of the unit structure, and sets the three-dimensional shape of the structure in which a plurality of the unit structures are combined as the three-dimensional shape of the object. The information processing system according to claim 8. The three-dimensional shape of the object is estimated based on the image of the object on the screen.
Based on the three-dimensional shape of the object, the space on the screen occupied by the object is estimated, and an image projected on the screen is generated while avoiding the space.
A method of information processing performed by a computer that has. The three-dimensional shape of the object is estimated based on the image of the object on the screen.
Based on the three-dimensional shape of the object, the space on the screen occupied by the object is estimated, and an image projected on the screen is generated while avoiding the space.
A program that makes a computer realize that.

Images