JP2015162708A - space sharing device, space sharing system, space sharing method, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To allow participants being in spaces different from each other to feel as if the respective participants were in an identical space, without causing deficits in the image of the participants.SOLUTION: A space sharing device 1 comprises a space acquisition unit 10, a space analysis unit 20, and a space reproduction unit 30. The space acquisition unit 10 acquires depth information and texture information at its base where the device is disposed. The space analysis unit 20 analyzes the depth information and texture information acquired by the space acquisition unit 10. The space reproduction unit 30 changes the coloration of a space image at another base on the basis of an analysis result obtained by the space analysis unit 20 at its base and that obtained by the space analysis unit 20 at another base mentioned above.

Description

  The present invention relates to a space sharing device, a space sharing system, a space sharing method, and a program.

  Conventionally, video conferencing systems, telepresence systems, and the like have been used as remote communication systems. In addition, a system has also been proposed that provides participants who are located at locations apart from each other with a sense that each participant is present in the same space (for example, the same conference room). In this system, the same physical space (for example, a conference room with the same equipment such as wallpaper, desks, chairs, etc.) is created in advance at each site, and the spatial images of each site including participants are displayed at other sites. Send to and reproduce. However, if the same physical space is created in advance at each base, the cost increases.

  Therefore, a technique has been proposed in which an image of a participant is extracted from the spatial video of each site, the extracted participant image is transmitted to another site, and is synthesized with the spatial video of the other site (for example, non-patent). Reference 1). According to this technique, it is not necessary to create the same physical space in advance at each base, so that an increase in cost can be suppressed.

Masaru Sugano, Kentaro Yamada, Satoshi Naito, “Space Sharing for Immersive Videoconferencing System”, 2012 Video Information Media Society Winter Conference, 2012

  In the technique disclosed in Non-Patent Document 1, as described above, the participant's image is extracted from the spatial video of each site. At this time, if an occlusion occurs such that the participant hides behind a desk or other participants in the spatial video, the extracted participant image is lost. When an image of a participant with a defect is combined with a spatial image of another site, this participation occurs even though there is no object in front of the participant (side closer to the camera than the participant) in the combined image. Displayed in a state where a part of the person's image is missing. For this reason, there is a possibility that other participants who are watching the synthesized video may feel uncomfortable.

  Therefore, the present invention has been made in view of the above-described problems, and each participant is present in the same space without giving a sense of incongruity to participants present in different spaces. The purpose is to provide a sense.

The present invention proposes the following matters in order to solve the above problems.
(1) The present invention provides an image of a spatial video captured in a second space (for example, corresponding to a space at another site described later) different from the first space (for example, corresponding to a space at the later site). A space sharing device to process, a space acquisition unit for acquiring information in the first space (e.g., corresponding to the space acquisition unit 10 in FIG. 1), and analyzing the information acquired by the space acquisition unit Spatial analysis means (for example, equivalent to the space analysis unit 20 in FIG. 1) and space reproduction means (for example, correcting the texture of the spatial video imaged in the second space based on the analysis result by the spatial analysis means) , Corresponding to the space reproduction unit 30 in FIG. 1).

  According to this invention, the information in the first space is acquired by the space acquisition means, the information acquired by the space acquisition means is analyzed by the space analysis means, and the analysis result by the space analysis means is analyzed by the space reproduction means. Based on the above, the texture of the spatial video shot in the second space is corrected. For this reason, while preventing the occurrence of a situation in which a part of the participant's image is missing despite the absence of an object in the foreground, the texture of the spatial image captured in the second space, It can be the same as the texture of the first space. Therefore, it is possible to provide a feeling that the participant existing in the second space exists in the same space without giving a sense of incongruity to the participant existing in the first space. .

  (2) The present invention relates to the space sharing device according to (1), in which the space reproduction unit uses a pattern in the spatial image captured in the second space as a texture of the spatial image captured in the second space. And a space sharing device that corrects at least one of colors.

  According to this invention, in the space sharing device of (1), among the patterns and colors in the spatial video imaged in the second space as the texture of the spatial video imaged in the second space by the space reproduction means. At least one of them was corrected. For this reason, at least one of the pattern and the color in the spatial image captured in the second space can be made the same as at least one of the pattern and the color of the first space.

  (3) According to the present invention, in the space sharing device of (1) or (2), the space acquisition unit acquires depth information acquisition unit that acquires depth information in the first space (for example, depth information of FIG. 3). And a texture information acquisition unit (e.g., equivalent to the texture information acquisition unit 12 of FIG. 3) that acquires texture information in the first space. Has proposed.

  According to this invention, in the space sharing device of (1) or (2), the depth acquisition unit for acquiring depth information in the first space and the texture information in the first space are acquired in the space acquisition unit. And a texture information acquisition means for acquiring. For this reason, depth information and texture information in the first space can be acquired as information in the first space by the space acquisition means.

  (4) In the space sharing apparatus according to (3), the spatial analysis unit is based on the depth information acquired by the depth information acquisition unit and the texture information acquired by the texture information acquisition unit. Point cloud information generating means for generating point cloud information having color information (for example, corresponding to the point cloud information generating unit 22 in FIG. 4), and point cloud information generated by the point cloud information generating means. Applying a predetermined surface model to the surface (for example, equivalent to a wall or floor described later) constituting the first space among the point cloud having the point cloud information generated by the point cloud information generating means. The surface specifying means (for example, corresponding to the surface specifying unit 23 in FIG. 4) for specifying the corresponding point and the point specifying information generated by the point group information generating means are specified by the surface specifying means. Food An object specifying unit (for example, the object specifying unit in FIG. 4) that clusters points to be excluded and specifies a point belonging to the object for each object (for example, equivalent to a chair or a table described later) arranged in the first space. 24), a point corresponding to the surface specified by the surface specifying unit, and a point belonging to the object specified by the object specifying unit (for example, color information extracting unit for statistically extracting color information) , Corresponding to the color information extraction unit 25 in FIG. 4), the texture information acquired by the texture information acquisition means, the information about the point corresponding to the surface specified by the surface specification means, and the object specification means A texture that divides the texture information in the first space acquired by the texture information acquisition means into a plurality of regions using information about points belonging to the object. A space sharing device is proposed, which includes a partitioning means (for example, corresponding to the texture partitioning unit 26 in FIG. 4).

  According to the present invention, in the space sharing device of (3), the space analysis means is provided with the point cloud information generation means, the surface specification means, the object specification means, the color information extraction means, and the texture division means. For this reason, the texture information in the first space can be divided into a plurality of regions in accordance with the surface constituting the first space and the object arranged in the first space.

  (5) In the space sharing device according to (4), the texture dividing unit may convert the texture information in the first space acquired by the texture information acquiring unit into a surface region and an object region. And a space sharing device that is divided into a region that is neither the surface region nor the object region.

  According to this invention, in the space sharing device of (4), the texture information in the first space acquired by the texture information acquisition means is obtained by the texture dividing means, the surface area, the object area, and the surface information. It can be divided into areas that are neither areas nor object areas.

  (6) In the space sharing device according to (4) or (5), the spatial analysis unit includes the depth information acquired by the depth information acquisition unit, and the texture information acquired by the texture information acquisition unit. And subject extraction means (for example, equivalent to the subject extraction unit 21 in FIG. 4) for extracting a subject area that is a subject image area from the texture information acquired by the texture information acquisition means. The texture dividing unit includes texture information acquired by the texture information acquiring unit, information about a point corresponding to the surface specified by the surface specifying unit, and points belonging to the object specified by the object specifying unit And the texture information acquisition means using the information about the subject area extracted by the subject extraction means The texture information in the first space obtained from the surface area, the object area, the subject area, the surface area, the object area, and the subject area, A space-sharing device that is characterized by being divided into two parts is proposed.

  According to the present invention, in the space sharing device of (4) or (5), the space analysis means is provided with subject extraction means for extracting a subject area from the texture information acquired by the texture information acquisition means. . For this reason, the texture information in the first space acquired by the texture information acquisition unit is converted into the surface area, the object area, the subject area, the surface area, the object area, and the subject area by the texture dividing means. However, it can be divided into areas that are not.

  (7) In the space sharing device according to any one of (4) to (6), the space reproduction unit may identify the surface corresponding to the point identified by the surface identification unit and the object identification unit. A space correspondence setting means (for example, corresponding to the space correspondence setting unit 31 in FIG. 5) for associating between the first space and the second space with respect to the object to which the designated point belongs, and the second space Color adjusting means for adjusting the color information of the surface or object area in the spatial image captured in step (b) using the color information of the surface or object area in the first space associated with the space correspondence setting means ( For example, a space sharing device is proposed that includes the color adjustment unit 32 of FIG.

  According to the present invention, in any one of the space sharing apparatuses (4) to (6), the space reproduction means is provided with the space correspondence setting means and the color adjustment means. Therefore, the color information of the surface or object region in the spatial image captured in the second space can be adjusted using the color information of the corresponding surface or object region in the first space.

  (8) In the space sharing device according to (7), the color information extraction unit is a color that handles the color information of the point group information generated by the point group information generation unit independently of the brightness information. Each of the surfaces to which the point specified by the surface specifying unit corresponds to each of the surfaces to which the point specified by the object specifying unit belongs, and each of the objects to which the point specified by the object specifying unit belongs , The average value and the variance value of each component of the color space, or the color histogram of each component of the color space is obtained, and the color adjusting means is a surface in a spatial image captured in the second space or The color information of the object area is obtained by using the result obtained by the color information extraction means for the surface or the object area in the first space associated by the space correspondence setting means. It proposes a space sharing apparatus characterized by adjust it.

  According to the present invention, in the space sharing device of (7), the space photographed in the second space using the average value and variance value of each component of the color space or the color histogram of each component of the color space. The color information of the surface or object area in the image can be adjusted.

  (9) The present invention corresponds to a first space sharing device arranged in a first space (for example, corresponding to the space of its own base described later) and a second space (for example, corresponding to a space of other base described later). And a second space sharing device disposed in the first space sharing device, wherein the first space sharing device acquires information in the first space (for example, , Corresponding to the space acquisition unit 10 in FIG. 1), first space analysis means for analyzing the information acquired by the first space acquisition means (for example, corresponding to the space analysis unit 20 in FIG. 1), First space reproduction means (for example, equivalent to the space reproduction unit 30 in FIG. 1) for correcting the texture of the spatial video imaged in the second space based on the analysis result by the first space analysis means; And the second space sharing device collects information in the second space. Second space acquisition means (for example, corresponding to the space acquisition unit 10 in FIG. 1) and second space analysis means (for example, the space in FIG. 1) for analyzing information acquired by the second space acquisition means. And a second space reproduction means (for example, FIG. 1) that corrects the texture of the spatial video imaged in the first space based on the analysis result by the second space analysis means. A space sharing system characterized in that the space sharing system is provided.

  According to the present invention, the same effects as described above can be obtained.

  (10) The present invention provides space acquisition means (for example, equivalent to the space acquisition unit 10 in FIG. 1), space analysis means (for example, equivalent to the space analysis unit 20 in FIG. 1), and space reproduction means (for example, FIG. 1). In the second space (e.g., equivalent to the space of another base described later) different from the first space (e.g., equivalent to the space of the own base described later). A space sharing method in a space sharing apparatus that performs image processing of a space image, wherein the space acquisition unit acquires information in the first space, and the space analysis unit includes the first A second step of analyzing the information acquired in the step; and the space reproduction means corrects the texture of the spatial video imaged in the second space based on the analysis result in the second step. 3 steps Proposes a space sharing method characterized by.

  According to the present invention, the same effects as described above can be obtained.

  (11) The present invention provides a space acquisition unit (for example, equivalent to the space acquisition unit 10 in FIG. 1), a space analysis unit (for example, the space analysis unit 20 in FIG. 1), and a space reproduction unit (for example, FIG. 1). In the second space (e.g., equivalent to the space of another base described later) different from the first space (e.g., equivalent to the space of the own base described later). A program for causing a computer to execute a space sharing method in a space sharing device that performs image processing of a space image, wherein the space acquisition unit acquires information in the first space; A second step in which the space analysis unit analyzes the information acquired in the first step, and the space reproduction unit is photographed in the second space based on the analysis result in the second step. Of spatial images We propose a program for executing the third step of correction, the computer Kusucha.

  According to the present invention, the same effect as described above can be obtained by executing the program using a computer.

  According to the present invention, it is possible to provide a sense that each participant is present in the same space without giving a sense of incongruity to participants present in different spaces.

1 is a block diagram of a space sharing system according to an embodiment of the present invention. It is a figure for demonstrating operation | movement of the space sharing system which concerns on the said embodiment. It is a block diagram of the space acquisition part with which the space sharing device in the space sharing system concerning the embodiment is provided. It is a block diagram of the space analysis part with which the space sharing device in the space sharing system concerning the embodiment is provided. It is a block diagram of the space reproduction part with which the space sharing apparatus in the space sharing system concerning the embodiment is provided.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. Note that the constituent elements in the following embodiments can be appropriately replaced with existing constituent elements, and various variations including combinations with other existing constituent elements are possible. Accordingly, the description of the following embodiments does not limit the contents of the invention described in the claims.

[Configuration and operation of space sharing system AA]
FIG. 1 is a block diagram of a space sharing system AA according to an embodiment of the present invention. The space sharing system AA includes a plurality (two in FIG. 1) of space sharing devices 1. Each of the plurality of space sharing apparatuses 1 is disposed in a different space and is provided so as to be able to communicate with each other via a communication line 100 such as the Internet or a telephone line.

  The space sharing device 1 transmits a space image of the space in which the space sharing device 1 is arranged to the other space sharing device 1 via the communication line 100, and also changes the color arrangement of the space image transmitted from the other space sharing device 1. , Change according to the space where it is located. An overview of processing performed by the space sharing device 1 will be described below with reference to FIG.

  FIG. 2 provides a feeling that the participant B11 exists in the same space without giving a sense of incongruity to the participant A11 existing at the base A by the space sharing system AA. It is a figure which shows the example which provides the feeling that participant A11 exists in the same space, without giving discomfort to participant B11 which exists in the base B.

  In the base A, the participant A11 is seated on the chair A21, and the table A31 is arranged in the vicinity of the participant A11. On the other hand, at the base B, the participant B11 is seated on the chair B21, and the table B31 is arranged in the vicinity of the participant B11. The chair A21 and the chair B21 have different shapes, sizes, and color schemes, and the table A31 and the table B31 have different shapes, sizes, and color schemes.

  In addition, at the site A, a space sharing device 1 (not shown) and display means A100 for displaying a spatial image of the site B are arranged. The space sharing device 1 arranged at the site A captures a spatial image of the site A and transmits it to the space sharing device 1 arranged at the site B. Further, the space sharing device 1 arranged at the site A changes the color scheme of the spatial video transmitted from the space sharing device 1 arranged at the site B according to the site A and displays it on the display unit A100. Let

  In the above-described color change change by the space sharing device 1 arranged at the base A, the color arrangement of the chair B21 is changed to the color arrangement of the chair A21 in the space video transmitted from the space sharing device 1 arranged at the base B. Similarly, the color scheme of the table B31 is changed in the same manner as the color scheme of the table A31. For this reason, on the display means A100, the chair B21 is displayed as the chair B61 in the same color scheme as the chair A21, and the table B31 is displayed as the table B71 in the same color scheme as the table A31. According to this, the participant A11 who is viewing the video displayed on the display means A100, the participant B11 is seated in a chair with the same color scheme as the chair A21 in which he is seated, and himself / herself It is felt that a table with a color scheme similar to that of the table A31 arranged in the vicinity of the player B11 is arranged in the vicinity of the participant B11. Therefore, it is possible to provide a feeling that the participant B11 exists in the same space without giving the participant A11 a sense of incongruity.

  In addition, at the base B, a space sharing device 1 (not shown) and display means B100 for displaying the spatial video of the base A are arranged. The space sharing device 1 arranged at the site B captures a spatial image of the site B and transmits it to the space sharing device 1 arranged at the site A. Further, the space sharing device 1 arranged at the site B changes the color scheme of the spatial video transmitted from the space sharing device 1 arranged at the site A according to the site B and displays it on the display means B100. Let

  In addition, in the change of the above-described color arrangement by the space sharing device 1 arranged at the base B, the color arrangement of the chair A21 is changed to the color arrangement of the chair B21 in the space image transmitted from the space sharing device 1 arranged at the base A. Similarly, the color scheme of the table A31 is changed in the same manner as the color scheme of the table B31. Therefore, the chair A21 is displayed as the chair A61 in the same color scheme as the chair B21, and the table A31 is displayed as the table A71 in the same color scheme as the table B31. According to this, the participant B11 who is viewing the video displayed on the display means B100 is seated on the chair having the same color scheme as the chair B21 on which the participant B11 is seated, It can be felt that a table with the same color scheme as the table B31 arranged in the vicinity of the player A11 is arranged in the vicinity of the participant A11. For this reason, it is possible to provide a feeling that the participant A11 exists in the same space without giving the participant B11 a sense of incongruity.

[Configuration of Space Sharing Device 1]
Returning to FIG. 1, the space sharing device 1 includes a space acquisition unit 10, a space analysis unit 20, a space reproduction unit 30, a transmission unit 40, and a reception unit 50.

[Configuration and Operation of Space Acquisition Unit 10]
FIG. 3 is a block diagram of the space acquisition unit 10. The space acquisition unit 10 acquires depth information and texture information in a space in which the space acquisition unit 10 is arranged. The space acquisition unit 10 is configured using a camera such as Kinect (registered trademark), for example, and includes a depth information acquisition unit 11 and a texture information acquisition unit 12.

  The depth information acquisition unit 11 is configured using, for example, a depth camera, and is arranged at a position where the participant can be photographed from almost the front of the participant. The depth information acquisition unit 11 acquires depth information in the visual field area.

  The texture information acquisition unit 12 is configured by using, for example, a color camera, is disposed at substantially the same position as the depth information acquisition unit 11, and has a visual field region substantially the same as the depth information acquisition unit 11. The texture information acquisition unit 12 acquires texture information (spatial video) in the visual field region.

[Configuration and Operation of Spatial Analysis Unit 20]
FIG. 4 is a block diagram of the space analysis unit 20. The space analysis unit 20 analyzes the depth information and texture information acquired by the space acquisition unit 10. The space analysis unit 20 includes a subject extraction unit 21, a point group information generation unit 22, a surface identification unit 23, an object identification unit 24, a color information extraction unit 25, and a texture division unit 26.

  The subject extraction unit 21 uses the depth information and texture information acquired by the space acquisition unit 10 to extract a subject region that is a region of the participant's image from the spatial video of the space in which the subject extraction unit 21 is located. . For extracting the subject area, for example, “Kentaro Yamada, Masaru Kanno, Satoshi Naito, Hiromasa Yanagihara,“ Extracting Person Area by Adaptive Reference of Distance and Image Information for Telepresence ”, 2013 Annual Meeting of the Institute of Image Electronics Engineers of Japan. It is possible to use the method shown in the next competition, 2013.

  The point cloud information generation unit 22 includes depth information and measurement conditions acquired by the depth information acquisition unit 11, texture information acquired by the texture information acquisition unit 12, and a subject area extracted by the subject extraction unit 21. Based on this, point cloud information having color information is generated.

  Specifically, the point cloud information generation unit 22 first removes the subject area extracted by the subject extraction unit 21 from the texture information acquired by the texture information acquisition unit 12. Next, the depth information d (x, y) as the mapping on the coordinates (x, y) in the texture information after removing the subject area using the measurement conditions described above, the coordinates of the corresponding three-dimensional real space Convert to values (X, Y, Z). Next, color group information is added to each coordinate value of the converted real space to generate point cloud information.

  The coordinate values x and y in the spatial image are generally pixel positions and are given as integers, but the values X, Y, and Z in the point group information are generally real numbers. In addition, in order to add color information as described above, it is assumed that the correspondence relationship between the depth information acquisition unit 11 and the texture information acquisition unit 12 is calibrated in advance using a calibration tool or the like. In addition, the measurement condition means which direction each coordinate (x, y) on the spatial image acquired by the texture information acquisition unit 12 is viewed from the depth information acquisition unit 11. In the case where the acquisition unit 11 is configured using a depth camera, it refers to parameters such as the angle of view and the focal length of the depth camera.

  The surface specifying unit 23 specifies a point corresponding to a surface (for example, a wall or a floor) constituting the space from the point group having the point group information generated by the point group information generating unit 22. According to this, out of the point cloud having the point cloud information generated by the point cloud information generating unit 22, those excluding the points corresponding to the surface specified by the surface specifying unit 23 are arranged in the above-described space. Candidate points belong to an object (for example, a chair or a table).

  By giving a predetermined surface model, the surface can be specified as a surface having an arbitrary shape such as a curved surface. For example, when the surface shape is modeled as a plane, points belonging to each plane can be estimated based on the RANSAC model. Therefore, when the surface specifying unit 23 specifies a point corresponding to the surface among the point groups having the point group information generated by the point group information generating unit 22, the points generated by the point group information generating unit 22 are used. A predetermined surface model is applied to the group information.

  When estimating points belonging to each plane based on the RANSAC model, first, as a first process, points are randomly acquired from the point cloud having the point cloud information generated by the point cloud information generation unit 22. A plane is formed and the distance to the plane is obtained for all other points. Next, as a second process, a point where the obtained distance is equal to or greater than a predetermined threshold is set as an outlier. Next, the first process and the second process described above are repeated so that the distances between all points except for outliers and the plane are less than the threshold value described above. According to this, a final plane is obtained, and a point that does not become an outlier with respect to the obtained plane can be specified as a point belonging to this plane.

  In addition, after removing what was specified as the point which belongs to a plane as mentioned above from the point cloud which has the point cloud information produced | generated by the point cloud information production | generation part 22, the above-mentioned 1st process and 2nd By repeating this process again, a plurality of final planes may be obtained, and points belonging to the obtained planes may be specified. In this case, for example, the number of point groups remaining after removing the points extracted as the points belonging to the plane as described above by repeating the first process and the second process described above is equal to or less than a predetermined number. Or when the number of planes finally obtained reaches a predetermined number.

  The object specifying unit 24 specifies a point belonging to each object for a point group having the point group information generated by the point group information generating unit 22 excluding the point specified by the surface specifying unit 23. When specifying the points belonging to each object, for example, clustering based on the Euclidean distance is performed on the point group information generated by the point group information generation unit 22 excluding the points specified by the surface specification unit 23. Go and group. Specifically, the maximum Euclidean distance to be integrated as one object is specified, and a point where the Euclidean distance is equal to or less than a specified value is set as a point belonging to the same object. According to this, points excluding the points specified by the surface specifying unit 23 among the point groups having the point group information generated by the point group information generating unit 22 belong to each object, and the surface is also an object. It is classified as a point that does not belong.

  The color information extraction unit 25 statistically extracts color information from the points corresponding to the surface specified by the surface specification unit 23 and the points belonging to each object specified by the object specification unit 24. Extraction of color information is performed for each surface or object. For example, in the case of a YCbCr color space, an average value and a dispersion value of Y, Cb, and Cr values, and a color histogram of Y, Cb, and Cr are used as color information. Desired.

  The texture dividing unit 26 includes the texture information acquired by the texture information acquiring unit 12, information about the subject area extracted by the subject extracting unit 21, and information about points corresponding to the surface specified by the surface specifying unit 23. And a spatial image acquired by the texture information acquisition unit 12 using the information about the points belonging to each object specified by the object specifying unit 24, a subject area, a surface area, and an object area , And other areas.

  Note that the subject area extracted by the subject extraction unit 21 is applied as it is to the subject area divided from the spatial video. On the other hand, for the surface area, a point corresponding to the surface specified by the surface specifying unit 23 is projected onto the spatial image acquired by the texture information acquiring unit 12, and the projected position is used as a seed to generate an image area using, for example, the GrabCut algorithm. Segmentation is performed to extract the surface area. As for the object region, the points belonging to each object specified by the object specifying unit 24 are projected onto the spatial video acquired by the texture information acquisition unit 12, and the projected position is used as a seed to generate an image region using, for example, the GrabCut algorithm. Segmentation is performed to extract an object region. For other regions, a region that does not belong to any of the subject region, the surface region, and the object region in the spatial video acquired by the texture information acquisition unit 12 is applied.

[Operation of Transmitter 40]
The transmission unit 40 transmits a spatial image acquired by the texture information acquisition unit 12 of the site (local site) where the transmission unit 40 is located and an analysis result by the spatial analysis unit 20 of the local site different from the local site ( It is transmitted to the receiving unit 50 at another base. The analysis result by the spatial analysis unit 20 at the local site belongs to the point cloud information about the point corresponding to the surface identified by the surface identification unit 23 at the local site and the object identified by the object identification unit 24 at the local site. The point group information about the point, the color information extracted by the color information extraction unit 25 at the local site, and information related to the division result by the texture division unit 26 at the local site are included.

[Operation of Receiving Unit 50]
The receiving unit 50 receives the spatial video acquired by the texture information acquisition unit 12 at the other site and the analysis result by the spatial analysis unit 20 at the other site. The analysis result by the space analysis unit 20 at the other site belongs to the point group information about the point corresponding to the surface specified by the surface specifying unit 23 at the other site and the object specified by the object specifying unit 24 at the other site. The point group information about the point, the color information extracted by the color information extraction unit 25 at the other site, and the information regarding the division result by the texture division unit 26 at the other site are included.

[Configuration and operation of space reproduction unit 30]
FIG. 5 is a block diagram of the space reproduction unit 30. The space reproduction unit 30 changes the color scheme of the spatial video at the other site based on the analysis result by the space analysis unit 20 at the own site and the analysis result by the space analysis unit 20 at the other site. The space reproduction unit 30 includes a space correspondence setting unit 31 and a color adjustment unit 32.

  The space correspondence setting unit 31 includes the color information extracted by the color information extraction unit 25 at the local site, the color information extracted by the color information extraction unit 25 at the other site, the surface identification unit 23 and the object identification unit at the local site. 24 using each of the point group information about the points specified by each of the points 24 and the point group information about the points specified by each of the surface specifying unit 23 and the object specifying unit 24 of the other bases. Set the correspondence of the surface and each object.

  Setting the correspondence between each surface and each object at each base is, for example, associating a point cloud corresponding to the floor of its own base with a point group corresponding to the floor of another base, Corresponding point groups are associated with point groups corresponding to chairs at other locations. With these corresponding settings, the distance (difference) between the three-dimensional shapes formed by the points belonging to each surface and each object, the distance between the color information of each surface and each object (average / dispersion of luminance Y, or , The distance of the histogram) is obtained, and the surfaces or objects having a small distance (that is, similar) are associated with each other. The distance between the three-dimensional shapes can be obtained by using, for example, a method for calculating the distance between the feature amounts representing the three-dimensional shape features.

  In the above-mentioned correspondence setting, when the surfaces and objects are associated one-to-one, when the surfaces and objects are associated one-to-many, There are cases where there are surfaces or objects of other bases that do not correspond. For example, when the distance between the above-described three-dimensional shapes is equal to or greater than a predetermined threshold, it may be determined that the surface or object at the other site does not correspond to any surface or object at the own site. In addition, for example, information on the surface and objects at the local site and other sites may be presented to the user, and the user may perform the above-described setting.

  The color adjustment unit 32 extracts the color information extracted by the color information extraction unit 25 of the local site, the correspondence information set by the spatial correspondence setting unit 31 of the local site, and the color information extraction unit 25 of the other site. Color adjustment processing is performed for each area on the spatial video acquired by the texture information acquisition unit 12 at the other site using the color information thus obtained and the result of division by the texture division unit at the other site. Specifically, among the regions divided by the texture dividing unit 26 at the other site as the surface region or the object region, the region associated with the surface or the object at the own site by the space correspondence setting unit 31 at the own site Then, the color adjustment of the spatial video acquired by the texture information acquisition unit 12 at another base is performed.

  The color adjustment process described above will be described below by taking as an example a case where the object P at the local site and the object Q at the other site are associated with each other by the space correspondence setting unit 31 at the local site. In this case, the color of the object Q is brought close to the object P in the spatial image acquired by the texture information acquisition unit 12 at another base. Specifically, first, the difference between the average brightness value of the area of the object P and the average brightness value of the area of the object Q is obtained, and the obtained difference is added to the brightness value of each pixel constituting the area of the object Q. To do. Next, a value is randomly generated using the average and variance of the color differences in the area of the object P, and the color difference values (Cb, Cr) of each pixel constituting the area of the object Q are replaced with the generated values.

  It should be noted that the above-described color adjustment processing is not performed on the subject region or the region divided as the other region by the texture dividing unit 26 at another base. Of the areas divided by the texture dividing unit 26 at the other site as the surface area or the object area, the space correspondence setting unit 31 at the own site has determined that it does not correspond to any surface or object at the own site. The above-described color adjustment process is not performed for the area.

  According to the above space sharing system AA, the following effects can be produced.

  The space sharing system AA can provide a feeling that the participant B11 exists in the same space without giving the participant A11 a sense of incongruity.

  Further, the space sharing system AA can provide a feeling that the participant A11 exists in the same space without giving the participant B11 a sense of incongruity.

  The processing of the space sharing apparatus 1 of the present invention is recorded on a computer-readable non-transitory recording medium, and the program recorded on the recording medium is read into the space sharing apparatus 1 and executed, thereby The invention can be realized.

  Here, for example, a nonvolatile memory such as an EPROM or a flash memory, a magnetic disk such as a hard disk, a CD-ROM, or the like can be applied to the above-described recording medium. Further, reading and execution of the program recorded on the recording medium is performed by a processor provided in the space sharing apparatus 1.

  Further, the above-described program may be transmitted from the space sharing apparatus 1 storing the program in a storage device or the like to another computer system via a transmission medium or by a transmission wave in the transmission medium. Here, the “transmission medium” for transmitting the program refers to a medium having a function of transmitting information, such as a network (communication network) such as the Internet or a communication line (communication line) such as a telephone line.

  Further, the above-described program may be for realizing a part of the above-described function. Furthermore, what can implement | achieve the above-mentioned function in combination with the program already recorded on the space sharing apparatus 1, and what is called a difference file (difference program) may be sufficient.

  The embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to this embodiment, and includes a design that does not depart from the gist of the present invention.

  For example, in the above-described embodiment, only the color scheme of the chair and the table is changed as described above with reference to FIG. However, the present invention is not limited to this, and for example, the color scheme of walls and floors may be changed. Moreover, a pattern may be changed about a chair, a table, a wall, or a floor, and a pattern and a color scheme may be changed.

  In the above-described embodiment, YCbCr is used as the color space. However, the present invention is not limited to this, and any color space that handles brightness information such as YUV and HSV independently can be used.

  Further, in the above-described embodiment, when the color information included in the point cloud information is not represented in a color space that handles the above-described brightness information independently (for example, an RGB color space), color information extraction is performed. Before the color information is statistically extracted by the unit 25, the color space of the color information is converted into a color space that handles brightness information independently.

AA ... Space sharing system 1 ... Space sharing device 10 ... Space acquisition unit 11 ... Depth information acquisition unit 12 ... Texture information acquisition unit 20 ... Spatial analysis unit 21 ... Subject extraction Unit 22 ... Point cloud information generation unit 23 ... Surface identification unit 24 ... Object identification unit 25 ... Color information extraction unit 26 ... Texture division unit 30 ... Spatial reproduction unit 31 ... Spatial correspondence setting unit 32 ... color adjustment unit 40 ... transmission unit 50 ... reception unit

Claims (11)

A space sharing device that performs image processing on a spatial video captured in a second space different from the first space,
Space acquisition means for acquiring information in the first space;
Space analysis means for analyzing the information acquired by the space acquisition means;
A space sharing device, comprising: a space reproduction unit that corrects a texture of a spatial image captured in the second space based on an analysis result by the space analysis unit.   The space reproduction unit corrects at least one of a pattern and a color in a spatial image captured in the second space as a texture of the spatial image captured in the second space. Item 2. The space sharing device according to Item 1. The space acquisition means includes
Depth information acquisition means for acquiring depth information in the first space;
Texture information acquisition means for acquiring texture information in the first space;
The space sharing apparatus according to claim 1, further comprising: The spatial analysis means includes
Point cloud information generating means for generating point cloud information having color information based on the depth information acquired by the depth information acquiring means and the texture information acquired by the texture information acquiring means;
By applying a predetermined surface model to the point cloud information generated by the point cloud information generating means, the first space among the point clouds having the point cloud information generated by the point cloud information generating means. A surface specifying means for specifying a point corresponding to the surface constituting
Clustering the point cloud having the point cloud information generated by the point cloud information generating means excluding the point cloud specified by the surface specifying means, the object for each object placed in the first space An object specifying means for specifying a point belonging to
Color information extracting means for statistically extracting color information from a point corresponding to the surface specified by the surface specifying means and a point belonging to the object specified by the object specifying means;
Using the texture information acquired by the texture information acquisition means, information about points corresponding to the surface specified by the surface specification means, and information about points belonging to the object specified by the object specification means, The space sharing apparatus according to claim 3, further comprising: a texture dividing unit that divides the texture information in the first space acquired by the texture information acquiring unit into a plurality of regions.   The texture dividing means includes texture information in the first space acquired by the texture information acquiring means, a surface area, an object area, an area that is neither the surface area nor the object area, The space sharing apparatus according to claim 4, wherein the space sharing apparatus is divided into two. The space analysis means uses the depth information acquired by the depth information acquisition means and the texture information acquired by the texture information acquisition means, from among the texture information acquired by the texture information acquisition means. A subject extracting means for extracting a subject area that is a subject image area;
The texture dividing unit includes the texture information acquired by the texture information acquiring unit, the information about the point corresponding to the surface specified by the surface specifying unit, and the point belonging to the object specified by the object specifying unit. Using the information and the subject area extracted by the subject extracting means, the texture information in the first space acquired by the texture information acquiring means, the surface area, and the object area, 6. The space sharing apparatus according to claim 4, wherein the subject region is divided into a region that is neither the surface region, the object region, nor the subject region. The space reproduction means includes
Space correspondence setting means for associating between the first space and the second space the surface to which the point specified by the surface specifying means corresponds and the object to which the point specified by the object specifying means belongs When,
The color information of the surface or object region in the spatial image captured in the second space is adjusted using the color information of the surface or object region in the first space associated by the space correspondence setting means. The space sharing apparatus according to claim 4, further comprising a color adjusting unit configured to perform color adjustment. The color information extraction unit converts the color information of the point group information generated by the point group information generation unit into a color space that handles brightness information independently, and the point specified by the surface specification unit corresponds The average value and variance value of each component of the color space or the color histogram of each component of the color space is obtained for each of the surfaces to be processed and each of the objects to which the points specified by the object specifying means belong. ,
The color adjustment unit is configured to display color information of a surface or an object region in a spatial image captured in the second space with respect to the surface or object region in the first space associated with the space correspondence setting unit. The space sharing apparatus according to claim 7, wherein adjustment is performed using a result obtained by the color information extraction unit. A space sharing system comprising: a first space sharing device arranged in a first space; and a second space sharing device arranged in a second space,
The first space sharing device includes:
First space acquisition means for acquiring information in the first space;
First space analysis means for analyzing the information acquired by the first space acquisition means;
First spatial reproduction means for correcting the texture of the spatial video imaged in the second space based on the analysis result by the first spatial analysis means,
The second space sharing device includes:
Second space acquisition means for acquiring information in the second space;
Second space analysis means for analyzing the information acquired by the second space acquisition means;
A space sharing system, comprising: second space reproduction means for correcting a texture of a spatial video imaged in the first space based on an analysis result by the second space analysis means. A space sharing method in a space sharing apparatus that includes a space acquisition unit, a space analysis unit, and a space reproduction unit, and performs image processing on a spatial video captured in a second space different from the first space,
A first step in which the space acquisition means acquires information in the first space;
A second step in which the spatial analysis means analyzes the information acquired in the first step;
And a third step of correcting the texture of the spatial image captured in the second space based on the analysis result of the second step. . In order to cause a computer to execute a space sharing method in a space sharing apparatus that includes a space acquisition unit, a space analysis unit, and a space reproduction unit and performs image processing on a spatial video imaged in a second space different from the first space The program of
A first step in which the space acquisition means acquires information in the first space;
A second step in which the spatial analysis means analyzes the information acquired in the first step;
A program for causing the computer to execute a third step in which the space reproduction means corrects a texture of a spatial video imaged in the second space based on the analysis result in the second step.

Images