WO2021075307A1

WO2021075307A1 - Information processing device, information processing method, and information processing program

Info

Publication number: WO2021075307A1
Application number: PCT/JP2020/037806
Authority: WO
Inventors: 浩一川崎
Original assignee: ソニー株式会社
Priority date: 2019-10-15
Filing date: 2020-10-06
Publication date: 2021-04-22

Abstract

This information processing device comprises: an information acquisition unit that acquires, from an image, information within an image-capture range; a plotting unit that plots the information onto the surface of a virtual three-dimensional object; and a disposing unit that disposes the three-dimensional object at a disposition target.

Description

Information processing equipment, information processing methods and information processing programs

This technology relates to information processing devices, information processing methods, and information processing programs.

In recent years, technologies called augmented reality (AR) and mixed reality (MR), which superimpose additional information on the real space and present it to the user, have been attracting attention. In addition, autonomous mobile bodies that move autonomously, such as cleaning robots and pet-type robots, are also attracting attention.

In such AR technology, MR technology, autonomous mobile technology, etc., it is necessary to recognize the three-dimensional structure of real space and objects in real space, and as a recognition method, for example, SLAM (Simultaneous Localization And Mapping) There is a technique called. SLAM is a technology that simultaneously estimates the self-position and creates an environmental map from information acquired from various sensors. For example, using the feature points detected from the input image, the position of the feature points and the position of the camera in the environment. And posture recognition are performed at the same time.

The accuracy of the environment recognition technology based on the feature point cloud in the image depends on the distribution of the feature points in the image. The larger the number of feature points, the higher the accuracy, and the smaller the number of feature points, the lower the accuracy. .. However, the user who uses AR or the like does not pay attention to the characteristics of such an environment recognition technology, and depending on the usage mode of the user, the user does not recognize the environment by using it in a direction unfavorable for the environment recognition technology. It becomes stable and may interfere with its use. Therefore, a mechanism has been proposed that can avoid such a situation when providing it to a user such as AR (Patent Document 1).

Japanese Unexamined Patent Publication No. 2013-225245

In SLAM, if the self-position that can be acquired includes an error, there is also a problem that the error is accumulated and gradually deviates from the real space. To solve this problem, it is necessary to reduce the error or reset the error. In order to do so, it is necessary to grasp in advance information about the environment that the accuracy of detecting feature points in the real space is low.

This technology was made in view of these points, and aims to provide an information processing device, an information processing method, and an information processing program that can easily grasp information about the environment.

In order to solve the above-mentioned problems, the first technique is to arrange an information acquisition unit that acquires information within the shooting range from an image, a plot processing unit that plots information on the surface of a virtual three-dimensional object, and a virtual three-dimensional object. It is an information processing device including an arrangement processing unit to be arranged on a target.

The second technique is an information processing method in which information within the shooting range is acquired from an image, the information is plotted on the surface of a virtual three-dimensional object, and the virtual three-dimensional object is placed as an arrangement target.

Further, the third technology is an information processing program that acquires information within the shooting range from an image, plots the information on the surface of a virtual three-dimensional object, and causes a computer to execute an information processing method for arranging the virtual three-dimensional object on the placement target. Is.

It is a block diagram which shows the structure of the terminal apparatus 100. It is a block diagram which shows the structure of an information processing apparatus 200. FIG. 3A is an explanatory diagram of the score plot on the sphere, and FIG. 3B is an explanatory diagram of the arrangement of the sphere. It is a flowchart which shows the score acquisition process. It is a flowchart which shows the plot processing to a sphere and the arrangement process of a sphere. It is explanatory drawing of the calculation of a score. It is explanatory drawing of the calculation of a score. It is explanatory drawing of the estimation of a score. It is explanatory drawing of the process of determining the arrangement position of a sphere. It is explanatory drawing of the process of determining the arrangement position of a sphere. It is explanatory drawing of the plot of the score on a sphere. It is explanatory drawing of the plot of the score on a sphere. It is explanatory drawing of the score calculation on the whole surface to a sphere. It is explanatory drawing of the score calculation on the whole surface to a sphere. It is explanatory drawing of the plot of the score on a sphere. It is a figure which shows the specific example of the arrangement of a sphere. It is a figure which shows the specific example of the arrangement of a sphere. FIG. 18A is an explanatory diagram of an example showing a score value on a sphere, and FIG. 18B is an explanatory diagram of score complementation. It is explanatory drawing of the presentation of a recommended route.

Hereinafter, embodiments of the present technology will be described with reference to the drawings. The explanation will be given in the following order.

<1. Embodiment>
[1-1. Configuration of terminal device 100]
[1-2. Configuration of information processing device 200]
[1-3. Processing by information processing device 200]
<2. Modification example>

<1. Embodiment>
[1-1. Configuration of terminal device 100]
First, the configuration of the terminal device 100 and the information processing device 200 in the present embodiment will be described. The terminal device 100 captures an image for calculating a score by the information processing device 200, displays information calculated by the information processing device 200, and the like. As shown in FIG. 1, the terminal device 100 includes a control unit 101, a camera unit 102, a sensor unit 103, an input unit 104, a display unit 105, a communication unit 106, a storage unit 107, and an information processing device 200. ..

The control unit 101 is composed of a CPU (Central Processing Unit), a RAM (Random Access Memory), a ROM (Read Only Memory), and the like. The ROM stores a program or the like that is read and operated by the CPU. The RAM is used as the work memory of the CPU. The CPU controls the entire terminal device 100 and each part by executing various processes according to the program stored in the ROM and issuing commands.

The camera unit 102 includes an image sensor, an image processing engine, and the like, and has a function as a camera capable of capturing RGB or monochrome two-dimensional still images and moving images. The camera unit 102 is provided in the terminal device 100 itself, or may be a digital camera, a single-lens reflex camera, or the like which is separate from the terminal device 100 and can communicate with the terminal device 100, or may be attached to the head or clothes. It may be a so-called action camera that can shoot.

The sensor unit 103 is a sensor that can detect the position of a GPS (Global Positioning System) module or the like. GPS is a system that knows the current position by receiving a signal from an artificial satellite located around the earth with a receiver.

In addition to GPS, the sensor unit 103 may include a sensor capable of measuring a distance such as LiDAR (Laser Imaging Detection and Ringing). LiDAR measures scattered light with respect to laser irradiation that emits pulsed light, and analyzes the distance to an object at a long distance and the properties of the object.

Further, the sensor unit 103 may include a sensor that detects acceleration and angular velocity, such as an IMU (Inertial Measurement Unit) module. The IMU module is an inertial measurement unit that detects the attitude, tilt, etc. of the information processing device 200 by obtaining three-dimensional angular velocity and acceleration with an acceleration sensor, angular velocity sensor, gyro sensor, etc. in two or three axial directions. To do.

In the present embodiment, when the user takes a picture with the camera unit 102, the sensor unit 103 detects the shooting position and the shooting posture and supplies the information processing device 200 to the information processing device 200.

The input unit 104 is for the user to input for the operation of the terminal device 100. When an input is made to the input unit 104 by the user, a control signal corresponding to the input is generated and supplied to the control unit 101. Then, the control unit 101 performs various processes corresponding to the control signal. The input unit 104 includes a touch screen integrally configured with the display which is the display unit 105, a physical button, a voice input function by voice recognition, and the like. The input unit 104 also includes a release button for taking an image.

The display unit 105 is a display device such as a display that displays an image / video, a GUI (Graphical User Interface), a through image at the time of shooting by the camera unit 102, information processed by the information processing device 200, and the like.

The communication unit 106 is a communication module for transmitting / receiving data to / from an external server or the like. Communication is wireless LAN (Local Area Network), WAN (Wide Area Network), WiFi (Wireless Fidelity), 4G (4th generation mobile communication system), 5G (5th generation mobile communication system), Bluetooth (registered trademark), It is performed by wireless communication such as ZigBee (registered trademark). Further, the communication with the image pickup apparatus 400 may be a wired communication such as a USB (Universal Serial Bus) communication in addition to the wireless communication.

The storage unit 107 is a storage medium composed of, for example, an HDD (Hard Disc Drive), a semiconductor memory, an SSD (solid state drive), and the like, and the data, score, image shooting position, and shooting posture detected by the sensor unit 103. Information necessary for processing by the information processing apparatus 200 is stored.

[1-2. Configuration of information processing device 200]
Next, the configuration of the information processing apparatus 200 will be described with reference to FIG. The information processing device 200 includes an information acquisition unit 201, a plot processing unit 202, and an arrangement processing unit 203.

The information acquisition unit 201 acquires information about the photographing range from the image captured by the camera unit 102 of the terminal device 100. In this embodiment, the information has a score and a reliability. Details of the score and reliability will be described later. In addition, the information acquisition unit 201 also performs a process of estimating a score outside the shooting range of the image using the calculated score. The image may be a still image or a frame image constituting a moving image. Further, the image may be an image taken by another camera and supplied to the information processing apparatus 200 in addition to the image taken by the camera unit 102.

The plot processing unit 202 performs a process of plotting the score and reliability calculated and estimated by the information acquisition unit 201 on a sphere which is a virtual three-dimensional object.

The placement processing unit 203 performs a process of arranging a sphere on which a score and a reliability are plotted as an arrangement target. The placement target is spatial data, and includes, for example, two-dimensional or three-dimensional map data, images and videos of the space, and virtual space in AR and MR. The details of the arrangement of the spheres will be described later.

The information processing device 200 is configured as described above. The information processing device 200 may operate in the terminal device 100 as in the present embodiment, or may operate in another device different from the terminal device 100, another external server, or the like. The information processing device 200 is realized by executing a program, and the program may be installed in the terminal device 100 or the like in advance, or may be distributed by download, storage medium, or the like so that the user can install it by himself / herself. Good. Further, the information processing device 200 is not only realized by a program, but may also be realized by combining a dedicated device, a circuit, or the like by hardware having the function.

[1-3. Processing by information processing device 200]
Next, the processing by the information processing apparatus 200 will be described. First, the score and reliability as information in the present embodiment, the plot of the score and reliability on the sphere as a virtual three-dimensional object, and the outline of the arrangement of the sphere will be described.

The score is an index showing the characteristics of the environment within the shooting range of the image, which is determined in a complex manner from the number, position, and amount of feature points that can be acquired from the image. The score can represent, for example, the ease and accuracy of SLAM self-position estimation and environmental map creation.

Since the score is calculated as a value from the feature points within the shooting range detected from the image, if the shooting direction (viewpoint) is different and the feature points in the shooting range are different even at the same position, The score will also be different. The score is calculated so that one score corresponds to one combination of the shooting position and the shooting posture.

Reliability is an index showing the reliability of the score. The score calculated directly from the image has a reliability of 100%, and the score estimated using other scores has a reliability of 100% or less.

In the present embodiment, as shown in FIG. 3A, the score is plotted on the surface of the sphere as a virtual three-dimensional object using the hue of the HSV color space. 3A-a in FIG. 3A is a front view of the sphere, FIG. 3A-b is a top view of the sphere, and FIG. 3A-c is a side view of the sphere. Confidence is also plotted on the surface of the sphere using saturation or lightness.

As shown in FIG. 3B, for example, this sphere is arranged at a position corresponding to the position where the image used for calculating the score was taken on the two-dimensional map data. Then, it is displayed on the display unit 105 or the like of the terminal device 100. This allows the user to easily grasp the score and reliability. The present technology creates a map (score map) having information acquired from an image by arranging a sphere on which a score is plotted in this way as an arrangement target.

Next, specific processing by the information processing device 200 will be described. FIG. 4 is a flowchart showing a score acquisition process performed by the information acquisition unit 201. Further, FIG. 5 is a flowchart showing a plot process by the score plot processing unit 202 and a sphere arrangement process by the arrangement processing unit 203.

In the score acquisition process, first, an image captured in the range for calculating the score captured by the camera unit 102 of the terminal device 100 in step S101 is input. Next, the shooting position and shooting posture when the image is shot in step S102 are estimated. The shooting position and shooting posture can be estimated from the shooting position information and the shooting posture information which are the sensor information acquired by the GPS, LiDAR, IMU module, etc. as the sensor unit 103.

Next, in step S103, the score is calculated from the image. Here, an example of the score calculation method will be described with reference to FIGS. 6 and 7.

First, as shown in FIG. 6A, the image is divided into 16 squares by dividing the image into L layers. The number of squares is only one example, and the present technology is not limited to this. Next, the weight Wl is calculated for each layer l. The weight Wl can be calculated by the following equation 1.

[Equation 1]
Wl = 2 ^l

Next, the feature points are detected for each layer l, and the number Nl of the cells containing the feature points is counted. As a method of detecting feature points, for example, there is a method called a luminance gradient. In the luminance gradient, first, attention is paid to one pixel in the image, and when the luminance gradient with the adjacent pixel appears by a predetermined amount or more, that one pixel is detected as a feature point. The larger the gradient of brightness, the larger the feature amount of the feature point. This process is performed on all the pixels that make up the image.

Then, calculate the score by the following formula 2. In Equation 2, the score is represented as S.

[Equation 2]

As a result, for example, in the example shown in FIG. 6B, the score is calculated as 66. Further, in the example of FIG. 7A, the score is calculated as 80, in the example of FIG. 7B, the score is calculated as 146, and in the example of FIG. 7C, the score is calculated as 200. The rectangles in FIGS. 6B and 7A to 7C indicate an image, and the points in the rectangle represent the feature points detected in the image.

Feature points exist in various places in the image. However, if the subject in the image does not have feature points such as a white wall, or if it is difficult to detect the feature points, the score of the image as a whole becomes low.

Return to the explanation of the flowchart in FIG. Next, in step S104, the score is estimated and the reliability of the estimated score is calculated. The score is estimated when it is necessary to plot the score corresponding to the position other than the shooting position where the score was calculated or the posture other than the shooting posture on the sphere. Therefore, score estimation is not an essential process. It is preferable that the user can decide whether or not to estimate the score.

First, the first method of estimating the score and calculating the reliability will be described with reference to FIG. 8A. The solid rectangular frame in FIG. 8A indicates the shooting range (field of view) of the image shot by the camera unit 102, and the score can be calculated from the image for this shooting range by the method described above. The reliability of the score calculated from the captured image is 100%. The score is estimated using the score calculated from this image.

The broken line rectangular frame in FIG. 8A indicates the first estimation range for estimating the score. Further, the alternate long and short dash line rectangular frame in FIG. 8A indicates a second estimation range for estimating the score. In this first method, as shown in FIG. 8A, the estimated range for estimating the score needs to partially overlap with the shooting range of the image for which the score has been calculated.

In the first method, the score of the first estimated range that overlaps with the shooting range is estimated to be the same as the score of the shooting range. Therefore, for example, when the score of the shooting range is calculated to be 100, the score of the first estimated range is estimated to be 100. It is estimated that the score of the second estimated range, which partially overlaps the shooting range, is also the same value as the score of the shooting range.

Then, the reliability of the score of the first estimated range is calculated from the ratio of the solid angle where the shooting range and the first estimated range overlap. For example, when the ratio of the solid angles overlapping the shooting range and the first estimated range is 20%, the reliability of the score in the first estimated range is 20%.

Similarly, the reliability of the score of the second estimated range is calculated from the ratio of the solid angle where the shooting range and the second estimated range overlap. For example, when the ratio of the solid angles overlapping in the shooting range and the second estimation range is 40%, the reliability of the score in the second estimation range is 40%.

Next, the second method of estimating the score and calculating the reliability will be described with reference to FIG. 8B.

The solid line on the left side in FIG. 8B shows the shooting range (field of view) of the image shot by the camera unit 102, and the score can be calculated from the image for this shooting range. The points in FIG. 8B show the feature points detected from the image for score calculation. In the example of FIG. 8B, it is assumed that eight feature points (black points and white points) are detected from the image. The feature points are detected as three-dimensional coordinates.

Since the score can be calculated from the image in the shooting range, the reliability of the score is 100%. The score is estimated using the score calculated from this image.

The broken line on the right side in FIG. 8B shows the estimated range (field of view) from the position / posture at which the score is estimated. In the second method, the score of the estimated range in which the shooting range and a part of the field of view overlap is estimated to be the same as the score of the shooting range. Therefore, for example, when the score of the shooting range in FIG. 8B is calculated to be 100, the score of the estimated range of the broken line where the shooting range and a part of the field of view overlap is estimated to be 100. Then, the reliability of the score in the estimation range can be calculated by the following equation 3 using the number of feature points.

[Equation 3]
(Number of feature points common to the shooting range in the estimated range / Number of feature points in the shooting range) x 100

In the example of FIG. 8B, eight feature points (black spots and white spots) are detected in the shooting range, six feature points (black spots) are detected in the estimated range, and the six feature points are shot. Since it is common to the feature points in the field of view of the range, the reliability can be calculated to be 75%.

Return to the explanation of the flowchart in FIG. Next, in step S105, the shooting position and shooting posture of the image are associated with the score. The score associated with each shooting position / shooting posture is stored in, for example, the storage unit 107 of the terminal device 100. The score acquisition process is performed in this way. The score acquisition process is performed every time a photograph is taken to acquire a score.

When the sphere placement process is performed later without performing the score acquisition process and the sphere placement process in parallel, in the score acquisition process, the score is made to correspond to the shooting position and the shooting posture in which the image was taken to calculate the score. Need to keep. Then, a score map can be created by arranging the spheres on the grid corresponding to the shooting position information in the later arrangement process. In order to realize this, it is necessary to detect the shooting position and shooting posture with GPS, LiDAR, AR marker, etc. at the time of shooting.

Next, the score and reliability plot processing and the sphere arrangement processing in the sphere, which is a virtual three-dimensional object, will be described with reference to the flowchart of FIG. First, in step S201, the plot processing unit 202 reads out the score and reliability for each shooting position / shooting posture stored in the storage unit 107. Next, in step S202, the arrangement processing unit 203 determines the position and orientation in which the sphere is arranged.

Here, the method of determining the placement position of the sphere will be explained. First, as shown in FIG. 9A, a grid of a predetermined size is set on the map data showing the entire score map creation area in which an image is taken and the spheres are arranged. The size of this grid corresponds to the size of the sphere to be finally placed, and the size may be determined in advance by default, or the user may arbitrarily set the size. For example, if you want to know the score roughly, make the grid larger and make the spheres placed larger. On the other hand, if you want to know the score in detail, you can make the grid smaller and the spheres to be placed smaller. The grid may or may not be superimposed on the map data and displayed on the display unit 105.

Then, as shown by the arrow in FIG. 9B, when the user moves and shoots in the score map creation area, the sensor unit 103 of the terminal device 100 acquires the shooting position and shooting posture each time the shooting is performed. Then, when the information acquisition unit 201 calculates the score from the image, as shown in FIG. 10A, a sphere plotting the score calculated from the image taken at the shooting position is arranged on the grid where the shooting position exists inside. The size of the sphere must be less than or equal to the size of the grid, and the sphere in each grid will be plotted with all the scores calculated from the image whose shooting position is within that grid.

The sphere is not placed on the grid where shooting is not performed, that is, there is no shooting position inside. However, as shown in FIG. 10B, even if the grid is not photographed, the spheres can be arranged by estimating the score from the scores plotted on the spheres arranged on the adjacent grids.

Return to the explanation of the flowchart of FIG. Next, in step S203, the score included in the grid on which the spheres are arranged (having the position information in the grid) is read from the storage unit 107 and listed.

Next, in step S204, the plot processing unit 202 plots the score and reliability of the sphere. The plotting process will be described with reference to FIGS. 11 and 12.

First, the determination of the reference position (plot reference position) when plotting the score on the surface of the sphere will be explained. For example, as shown in FIG. 11A, it is assumed that the image P1, the image P2, and the image P3 are acquired by shooting at three shooting positions. Then, as shown in FIG. 11B, it is assumed that the score of the image P1 is calculated to be 100, the score of the image P2 is 200, and the score of the image P3 is 300 according to the shooting position and the shooting posture based on each image.

As shown in FIG. 11B, the score and the shooting position / shooting posture are associated with each other by assuming a pseudo sphere of a predetermined or arbitrary size and matching the shooting position with the center of the sphere. Then, the score is associated with the point where the surface of the sphere intersects with the extension line in the opposite direction of the line segment extending from the shooting position in the shooting direction.

Then, when the shooting positions of the three scores are located in one grid as shown in FIG. 12A, the three scores are placed on the grid as if they were shot at the same position in the same grid as shown in FIG. 12B. Plot on one corresponding sphere. When the score is plotted on a sphere corresponding to the grid as shown in FIG. 12B, the score plot reference position on the surface of the sphere is an extension line in the opposite direction of the line segment extending from the shooting position of each score in the shooting direction and the sphere. It becomes a point where the surfaces of are intersected.

The score will be different depending on the shooting posture even if the shooting position is the same, but by plotting the score at each shooting position on the surface of the sphere in this way, the score based on each shooting posture can be shown on one sphere. As a result, the user can grasp the score plotted on the sphere even when viewed from one direction without moving and changing his / her viewpoint.

All scores calculated from images taken in the same grid are plotted on the sphere as if they were taken at the same position (but with different postures), so the grid size is also the basis for the error tolerance of the shooting position. ..

If there is no grid setting, or if the shooting position and shooting posture are not in contact with each other due to the size of the grid, the plot reference position is determined for the sphere set for each shooting position and shooting posture in the state shown in FIG. 11B. You may.

Once the plot reference position of the score on the surface of the sphere is determined in this way, then plot on the entire surface of the sphere to plot the score concentrically around the plot reference position in the HSV color space over the entire surface of the sphere. You need to estimate the score.

The calculation of the score on the entire surface of the sphere will be described with reference to FIGS. 13 and 14. In calculating the score on the entire surface of the sphere, as shown in FIG. 13A, the sphere is a solid that approximates a sphere composed of only triangular surfaces, and the score for each triangular surface of the solid is estimated based on the score calculated from the image. To do.

The first method of score estimation for each triangular surface was calculated from an image on a triangular surface where an extension line in the opposite direction of a line segment extending from the center of the sphere in the shooting direction intersects as shown in FIG. It is a method of associating scores. The point where the extension lines intersect on the triangular surface is the plot reference position.

The second method of calculating the score for each triangular surface is a method of calculating the score for the triangular surface to which the score cannot be associated with the first method. In the second method, the plot reference position in the triangular plane to which the score is associated in the first method and the plot reference position in another triangular plane to which the score is associated are connected by a line segment. Then, when the line segment overlaps the triangular surface to which the score is not associated, the score of the surface to which the score is not associated is calculated by linear interpolation.

As shown in FIG. 13B, assuming that the triangular faces to which the scores are associated by the first method are face a and face b, connecting the plot reference position in the face a and the plot reference position in the face b is a face. Since it overlaps with c, the score of surface c is calculated by linear interpolation from the score corresponding to surface a and the score corresponding to surface b.

The third method of calculating the score for each triangular surface is a score in which the average of the scores in the adjacent triangular surfaces is associated with the triangular surface for the triangular surfaces whose scores cannot be associated with either the first method or the second method. And.

By implementing the first to third methods on all triangular surfaces, the score can be associated with the entire surface of the sphere. The score on the entire surface of the sphere is shown in FIG. 15 in HSV color space.

As shown in FIG. 15A, the score uses the hue of the HSV color space, for example, the color corresponding to the highest score is red, and as the score decreases, it changes to orange, yellow, green, light blue, and purple. To. For example, the scores of all the captured images are compared, normalized from the minimum to the maximum score, and then 1.0 is red and 0.0 is purple. The color corresponding to the score of an arbitrary position / posture can be determined by the following equation 4.

[Equation 4]
(S-Smin) / (Smax-Smin)

Such a color change is a method used when data with a height difference such as a high temperature is represented by a color, and it is possible to intuitively grasp a part with a high score and a part with a low score by visual inspection. By calculating the score plotted on the entire surface of the sphere in this way, the plot reference position has the highest score, and the score decreases concentrically around the plot reference position. Regarding the correspondence between hue and score, the hue scale value or RGB value and score value may be directly associated and plotted, or a unique standard for associating score and hue may be established and the color may be based on the standard. May be used to plot the score on a sphere.

Specific examples of the sphere on which the score is plotted are shown in FIGS. 15B and 15C. It should be noted that FIGS. 15B and 15C are diagrams showing a display state in which the actual color of the sphere on which the score is plotted changes in a gradation shape for convenience of explanation and drawing of drawings (FIGS. 15B-a and 15B-c). 15C-a, 15C-c) and diagrams (FIGS. 15B-b, 15B-d, 15C-b, 15C-d) showing colors hierarchically to explain color changes. Expressed by.

FIG. 15B shows a case where the score calculated from the image is one. FIG. 15B-b shows the hierarchical change of the color of FIG. 15B-a shown by the gradation, and FIG. 15B-d shows the hierarchical change of the color of FIG. 15B-c shown by the gradation. The plot reference position on the surface of the sphere has the highest score, and the score is plotted so that the score decreases concentrically from the plot reference position in red, orange, yellow, green, light blue, and blue. To.

FIG. 15C shows a case where there are two scores calculated from the image. FIG. 15C-b shows the hierarchical change of the color of FIG. 15C-a shown by the gradation, and FIG. 15C-d shows the hierarchical change of the color of FIG. 15C-c shown by the gradation. Each plot position of the two scores is displayed in the color with the highest score, and the scores are plotted so as to decrease concentrically around each of the two plot reference positions.

The reliability is also plotted on the sphere by changing the saturation using the saturation. Since the score calculated from the image corresponds to the plot reference position, the reliability is the highest. Therefore, the reliability is plotted so that the plot reference position has the highest saturation and the saturation decreases concentrically from the plot reference position.

In the plotting method of FIG. 15C, the two plot reference positions correspond to different shooting postures. In other words, the plotting method of FIG. 15C is a process of plotting information on a plurality of environments corresponding to a plurality of shooting postures on a single virtual three-dimensional object corresponding to a single shooting position. According to this plotting method, the user can simultaneously and three-dimensionally confirm information on a plurality of environments corresponding to a plurality of shooting postures with respect to a single shooting position. Therefore, the convenience (usability) of the user when the user changes the shooting posture and re-shoots is improved. The score calculated from the image plotted on the sphere may be 3 or more, and the number is not particularly limited.

Return to the explanation of the flowchart. Next, in step S205, the arrangement processing unit 203 arranges the sphere on the arrangement target to make the sphere visible to the user. The spheres can be arranged on the map data as shown in FIG. The sphere can be arranged on a two-dimensional map as shown in FIG. 16A, or can be arranged on a three-dimensional map as shown in FIG. 16B. The display on the display unit 105 may not be updated until the ball placement process in step S205 is completed, and the display may be updated to display the ball on which the score is plotted after the ball placement process is completed. , The display may be updated every time each process is completed. The present technology is not limited to either.

In order to arrange the spheres on this map, first, the grid to which the shooting position belongs is grasped based on the position information of the shooting position acquired by the sensor unit 103 in advance. Then, it can be realized by arranging a sphere on which the score calculated from the image taken at the shooting position is plotted at the position on the map of the grid to which the shooting position belongs. The score map can be created in this way.

Further, as shown in FIG. 17A, a two-dimensional image of a sphere can be combined and arranged on the image. The image on which the sphere is placed may be the same as or different from the image taken to calculate the score. Further, the sphere can be arranged on the image by arranging the sphere on the frame image constituting the image.

In order to synthesize and arrange the spheres on the image in this way, first, the position information of the shooting position is acquired in advance by the sensor unit 103. In addition, the position information of the shooting position of the image is also acquired. Then, it can be realized by superimposing a sphere displaying the score on an image having position information corresponding to the position information of the shooting position for score calculation and synthesizing it.

Furthermore, as shown in FIG. 17B, a sphere can be superimposed and displayed on a real space image in AR and MR. This makes it possible to present the score to the user as if the sphere were placed in the real world. FIG. 17B is an example in which a sphere is superimposed and displayed on a real space image captured and displayed by the camera of the smartphone SP.

In order to arrange the spheres in the AR and MR, the position information of the shooting position is acquired in advance by the sensor unit 103. Also, create a sphere displaying the score as an AR object. Furthermore, an AR marker associated with the display of the sphere is installed in the real world corresponding to the position information. Then, when the user observes the AR marker in the real world through an AR device such as a smartphone, a wearable device, or a head-mounted display, or an MR device, the sphere is displayed on the display of the AR device or MR device. In addition to the method using the AR marker, a method of displaying the AR object on the AR device or MR device using GPS information may be used.

In this way, the sphere is placed at the shooting position. In the above description, the sphere is placed at the shooting position, but even for the position where shooting is not performed, the estimated score is calculated at the position where shooting is not performed by estimating the score by the score estimation method described above. Plotted spheres can be placed. As a result, spheres with plots of scores can be placed even at positions where shooting is not performed to complement the score information, or scores can be estimated and spheres can be placed in a grid pattern so as to fill positions where shooting is not performed. It is possible to make the score information more detailed.

In any of the above-mentioned sphere arrangement methods, the spheres are arranged in a semi-transparent manner in order to prevent the spheres from being filled up in space due to the dense arrangement of the spheres and making the spheres difficult to see. You may do things such as automatically setting an interval.

The processing by the information processing device 200 is performed as described above. It is also possible to synchronously execute the score acquisition process and the score plot process in parallel processing, or it is also possible to acquire the score first and then perform the score display process. When shooting, score acquisition processing, score plotting, and placement processing are synchronously executed in parallel processing, the user moves from grid to grid while shooting with the camera unit 102, and shoots and moves while arranging spheres on the passed grid. It is also possible to continue.

Note that only the score may be plotted on the sphere, or only the reliability may be plotted. By defining the reliability of the position and posture for which the score has not been calculated as 0% and plotting it on the sphere, the position and posture for which the score has not been calculated can be confirmed at a glance. For example, when this reliability of 0% is represented by a black sphere, a score map can be efficiently created by calculating and plotting the score so as to erase the black sphere.

As shown in FIG. 18A, the score may be plotted on the sphere in the HSV color space, and the numerical value of the score may be plotted on the sphere. Further, in addition to the score value, the reliability value may be plotted.

It is also possible to estimate the score at the position where the score is not calculated from the score plotted on the sphere. As shown in FIG. 18B-a, two

spheres

1 and 2 on which the score calculated from the image is plotted are arranged, and no sphere is arranged in the space between them, and the sphere is not arranged at the midpoint between the sphere 1 and the sphere 2. Consider the case where the sphere 3 is to be arranged. In this case, if the plot positions on the two

spheres

1 and 2 on which the score is plotted are connected by a line and the line overlaps the sphere 3 on which the score is not plotted, the score of the sphere 3 is the score at the plot position of the sphere 1. It can be calculated by linear interpolation from the score at the plot position of sphere 1. The score plotted on the sphere 3 thus calculated has a color scheme indicating an intermediate value of the scores plotted on the sphere 1 and the sphere 2 as shown in FIGS. 18B-b and 18B-c.

Further, in the score map in which the sphere on which the score is plotted is arranged, as shown in FIG. 19A, the position and posture in which the score is not calculated or the reliability is lower than the predetermined value are presented and the image is taken at that position and posture. May be encouraged. In FIG. 19A, the thick line on the score map represents the position and the recommended movement route for photographing the posture without the score, and the thin arrow line indicates the direction without the score. The presentation of this recommended route is also possible in an image in which spheres are superimposed as shown in FIG. 19B.

The processing by the information processing device 200 is performed as described above. According to the present technology, the user can easily grasp the score, which is the information obtained from the captured image, and its reliability only by visual inspection. In addition, by expressing the score and reliability in color on the surface of the virtual sphere, the user can intuitively grasp the score. In addition, by plotting the score and reliability on a spherical surface instead of a flat surface, it is possible to grasp the score and reliability that differ depending on the shooting posture even when viewed from one direction.

<2. Modification example>

Although the embodiment of the present technology has been specifically described above, the present technology is not limited to the above-described embodiment, and various modifications based on the technical idea of the present technology are possible.

The terminal device 100 may be a personal computer, a tablet terminal, a digital camera, a mobile phone, a portable game machine, a head-mounted display, a wearable device, or the like, in addition to a smartphone.

Further, the terminal device 100 used as a camera by using the camera unit 102 and the terminal device 100 used as a display device for displaying a sphere which is a virtual three-dimensional object may be the same device, or may be separate devices. There may be. Further, the device used as a camera by using the camera unit 102, the device for displaying a sphere which is a virtual three-dimensional object, and the device operating as the information processing device 200 may all be separate devices.

The present technology can also have the following configurations.
(1)
An information acquisition unit that acquires information on the environment within the shooting range from the image,
A plot processing unit that plots the above information on the surface of a virtual three-dimensional object,
An information processing device including an arrangement processing unit for arranging the virtual three-dimensional object as an arrangement target.
(2)
The information processing device according to (1), wherein the plot processing unit plots the information on the virtual three-dimensional object based on the shooting position and shooting posture of the image.
(3)
The information processing apparatus according to (1) or (2), wherein the plot processing unit plots the information corresponding to the plurality of shooting postures on the single virtual three-dimensional object corresponding to the single shooting position. ..
(4)
The information processing apparatus according to any one of (1) to (3), wherein the arrangement processing unit arranges the virtual three-dimensional object at a position in the arrangement target corresponding to the shooting position of the image.
(5)
A grid is set in the area where the image is taken, and the grid is set.
The arrangement processing unit arranges the virtual three-dimensional object in the grid in which the photographing position exists.
The information processing apparatus according to (4), wherein the plot processing unit plots the information acquired from the image in which the shooting position exists in the grid onto the virtual three-dimensional object arranged on the grid.
(6)
The information processing apparatus according to any one of (1) to (5), which is an index showing the characteristics of the environment of the image within the shooting range and is a numerical score.
(7)
The information processing apparatus according to (6), wherein the information is the reliability of the score.
(8)
The information processing device according to (6), wherein the information acquisition unit calculates a score as the information from the image.
(9)
The information processing apparatus according to (8), wherein the information acquisition unit estimates a score outside the shooting range of the image from the score calculated from the image.
(10)
The information processing device according to any one of (1) to (9), wherein the virtual three-dimensional object is a sphere.
(11)
The information processing device according to any one of (1) to (10), wherein the arrangement target is spatial data.
(12)
The information processing device according to (11), wherein the arrangement target is two-dimensional and / or three-dimensional map data.
(13)
The information processing device according to (11) or (12), wherein the arrangement target is an image and / or a moving image.
(14)
The information processing device according to any one of (11) to (13), which is a virtual space.
(15)
The information processing apparatus according to any one of (1) to (14), wherein the plot processing unit plots the information on the virtual three-dimensional object using the HSV color space.
(16)
The information processing apparatus according to (15), wherein the plot processing unit plots the score on the virtual three-dimensional object using hue, and plots the reliability on the virtual three-dimensional object using saturation.
(17)
The arrangement processing unit sets a grid in the real space as an area for capturing the image, and arranges the sphere as the virtual three-dimensional object at a specific position in the spatial data as the arrangement target corresponding to the grid. ,
The information processing apparatus according to any one of (1) to (16), wherein the plot processing unit plots a score calculated from the image on the surface of the sphere.
(18)
Obtain information within the shooting range from the image and
The above information is plotted on the surface of a virtual three-dimensional object.
An information processing method for arranging the virtual three-dimensional object as an arrangement target.
(19)
Obtain information within the shooting range from the image and
The above information is plotted on the surface of a virtual three-dimensional object.
An information processing program that causes a computer to execute an information processing method for arranging the virtual three-dimensional object as an arrangement target.

200 ... Information processing device.
201 ... Information acquisition unit 202 ... Plot processing unit 203 ... Arrangement processing unit

Claims

An information acquisition unit that acquires information on the environment within the shooting range from the image,
A plot processing unit that plots the above information on the surface of a virtual three-dimensional object,
An information processing device including an arrangement processing unit for arranging the virtual three-dimensional object as an arrangement target.
The information processing device according to claim 1, wherein the plot processing unit plots the information on the virtual three-dimensional object based on the shooting position and shooting posture of the image.
The information processing device according to claim 1, wherein the plot processing unit plots the information corresponding to the plurality of shooting postures on the single virtual three-dimensional object corresponding to the single shooting position.
The information processing device according to claim 1, wherein the arrangement processing unit arranges the virtual three-dimensional object at a position in the arrangement target corresponding to the shooting position of the image.
A grid is set in the area where the image is taken, and the grid is set.
The arrangement processing unit arranges the virtual three-dimensional object in the grid in which the photographing position exists.
The information processing apparatus according to claim 4, wherein the plot processing unit plots the information acquired from the image in which the photographing position exists in the grid onto the virtual three-dimensional object arranged on the grid.
The information processing apparatus according to claim 1, wherein the information is an index showing the characteristics of the environment within the shooting range of the image, and is a score represented by a numerical value.
The information processing device according to claim 6, wherein the information is the reliability of the score.
The information processing device according to claim 6, wherein the information acquisition unit calculates a score as the information from the image.
The information processing device according to claim 8, wherein the information acquisition unit estimates a score outside the shooting range of the image from the score calculated from the image.
The information processing device according to claim 1, wherein the virtual three-dimensional object is a sphere.
The information processing device according to claim 1, wherein the arrangement target is spatial data.
The information processing device according to claim 11, wherein the arrangement target is two-dimensional and / or three-dimensional map data.
The information processing device according to claim 11, wherein the arrangement target is an image and / or a moving image.
The information processing device according to claim 11, wherein the arrangement target is a virtual space.
The information processing device according to claim 1, wherein the plot processing unit plots the information on the virtual three-dimensional object using the HSV color space.
The information processing apparatus according to claim 15, wherein the plot processing unit plots the score on the virtual three-dimensional object using hue and plots the reliability on the virtual three-dimensional object using saturation.
The arrangement processing unit sets a grid in the real space as an area for capturing the image, and arranges the sphere as the virtual three-dimensional object at a specific position in the spatial data as the arrangement target corresponding to the grid. ,
The information processing device according to claim 1, wherein the plot processing unit plots a score calculated from the image on the surface of the sphere.
Obtain information within the shooting range from the image and
The above information is plotted on the surface of a virtual three-dimensional object.
An information processing method for arranging the virtual three-dimensional object as an arrangement target.
Obtain information within the shooting range from the image and
The above information is plotted on the surface of a virtual three-dimensional object.
An information processing program that causes a computer to execute an information processing method for arranging the virtual three-dimensional object as an arrangement target.