JP2013121150A - Information processing device and information processing method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a new and improved information processing device that reduces time and effort during imaging and can generate a composite image having natural perspective regardless of characteristics of an object.SOLUTION: According to a viewpoint of the present invention, an information processing device includes: an image acquisition part that acquires a right-eye image and a left-eye image in each of which the same object is drawn at a different horizontal position; a region dividing part that divides a reference image, which is at least one image of the right-eye image and left-eye image, into a plurality of small regions; an attitude information acquisition part that acquires attitude information indicating the existence position of an overlapped image in a real space that is overlapped with at least one image of the right-eye image and left-eye image; a small region specification part that specifies an overlapped small region on which an overlapped image is overlapped, among a plurality of small regions on the basis of the attitude information; and a depth calculation part that calculates an existence position in the real space of at least an overlapped small region among the plurality of small regions on the basis of the right-eye image and left-eye image.

Description

  The present invention relates to an information processing apparatus and an information processing method.

  In recent years, a technique called augmented reality (AR “Augmented Reality”) that generates a composite image by superimposing a fictitious image (AR image) on a captured image obtained by photographing an object in real space has been used. It is getting to be. According to this AR technology, a user who has visually recognized a composite image recognizes as if the AR image exists in real space.

  In the conventional AR technology, the AR image is simply superimposed on the captured image. Therefore, when the object image (image on which the object is drawn) and the AR image overlap, the positional relationship between the AR image and the object in the real space. Regardless, the object image is blocked by the AR image. For this reason, the user may have a sense of incongruity in the perspective of the AR image.

  For example, even when a part of an AR image is occluded by an object in real space, the object image is occluded by the AR image in the synthesized image. I feel uncomfortable. Specifically, the user recognizes that the part of the AR image superimposed on the object image is present on the near side of the object, and recognizes that the other part is present on the back side of the object. End up.

  Patent Document 1 discloses a technique aimed at solving such a problem. In this technique, when a part of an AR image is shielded by an object in real space, the object image is erased from the captured image, and then the AR image is superimposed on the captured image.

JP 2010-33397 A

  However, in the technique disclosed in Patent Document 1, it is necessary to paste a marker indicating that the image is to be erased by the AR image to the object before shooting. For this reason, there has been a problem that the time and effort at the time of shooting increases. In addition, there is a problem that the characteristics of the object are limited in order to erase the object image without a sense of incongruity.

  Therefore, the present invention has been made in view of the above problems, and an object of the present invention is to reduce the time and effort at the time of shooting and to create a natural composite image with a sense of perspective regardless of the characteristics of the object. It is an object to provide a new and improved information processing apparatus and information processing method that can be generated.

  In order to solve the above problems, according to an aspect of the present invention, an image acquisition unit that acquires a right-eye image and a left-eye image in which the same object is drawn at different horizontal positions, and a right-eye image and a left-eye image An area dividing unit that divides a reference image, which is at least one of the images, into a plurality of small areas, and a real image of a superimposed image to be superimposed on at least one of the right-eye image and the left-eye image. A posture information acquisition unit that acquires posture information indicating an existing position, a small region specifying unit that specifies a superimposed small region on which a superimposed image is superimposed among a plurality of small regions based on the posture information, a right-eye image, and An information processing apparatus is provided that includes a depth calculation unit that calculates an existence position in a real space of at least a superimposed small region among a plurality of small regions based on a left-eye image.

  According to this aspect, the information processing apparatus divides the reference image, which is at least one of the right-eye image and the left-eye image, into a plurality of small areas, and among the plurality of small areas, Calculate the location in real space.

  Therefore, when superimposing the superimposed image on the reference image, the information processing apparatus recognizes the positional relationship between the superimposed small area and the superimposed image in the real space based on the position of the superimposed small area in the real space. be able to.

  Here, the area dividing unit may divide the reference image into a plurality of small areas by dividing pixels having similar pixel values into the same small area.

  According to this aspect, the information processing apparatus divides the reference image into a plurality of small areas by dividing pixels having similar pixel values into the same small area, so that the reference image is more accurately divided. be able to.

  In addition, the depth calculation unit includes a parallax calculation unit that extracts corresponding points indicating the same part of the object from the right-eye image and the left-eye image, and calculates the presence position of the corresponding point in real space based on the corresponding points. If the corresponding point exists in the superimposed small area, the position of the corresponding small area in the real space may be calculated based on the position of the corresponding point in the real space.

  According to this aspect, the information processing apparatus calculates the existence position of the superimposed small area in the real space based on the corresponding point, and thus can accurately calculate the existence position of the superimposed small area in the real space. .

  In addition, when there is no corresponding point in the overlapped small area, the depth calculation unit determines whether the overlapped small area in the real space is based on the position in the real space of the small area that exists around the overlapped small area. The existence position may be calculated.

  According to this aspect, when there is no corresponding point in the ultra-small area, the information processing apparatus determines whether the superimposed small area is based on the position in the real space of the small area that exists around the superimposed small area. Calculate the location in real space. Therefore, the information processing apparatus can more accurately calculate the existence position in the real space of the superimposed small region even when the number of corresponding points is small.

  In addition, based on the posture information, the superimposed small region position information indicating the position of the superimposed small region in the real space, and the position information of the imaging device that captured the reference image, the superimposed image is included in each portion of the superimposed image. By identifying a superimposable part closer to the imaging device than the small area and synthesizing the superimposable part, a superimposing image generating unit that generates a superimposable image, and the superimposable image is at least one of the right eye image and the left eye image An image composition unit that superimposes the image on one image.

  According to this aspect, the information processing device identifies a superimposable portion existing in a position closer to the imaging device (on the near side) than the superimposing small region among the portions of the superimposed image, and synthesizes the superimposable portion. Thus, a superimposable image is generated. The information processing apparatus superimposes the superimposable image on at least one of the right-eye image and the left-eye image. Thereby, the information processing apparatus 10 can generate a composite image in which the perspective of the AR image and the object image is more natural.

  The image composition unit may superimpose the superimposable image on both the right-eye image and the left-eye image.

  According to this aspect, since the information processing apparatus superimposes the superimposable image on both the right eye image and the left eye image, the superimposable image can be stereoscopically displayed.

  According to another aspect of the present invention, at least one of an image acquisition step for acquiring a right-eye image and a left-eye image in which the same object is drawn at different horizontal positions, and a right-eye image and a left-eye image. An area dividing step for dividing the reference image, which is an image, into a plurality of small areas, and posture information indicating a position in the real space of the superimposed image to be superimposed on at least one of the right-eye image and the left-eye image. Based on the posture information acquisition step to be acquired, and based on the posture information, a small region specifying step for specifying a superimposed small region on which the superimposed image is superimposed among a plurality of small regions, and on the right eye image and the left eye image, A depth calculation step of calculating an existence position of at least a superimposed small region in real space among a plurality of small regions is provided.

  In the information processing method according to this aspect, the reference image, which is at least one of the right-eye image and the left-eye image, is divided into a plurality of small regions, and among the plurality of small regions, at least in the real space of the superimposed small region. Is calculated.

  Therefore, in this information processing method, when the superimposed image is superimposed on the reference image, the positional relationship in the real space between the superimposed small region and the superimposed image is recognized based on the position of the superimposed small region in the real space. can do.

  As described above, according to the present invention, the information processing apparatus can grasp which part of the superimposed image is shielded by the superimposed small region without attaching a marker to each object. Therefore, labor during photographing is reduced. Furthermore, since the information processing apparatus can perform the above-described processing regardless of the object, it is possible to generate a natural composite image with a sense of perspective regardless of the characteristics of the object. Furthermore, the information processing apparatus can express that the superimposed image is shielded by the object image on the near side.

It is a block diagram which shows the structure of the information processing apparatus which concerns on embodiment of this invention. It is a block diagram which shows the structure of the depth estimation part concerning the embodiment. It is explanatory drawing for demonstrating the method to calculate the position of an object based on binocular parallax. It is explanatory drawing which shows the example of the image for right eyes, and the image for left eyes. It is explanatory drawing for demonstrating the method to divide | segment an image into a small area | region. It is explanatory drawing for demonstrating the method to calculate the position of each small area | region. It is explanatory drawing for demonstrating the method to calculate the position of each small area | region. It is explanatory drawing for demonstrating the positional relationship of each object and AR image. It is explanatory drawing which shows an example of AR image produced | generated by information processing apparatus. It is explanatory drawing which shows an example of the captured image with which AR image is superimposed. It is explanatory drawing which shows an example of the synthesized image by which AR image was superimposed on the captured image. It is a flowchart which shows the procedure of the process by an imaging device. It is explanatory drawing which shows an example of the conventional synthesized image. It is explanatory drawing which shows an example of the captured image used with the conventional AR technique. It is explanatory drawing which shows an example of the conventional synthesized image.

  Exemplary embodiments of the present invention will be described below in detail with reference to the accompanying drawings. In addition, in this specification and drawing, about the component which has the substantially same function structure, duplication description is abbreviate | omitted by attaching | subjecting the same code | symbol.

  In the present embodiment, the terms “back side” and “near side” indicate a positional relationship based on the focal point of the imaging apparatus. For example, when “one object exists behind another object”, the depth value from one object to the focus of the imaging device is longer than the depth value from the other object to the focus of the imaging device. Here, the “depth value” of each object is the distance in the optical axis direction from each object to the focal point.

<1. Consideration of conventional AR technology>
The present inventor has come up with the information processing apparatus 10 (see FIG. 1) according to the present embodiment by examining the conventional AR technology. First, the conventional AR technology and its problems will be described.

  As described above, the conventional AR technique merely superimposes the AR image on the captured image. Therefore, when the object image and the AR image overlap, regardless of the positional relationship between the AR image and the object in the real space. The object image is blocked by the AR image. For this reason, the user may have a sense of incongruity in the perspective of the AR image.

  An example of a conventional composite image is shown in FIG. In this example, the AR image 200 is superimposed on the captured image 100. In the captured image 100, a chest image 101, a marker image 102, and a plurality of floor panel images 103 are drawn. The marker image 102 is drawn on the floor panel image 103a and indicates the position where the AR image 200 is superimposed. The marker image 102 is synthesized with the captured image 100 or is obtained by drawing a marker in advance on the floor panel and photographing it.

  Therefore, since the AR image 200 is superimposed on the marker image 102, a part of the AR image 200 is shielded by a chase in real space. However, the AR image 200 is superimposed on the captured image 100 regardless of the positional relationship between the chest and the floor panel (object) in the real space. Specifically, the AR image 200 is superimposed on both the chest image 101 and the floor panel image disposed behind the chest in real space, that is, the floor panel image 103a.

  Therefore, the user who has visually recognized the composite image has a sense of incongruity in the perspective of the AR image 200. Specifically, in the AR image 200, the user recognizes that the portion superimposed on the chest image 101 is present in front of the chest, and the portion superimposed on the floor panel image 103a is behind the chest. It is recognized that it exists on the side.

  Patent Document 1 discloses a technique aimed at solving such a problem. In this technique, when a part of an AR image is shielded by an object, the object image is erased from the captured image, and then the AR image is superimposed on the captured image.

  When the technique disclosed in Patent Literature 1 is applied to the captured image 100 shown in FIG. 13, the following processing is performed. First, as shown in FIG. 14, a marker indicating that it is an object to be erased by an AR image is attached to the chest, and it is photographed. Thereby, the marker image 104 is drawn on the captured image 100 in addition to the above-described images 101, 102, and 103. Then, the technique disclosed in Patent Document 1 erases the chest image 101 to which the marker image 104 is attached from the captured image 100, and then superimposes the AR image 200 on the captured image 100 as shown in FIG. That is, in the technique disclosed in Patent Document 1, a marker is attached to an object so that an object image on which the AR image 200 may be superimposed can be identified.

  According to the technology disclosed in Patent Document 1, the user can recognize the perspective of the AR image 200 without a sense of incongruity. However, in the technique disclosed in Patent Document 1, it is necessary to paste a marker indicating that the AR image 200 is an object to be erased onto an object before shooting. For this reason, there has been a problem that the time and effort at the time of shooting increases. In addition, there is a problem that the characteristics of the object are limited in order to erase the object image without a sense of incongruity. Specifically, the object needs to be placed on the bottom of the planar texture, and is isolated from the small object and the other object. Furthermore, the technique disclosed in Patent Document 1 cannot express that the AR image is shielded by the near-side object image.

  JP-A-10-112831 and JP-A-2008-532149 recognize the existence position of an object in the real space, and based on the result, determine the positional relationship between the object and the AR image in the real space. Disclose the technology to grasp. According to these techniques, it is possible to superimpose only a portion existing on the near side of the object in the AR image on the captured image. And these gazettes disclose that the presence position of the object in the real space may be recognized based on the image for the right eye and the image for the left eye. However, since these documents do not disclose any specific recognition method based on the right-eye image and the left-eye image, it is impossible to apply these techniques to an actual apparatus.

  On the other hand, the information processing apparatus 10 according to the present embodiment solves any of the above problems. In other words, according to the present embodiment 10, labor during shooting is reduced, the AR image can be superimposed on the captured image regardless of the characteristics of the object, and the AR image is shielded by the object image on the front side. Can be expressed. Hereinafter, this embodiment will be described in detail.

<2. Configuration of information processing apparatus>
Next, the configuration of the information processing apparatus 10 will be described based on FIG. 1 and FIG. The information processing apparatus 10 includes a distortion aberration correction unit (image acquisition unit) 20, an attitude information generation unit 30, a depth estimation unit 40, a small region specification unit 51, a superimposed image generation unit 52, and an image synthesis unit 60. The display part 70 is provided. The information processing apparatus 10 has hardware such as a CPU, ROM, RAM, memory, display, etc., and the CPU reads and executes a program stored in the ROM, whereby processing by each functional block described above is performed. Is realized. That is, the ROM includes the imaging apparatus 10, the distortion correction unit 20, the posture information generation unit 30, the depth estimation unit 40, the small region specification unit 51, the superimposed image generation unit 52, the image composition unit 60, and the display unit 70. A program for realizing it is stored.

  The distortion correction unit 20 acquires a right-eye image and a left-eye image in which the same object is drawn at different horizontal positions, and corrects these distortions. The distortion aberration correction unit 20 outputs the corrected right-eye image and left-eye image to the depth estimation unit 40 and the image synthesis unit 60. Here, the distortion correction unit 20 may acquire the right-eye image and the left-eye image from the outside of the information processing apparatus 10. Alternatively, the right eye image and the left eye image may be stored in advance in the memory of the information processing apparatus 10, and the distortion correction unit 20 may acquire the right eye image and the left eye image from the memory. Alternatively, the information processing apparatus 10 may include two imaging devices, and the distortion correction unit 20 may acquire a right-eye image and a left-eye image from these imaging devices.

  The posture information generation unit 30 acquires posture information indicating the position of the AR image (superimposed image) in real space. Here, the posture information includes global coordinates (x, y, z) of predetermined reference points in the AR image, angles (κ, ω, φ) of the AR image, and an expansion coefficient α. The global coordinates are coordinates in the global coordinate system set in the real space. The angle of the AR image is an angle formed by a predetermined axis passing through the AR image and the xyz axis of the global coordinate system. The expansion coefficient α is a coefficient indicating how large the AR image is displayed. The AR image is defined by the global coordinates and surface information of each vertex (mathematical information for specifying the surface of the AR image). The size of the AR image is determined by multiplying the global coefficient and surface information of each vertex by the expansion coefficient α.

  Therefore, the position of the AR image in the real space (specifically, the global coordinates (x, y, z) of the reference point, the angles (κ, ω, φ) of the AR image, and the expansion coefficient α) Indicates the position of each part of the AR image). In the above prior art, these parameters are set according to the position, direction, and size of the marker. In the present embodiment, the presence position of the AR image in the real space may be set by the same method as in the conventional technique, or the user may input it manually. The posture information generation unit 30 outputs posture information to the small region specifying unit 51.

  As illustrated in FIG. 2, the depth estimation unit 40 includes a parallax calculation unit 41, a region division unit 42, and a depth calculation unit 43. The parallax calculation unit 41 extracts corresponding points indicating the same part of the object from the right-eye image and the left-eye image, and based on the extracted corresponding points and the following equation (1), the real space of each corresponding point The existence position at, specifically, the depth value is calculated.

  Here, each parameter in Formula (1) is demonstrated based on FIG.3 and FIG.4. FIG. 3 is a plan view showing a state in which the object Ob1 is photographed by the left and right imaging devices. FIG. 4A shows a left-eye image LG10 taken by the left imaging device, and FIG. 4B shows a right-eye image RG10 taken by the right imaging device.

  According to FIGS. 3 and 4, an object image LG11 that is an image of the object Ob1 is drawn on the left-eye image LG10, and an object image RG11 that is an image of the object Ob1 is drawn on the right-eye image RG10. . Further, these object images LG11, RG11 include corresponding points PL, PR corresponding to the same portion P (gazing point) on the object Ob1.

  The main surface of the left imaging device and the main surface of the right imaging device are common and are indicated by a main surface S. Point LEP is the focal point of the left imaging device, and point REP is the focal point of the right imaging device. A straight line L1 indicates the optical axis of the left imaging device, and a straight line R1 indicates the optical axis of the right imaging device.

  The depth value d is the distance in the optical axis L1 (or optical axis R1) direction from the gazing point P to the focus LEP of the left imaging device (or the focus REP of the right imaging device). The interfocal distance D is a distance from the focal point LEP to the focal point REP. The focal distance f is a distance from the focal point LEP (or focal point REP) to the main surface S. The distance x1 is a horizontal distance from the corresponding point PL to the center point of the left-eye image, and the distance x2 is a horizontal distance from the corresponding point PR to the center point of the right-eye image.

  The parallax calculation unit 41 generates depth information in which the position of each corresponding point in the left-eye image or the right-eye image and the depth value are associated with each other, and outputs the depth information to the depth calculation unit 43.

  The region dividing unit 42 divides the reference image into a plurality of small regions using either the right-eye image or the left-eye image as a reference image. Hereinafter, an example in which the left-eye image is a reference image will be described, but the right-eye image may be a reference image. When the display unit 70 performs stereoscopic display, the information processing apparatus 10 performs the following process using both the right-eye image and the left-eye image as reference images. First, an example of processing performed by the area dividing unit 42 will be described with reference to FIG. In this example, an image G10 shown in FIG. 5A is a reference image. The reference image G10 is divided into a region G11 having uniform luminance and a region G12 having uniform luminance different from the luminance of the region G11.

  As shown in FIG. 5B, the region dividing unit 42 divides the reference image G10 into a plurality of square regions SQ each having a predetermined length WL. The area dividing unit 42 further divides the square area SQ into four square areas SQ when the variation (for example, dispersion) of pixel values (for example, luminance) in the square area SQ is equal to or greater than a predetermined value. The area dividing unit 42 repeats the above processing until the variance in the square area SQ becomes less than the predetermined value or the length of one side of the square area SQ becomes the predetermined value WS (<WL).

  The area dividing unit 42 calculates an arithmetic average value of the pixel values in each square area SQ. Then, the area dividing unit 42 compares the arithmetic average values of the pixel values of each square area SQ and the square areas SQ adjacent to the square area SQ in the vertical and horizontal directions. When the difference between the arithmetic average values is within a predetermined range, the area dividing unit 42 determines that these square areas SQ constitute the same small area, and combines these square areas SQ. Thereby, the area dividing unit 42 forms small areas. A broken line L1 shown in FIG. 5C indicates a line that is a boundary of a small region in such a coupling process.

  The area dividing unit 42 repeats the above processing until there is no square area SQ to be combined or the horizontal width of the small area reaches the predetermined value WL. As a result, as shown in FIG. 5D, the reference image G10 is divided into a plurality of small regions SAR. The area dividing unit 42 outputs small area information regarding the position and size of each small area on the reference image to the depth calculating unit 43.

  The depth calculation unit 42 calculates the depth value of each small region. Here, the processing performed by the depth calculation unit 42 will be described with reference to FIGS. First, as shown in FIG. 6A, the depth calculation unit 42 assigns a corresponding point PL to each small region SAR. In this example, the reference image is the left-eye image. Next, for the small area SAR to which the corresponding point is assigned, the depth calculation unit 42 sets the arithmetic average value of the depth values of the corresponding points as the depth value of the small area SAR. Then, as illustrated in FIG. 6B, the depth calculation unit 42 divides the small area SAR into the first small area SAR 10 in which the depth value has been calculated and the second small area SAR 20 in which the depth value has not been calculated. Break down.

  Then, the depth calculation unit 42 specifies the first small area SAR10 adjacent to the second small area SAR20 in the upper, lower, left, and right directions, and uses the arithmetic average value of these depth values as the depth value of the second small area SAR20. . For example, as illustrated in FIG. 7A, the depth calculation unit 42 sets the depth value of the second small area SAR20a as the arithmetic average value of the depth values of the first small areas SAR10a and SAR10b. Then, the depth calculation unit 42 changes the second small area SAR20 in which the depth value is calculated to the first small area SAR10. Note that when there are a plurality of first small areas SAR10 adjacent in the same direction, the depth calculation unit 42 may calculate the depth value of the second small area SAR20 by using all the depth values. The depth value may be calculated using either one of them. The depth calculation unit 42 calculates the depth values of all the small regions SAR as shown in FIGS. 7B to 7C by sequentially repeating the above processing. That is, the depth calculation unit 42 finally determines the depth value of the second small area SAR20b based on the depth values of the first small areas SAR10c, 10d, and 10e as shown in FIG. calculate. The depth calculation unit 42 outputs small region depth value information regarding the position, size, and depth value of each small region on the reference image to the small region specifying unit 51.

  The small area specifying unit 51 acquires an AR image. Here, the small area specifying unit 51 may acquire the AR image from the outside of the information processing apparatus 10. Alternatively, the AR image may be stored in advance in the memory of the information processing apparatus 10, and the small area specifying unit 51 may acquire the AR image from the memory. Furthermore, the small area specifying unit 51 also acquires global coordinate information of the left imaging device. This information is also acquired by the same method as that for the AR image.

  The small region specifying unit 51 defines a virtual space corresponding to the real space, and arranges the AR image in the virtual space based on the AR image and the posture information. Then, the small area specifying unit 51 projects (superimposes) the AR image on the reference image based on the global coordinate information of the left imaging device.

  And the small area specific | specification part 51 specifies the small area on which AR image is superimposed among several small areas as a superimposed small area based on small area depth value information and AR image on a reference | standard image. Then, the small region specifying unit 51 outputs the superimposed small region information, the AR image, and the posture information regarding the superimposed small region to the superimposed image generating unit 52.

  The superimposed image generation unit 52 calculates a depth value d1 of a portion corresponding to the superimposed small region in the AR image based on the AR image and the posture information. Further, the small area specifying unit 51 acquires the depth value d2 of the superimposed small area based on the superimposed small area information. And the small area specific | specification part 51 recognizes the part which becomes d1 <d2 among each part of AR image as a superimposable part.

  FIG. 8 shows an example of a portion that can be superimposed and an example that is not. In this example, the left imaging device is photographing the object Ob2 and Ob3 disposed on the back side of the object Ob2, and the AR image AR10 is disposed between the object Ob2 and the object Ob3.

  According to FIG. 8, for example, a small area corresponding to the line of sight A or B is set as a superimposed small area. The depth value of the overlapping small area corresponding to the line of sight A is dpA, and the depth value of the portion corresponding to the line of sight A in the AR image AR10 is dcA (> dpA). Therefore, the portion corresponding to the line of sight A in the AR image AR10 is not a superimposable portion.

  On the other hand, the depth value of the overlapping small area corresponding to the line of sight B is dpB, and the depth value of the portion corresponding to the line of sight B in the AR image AR10 is dcB (<dpB). Therefore, a portion corresponding to the line of sight B in the AR image AR10 is a superimposable portion.

  The superimposed image generation unit 52 generates a superimposable AR image by combining the superimposable portions. FIG. 9 shows a superimposable AR image AR11 as an example of the superimposable AR image. The superimposable AR image AR11 is a part of the AR image 10 missing.

  The superimposed image generation unit 52 outputs the superimposable AR image to the image composition unit 60. The image composition unit 60 generates a left-eye composite image by superimposing a superimposable AR image on the left-eye image. In addition, when the superimposable AR image is generated also for the right-eye image, the superimposed image generation unit 52 generates a right-eye composite image by superimposing the superimposable AR image on the right-eye image.

  Examples of synthesis are shown in FIGS. In this example, a superimposable AR image AR11 is superimposed on the captured image 100 described above. According to this example, since the portion that can be superimposed on the chest image 101 is missing in the superimposable AR image 11, the sense of perspective of these images becomes more natural.

  The display unit 70 displays these composite images. That is, when the right-eye synthesized image and the left-eye synthesized image are generated, the display unit 70 causes the right-eye synthesized image to be visually recognized by the user's right eye, and causes the left-eye synthesized image to be visually recognized by the user's left eye. Note that when only the left-eye composite image is generated, the image composition unit 60 displays the left-eye composite image in 2D.

<3. Processing procedure by information processing apparatus>
Next, a processing procedure by the information processing apparatus will be described with reference to a flowchart shown in FIG. In the following description, an example in which the left-eye image is the reference image will be described, but the right-eye image may be the reference image. In step S10, the distortion correction unit 20 acquires right-eye images and left-eye images in which the same object is drawn at different horizontal positions, and corrects these distortion aberrations. The distortion aberration correction unit 20 outputs the corrected right-eye image and left-eye image to the depth estimation unit 40 and the image synthesis unit 60.

  In step S20, the posture information generation unit 30 acquires posture information indicating the position of the AR image (superimposed image) in real space. The posture information generation unit 30 outputs posture information to the small region specifying unit 51.

  In step S30, the parallax calculation unit 41 extracts corresponding points indicating the same part of the object from the right-eye image and the left-eye image, and based on the extracted corresponding points and the above-described equation (1), The position of the point in the real space, specifically, the depth value is calculated. The parallax calculation unit 41 generates depth information in which the position of each corresponding point in the left-eye image or the right-eye image and the depth value are associated with each other, and outputs the depth information to the depth calculation unit 43.

  In step S40, the area dividing unit 42 sets the left-eye image as a reference image. Next, the region dividing unit 42 divides the reference image into a plurality of small regions by the processing described above. The region dividing unit 42 outputs small region information regarding the position and size of each small region on the reference image to the depth calculating unit 43. Note that the information processing apparatus 10 may use the right-eye image as a reference image. When performing stereoscopic display, the information processing apparatus 10 also uses the right-eye image as a reference image, and performs the processing from step S40 onward for the right-eye image.

  In step S50, the depth calculation unit 42 calculates the depth value of each small region by the above-described processing. The depth calculation unit 42 outputs small region depth value information regarding the position, size, and depth value of each small region on the reference image to the small region specifying unit 51.

  In step S60, the small region specifying unit 51 acquires an AR image and global coordinate information of the left imaging device.

  The small region specifying unit 51 defines a virtual space corresponding to the real space, and arranges the AR image in the virtual space based on the AR image and the posture information. Then, the small area specifying unit 51 projects (superimposes) the AR image on the reference image based on the global coordinate information of the left imaging device.

  Next, the small region specifying unit 51 specifies a small region on which the AR image is superimposed as a superimposed small region among the plurality of small regions. Then, the small region specifying unit 51 outputs the superimposed small region information, the AR image, and the posture information regarding the superimposed small region to the superimposed image generating unit 52.

  In step S <b> 70, the superimposed image generation unit 52 calculates a depth value d <b> 1 of a portion corresponding to the superimposed small region in the AR image based on the AR image and the posture information. Further, the small area specifying unit 51 acquires the depth value d2 of the superimposed small area based on the superimposed small area information. And the small area specific | specification part 51 recognizes the part which becomes d1 <d2 among each part of AR image as a superimposable part. The superimposed image generation unit 52 generates a superimposable AR image by combining the superimposable portions. The superimposed image generation unit 52 outputs the superimposable AR image to the image composition unit 60.

  In step S80, the image synthesis unit 60 generates a left-eye synthesized image by superimposing a superimposable AR image on the left-eye image. In addition, when the superimposable AR image is generated also for the right-eye image, the superimposed image generation unit 52 generates a right-eye composite image by superimposing the superimposable AR image on the right-eye image.

  Next, the display unit 70 displays these composite images. That is, when the right-eye synthesized image and the left-eye synthesized image are generated, the display unit 70 causes the right-eye synthesized image to be visually recognized by the user's right eye, and causes the left-eye synthesized image to be visually recognized by the user's left eye. Note that when only the left-eye composite image is generated, the image composition unit 60 displays the left-eye composite image in 2D.

  As described above, according to the present embodiment, the information processing apparatus 10 divides the reference image, which is at least one of the right-eye image and the left-eye image, into a plurality of small regions, and among the plurality of small regions, At least the depth value of the superimposed small area is calculated.

  Therefore, when superimposing the AR image on the reference image, the information processing apparatus 10 can recognize the positional relationship in the real space between the superimposed small area and the AR image based on the depth value (step S60). To S70).

  Thereby, the information processing apparatus 10 can grasp which part of the AR image is shielded by the superimposed small region without attaching a marker to each object. Therefore, labor during photographing is reduced. Furthermore, since the information processing apparatus 10 can perform the above-described processing regardless of the object, it is possible to generate a natural synthetic image with a sense of perspective regardless of the characteristics of the object ( FIG. 11). Furthermore, the information processing apparatus 10 can express that the AR image is shielded by the near-side object image (see FIG. 11).

  In the present embodiment, the depth values of all the small regions are calculated, but the depth values may be calculated at least for the superimposed small regions.

  Furthermore, since the information processing apparatus 10 divides pixels having similar pixel values into the same small area, and divides the reference image into a plurality of small areas, the reference image can be more accurately divided. .

  Furthermore, since the information processing apparatus 10 calculates the depth value of each small region based on the corresponding points, the depth value of each small region can be accurately calculated.

  Further, when there is no corresponding point in the small area, the information processing apparatus 10 calculates the depth value of the small area based on the depth value of the small area existing around the small area. Therefore, the information processing apparatus 10 can calculate the depth value of each small region more accurately even when the number of corresponding points is small.

  Furthermore, the information processing apparatus 10 identifies a superimposable part that exists in a position closer to the imaging device (front side) than the superimposing small area among the parts of the AR image, and combines the superimposable parts, thereby superimposing Generate possible images. The information processing apparatus superimposes the superimposable image on at least one of the right-eye image and the left-eye image. Thereby, the information processing apparatus 10 can generate a composite image in which the perspective of the AR image and the object image is more natural.

  Furthermore, since the information processing apparatus 10 superimposes the superimposable image on both the right eye image and the left eye image, the superimposable image can be stereoscopically displayed.

  The preferred embodiments of the present invention have been described in detail above with reference to the accompanying drawings, but the present invention is not limited to such examples. It is obvious that a person having ordinary knowledge in the technical field to which the present invention pertains can come up with various changes or modifications within the scope of the technical idea described in the claims. Of course, it is understood that these also belong to the technical scope of the present invention.

DESCRIPTION OF SYMBOLS 10 Information processing apparatus 20 Distortion aberration correction part 30 Posture information generation part 40 Depth estimation part 41 Parallax calculation part 42 Area division part 43 Depth calculation part 51 Small area specific | specification part 52 Superimposed image generation part 60 Image composition part 70 Display part

Claims (7)

An image acquisition unit for acquiring a right-eye image and a left-eye image in which the same object is drawn at different horizontal positions;
An area dividing unit that divides a reference image that is at least one of the right-eye image and the left-eye image into a plurality of small areas;
A posture information acquisition unit that acquires posture information indicating a presence position in a real space of a superimposed image that is superimposed on at least one of the right-eye image and the left-eye image;
Based on the posture information, among the plurality of small regions, a small region identifying unit that identifies a superimposed small region on which the superimposed image is superimposed,
A depth calculation unit that calculates a presence position of at least the superimposed small region in real space among the plurality of small regions based on the right-eye image and the left-eye image. Processing equipment.   The information processing apparatus according to claim 1, wherein the region dividing unit divides the reference image into a plurality of small regions by dividing pixels having similar pixel values into the same small region. . A parallax calculating unit that extracts corresponding points indicating the same part of the object from the right-eye image and the left-eye image, and calculates a position of the corresponding point in real space based on the corresponding points;
When the corresponding point is present in the superimposed small area, the depth calculation unit calculates the presence position of the corresponding small area in the real space based on the position of the corresponding point in the real space. The information processing apparatus according to claim 1, wherein the information processing apparatus is an information processing apparatus.   When the corresponding point does not exist in the superimposed small area, the depth calculation unit determines the actual size of the superimposed small area based on the position in the real space of the small area existing around the superimposed small area. The information processing apparatus according to claim 3, wherein an existence position in space is calculated. Based on the posture information, the superimposed small region position information indicating the position of the superimposed small region in real space, and the position information of the imaging device that captured the reference image, Identifying a superimposable portion closer to the imaging device than the superimposing small region, and combining the superimposable portion to generate a superimposable image; and
The information according to any one of claims 1 to 4, further comprising: an image composition unit that superimposes the superimposable image on at least one of the right-eye image and the left-eye image. Processing equipment.   The information processing apparatus according to claim 5, wherein the image composition unit superimposes the superimposable image on both the right-eye image and the left-eye image. An image acquisition step of acquiring a right-eye image and a left-eye image in which the same object is drawn at different horizontal positions;
An area dividing step of dividing a reference image, which is at least one of the right-eye image and the left-eye image, into a plurality of small areas;
A posture information acquisition step of acquiring posture information indicating a presence position in a real space of a superimposed image superimposed on at least one of the right-eye image and the left-eye image;
Based on the posture information, a small area specifying step for specifying a superimposed small area on which the superimposed image is superimposed among the plurality of small areas;
A depth calculation step of calculating a position of at least the superimposed small region in real space among the plurality of small regions based on the right-eye image and the left-eye image. Processing method.

Images