JP2012216077A - Augmented reality providing device and superimposition method of virtual object - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an augmented reality providing device capable of providing augmented reality with no sense of incongruity, without using a marker.SOLUTION: A ground surface estimation part 5 detects a face from each of two imaged image data obtained by taking images by imaging parts 1, 2, and calculates a three-dimensional coordinate of a triangle connecting eyes and a mouth of the detected face. The ground surface estimation part 5 extracts a contour of a subject including the detected face from one of the two imaged image data, and performs back projection of the contour in three-dimensional space using the three-dimensional coordinate of the triangle. The ground surface estimation part 5 estimates a plane which includes an end of the contour in a direction to which a vertical line of a digital camera 100 extends in the three-dimensional space and which is perpendicular to the direction, to be a ground surface. The ground surface estimation part 5 superimposes a virtual object on the estimated ground surface and displays them, in a stereoscopic image displayed in a display part 11.

Description

  The present invention relates to an augmented reality providing apparatus that provides augmented reality and a virtual object superimposing method.

  Augmented reality technology that superimposes and displays a virtual object on an image of a scene taken by a camera has become widespread, and mobile phones and head mounted displays (HMD) that employ this technology have appeared.

  For example, in Patent Document 1, an image including a marker obtained by imaging with a digital camera is displayed on a monitor, and a three-dimensional object corresponding to the marker is displayed on a marker of a card displayed on the monitor. An augmented reality providing apparatus that superimposes and displays an image (virtual object) is described.

  In the apparatus described in Patent Document 1, a position where a virtual object is to be superimposed is determined by detecting a marker. For this reason, a virtual object can be superimposed only on an image including a marker, and versatility is low.

  Japanese Patent Application Laid-Open No. 2004-228561 describes a technique for specifying a positional relationship of a subject based on a photographer's viewpoint position, line-of-sight direction, and the like and superimposing a virtual object on a captured image. According to this technique, a virtual object can be superimposed on a captured image without using a marker.

JP 2000-322602 A JP 2008-78908 A

  However, since the technique described in Patent Document 2 cannot grasp the position of the ground in the captured image, the virtual object can be superimposed only at an appropriate angle with respect to the ground, giving the user a sense of incongruity. May end up.

  The present invention has been made in view of the above circumstances, and provides an augmented reality providing apparatus and a virtual object superimposing method capable of providing an augmented reality without using a marker without using a marker. Objective.

  The augmented reality providing device of the present invention is an augmented reality providing device that provides augmented reality to a user, and is one of two captured image data generated from two captured image signals having parallax. A first image based on certain first imaged image data, and a display unit that compositely displays a second image based on the second imaged image data that is the other of the two imaged image data in a stereoscopic manner; A face detection process for detecting a face from the first captured image data and the second captured image data, and face data detected in common from the first captured image data and the second captured image data are used. The face coordinate calculation process for calculating the coordinates of the face in a three-dimensional space with the augmented reality providing device as the origin, the contour of the subject including the face from the first captured image data or the second captured image data Extract A contour extraction process, a two-dimensional coordinate of the contour extracted by the contour extraction process, a coordinate conversion process for converting the coordinate of the face calculated by the face coordinate calculation process into a coordinate in the three-dimensional space, and the tertiary A ground estimation unit that performs a ground estimation process for estimating a plane perpendicular to the vertical line including an end of the outline in the original space in the augmented reality providing device in a direction in which the vertical line extends, and the display unit A display control unit that performs control to superimpose and display a virtual object on the ground estimated by the ground estimation unit in the stereoscopic image displayed on the screen.

  The virtual object superimposing method of the present invention includes a first image based on first captured image data that is one of two captured image data generated from two captured image signals having a parallax, and the two captured images. A second image based on the second captured image data, which is the other of the image data, is combined and displayed in a stereoscopic view, and a virtual object is superimposed on the stereoscopic image displayed in the stereoscopic view to provide augmented reality. A method of superimposing a virtual object in an augmented reality providing apparatus, wherein a face detection step of detecting a face from the first captured image data and the second captured image data, the first captured image data, and the A face coordinate calculation step of calculating the coordinates of the face in a three-dimensional space using the augmented reality providing device as an origin using face data commonly detected from second captured image data; The contour extraction step for extracting the contour of the subject including the face from the image image data and the second captured image data, and the two-dimensional coordinates of the contour extracted by the contour extraction step are calculated by the face coordinate calculation step. A coordinate conversion step for converting into coordinates in the three-dimensional space based on the coordinates of the face; and an end of the contour in the three-dimensional space in the direction in which the vertical line of the augmented reality providing device extends and the vertical line A ground estimation step for estimating a vertical plane as a ground; and a display control step for performing control to superimpose and display a virtual object on the ground estimated by the ground estimation step in the stereoscopic image.

  According to the present invention, it is possible to provide an augmented reality providing apparatus and a virtual object superimposing method capable of providing an augmented reality without a sense of incongruity without using a marker.

1 is an external view of a digital camera as an example of an augmented reality providing device for explaining an embodiment of the present invention. 1 is a block diagram showing the internal configuration of the digital camera 100 shown in FIG. The flowchart for demonstrating the operation | movement at the time of the augmented reality mode of the digital camera 100 shown by FIG. The figure for demonstrating the operation | movement at the time of the augmented reality mode of the digital camera 100 shown by FIG. The figure for demonstrating the operation | movement at the time of the augmented reality mode of the digital camera 100 shown by FIG. The figure for demonstrating the operation | movement at the time of the augmented reality mode of the digital camera 100 shown by FIG. The figure for demonstrating the operation | movement at the time of the augmented reality mode of the digital camera 100 shown by FIG. The figure for demonstrating the operation | movement at the time of the augmented reality mode of the digital camera 100 shown by FIG. The figure for demonstrating the operation | movement at the time of the augmented reality mode of the digital camera 100 shown by FIG. The figure which shows the example of the imaging | photography state at the time of the augmented reality mode of the digital camera 100 shown by FIG. The figure which shows the example of the imaging | photography state at the time of the augmented reality mode of the digital camera 100 shown by FIG. 1 is a block diagram showing an internal configuration of a digital camera 200 that is a modification of the digital camera 100 shown in FIG. Flowchart for explaining the operation of the digital camera 200 shown in FIG. 12 in the augmented reality mode.

  Hereinafter, an augmented reality providing device for explaining an embodiment of the present invention will be described with reference to the drawings. This augmented reality providing device has two left and right imaging units arranged at a distance, and the right and left captured image data obtained by imaging with each imaging unit can be stereoscopically viewed on the display unit Are combined and displayed. Further, this augmented reality providing device displays a virtual object formed by computer graphics superimposed on a stereoscopic image displayed on the display unit. According to this augmented reality providing device, the display unit displays an image in which a virtual object is superimposed on the object scene image being shot. Therefore, augmented reality is provided to a person observing the display unit. Can be given.

  FIG. 1 is an external view of a digital camera as an example of an augmented reality providing device for explaining an embodiment of the present invention. A digital camera 100 shown in FIG. 1 includes a box-shaped housing K, and a right-eye imaging unit 1 and a left-eye image provided on the front surface of the housing K at a distance in the horizontal direction (longitudinal direction of the housing K). And the release button 16a provided on the upper surface of the housing K. The imaging unit 1 includes a photographic lens 1R, and the imaging unit 2 includes a photographic lens 1L. The angle at which the optical axis R of the photographic lens 1R intersects the optical axis L of the photographic lens 1L is called the convergence angle, and the positions of the photographic lens 1R and the photographic lens 1L are set so that the convergence angle becomes a predetermined value. Yes.

  FIG. 2 is a block diagram showing an internal configuration of the digital camera 100 shown in FIG.

  As shown in FIG. 2, the digital camera 100 includes an imaging unit 1, an image input controller 3, an image processing unit 4, a ground estimation unit 5, a virtual object database (DB) 6, and a digital camera 100. A central processing unit (CPU) 8 for overall control, a random access memory (RAM) 9 used as a work memory, a display control unit 10 to which a display unit 11 is connected, and a detachable recording medium 13 The recording control unit 12 connected, the audio control unit 14 connected to the microphone and the speaker 15, and the operation unit 16 (including the release button 16a in FIG. 1) connected to the CPU 8 are provided. The other parts are connected to each other by a bus 17.

  The imaging units 1 and 2 are an optical system such as a photographing lens and a diaphragm, an imaging device such as a CCD image sensor and a CMOS image sensor, and an analog signal processing unit that performs analog signal processing on a captured image signal output from the imaging device. In addition, an AD conversion unit that digitally converts the signal processed by the analog signal processing unit, a drive unit that drives the imaging device, and the like are included. The imaging unit 1 and the imaging unit 2 are arranged at a distance in the horizontal direction in order to obtain two captured image signals with parallax, and this horizontal direction is the direction in which the digital camera 100 obtains parallax. .

  The image input controller 3 takes captured image signals output from the imaging units 1 and 2 and stores the captured image signals in the RAM 9.

  The image processing unit 4 performs well-known digital signal processing such as synchronization processing (processing for giving three color data to one pixel), white balance processing, RGB-YC conversion processing, etc., on the captured image signal stored in the RAM 9. To generate captured image data. Of the captured image data generated here, data generated from the captured image signal output from the imaging unit 1 is referred to as right captured image data, and generated from the captured image signal output from the imaging unit 2. This is called left captured image data. The right captured image data and the left captured image data are stored in the RAM 9.

  The display control unit 10 causes the display unit 11 to display a right image based on the right captured image data generated by the image processing unit 4 and a left image based on the left captured image data generated by the image processing unit 4. A composite display is made so as to enable stereoscopic viewing.

  The display unit 11 is configured by a liquid crystal display device or the like that supports a time-division parallax image method, a lenticular method, a parallax barrier method, and the like, and displays a right image and a left image alternately, for example, under control of the display control unit 10 By doing so, a stereoscopically viewable video (stereoscopic video) is displayed.

  In the virtual object DB 6, virtual object data to be superimposed on the stereoscopic video displayed on the display unit 11 is registered in advance. As virtual object data, right image virtual data to be superimposed on the right image and left image virtual data to be superimposed on the left image are registered.

  The ground estimation unit 5 estimates the position of the ground in the three-dimensional space where the digital camera 100 is placed. In this three-dimensional space, the center of gravity of the digital camera 100 is set as the origin, and a straight line passing through the origin and extending in the longitudinal direction of the housing K (direction in which parallax is obtained) is defined as the Y axis. , 2 is defined as the X axis as a straight line extending in the direction perpendicular to the front surface of the housing K (photographing direction), and the straight line passing through the origin and perpendicular to the XY plane (straight line extending in the short direction of the housing K) is defined as Z. It is a space defined by the coordinates as axes. In addition, each captured image data obtained by imaging by the imaging units 1 and 2 corresponds to a subject projected on the YZ plane of the three-dimensional space, and each pixel of each captured image data corresponds to each captured image data. The position is specified by coordinates (y, z) with the pixel at the center as the origin.

  The display control unit 10 superimposes and displays a virtual object image based on the virtual object data acquired from the virtual object DB 6 at a position corresponding to the ground estimated by the ground estimation unit 5 in the stereoscopic image being displayed on the display unit 11. Control is also performed. The display control unit 10 superimposes and displays a virtual object image based on the right image virtual data on the right image, and superimposes and displays a virtual object image based on the left image virtual data on the left image. The virtual object image is superimposed and displayed.

  The digital camera 100 captures images with the image capturing unit 1 and the image capturing unit 2, and associates the right captured image data and the left captured image data generated by the image capturing in a stereoscopically viewable format and records them in the recording medium 13. In addition to the mode, the imaging unit 1 and the imaging unit 2 capture a subject including a person, and a right image based on the right captured image data generated by the imaging and a left image based on the left captured image data are used as a stereoscopic video. While displaying, it is possible to set the augmented reality mode that provides the user with augmented reality by superimposing the virtual object image on the stereoscopic video, and the operation of the digital camera 100 in the augmented reality mode will be described below.

  FIG. 3 is a flowchart for explaining the operation of the digital camera 100 shown in FIG. 1 in the augmented reality mode. 4 to 9 are diagrams for explaining the operation of the digital camera 100 shown in FIG. 1 in the augmented reality mode. The following description is based on the premise that the digital camera 100 is photographed in a state where the Z-axis direction is perpendicular to the ground.

  When the digital camera 100 is set to the augmented reality mode, moving image shooting is started by the imaging unit 1 and the imaging unit 2, and the object scene image being shot is displayed on the display unit 11 as a stereoscopic image.

  When moving image shooting starts, the image processing unit 4 generates right captured image data and left captured image data for each frame and stores them in the RAM 9. The right captured image data and the left captured image data stored in the RAM 9 are displayed on the display unit 11 by the display control unit 10 so as to be stereoscopically viewable. Each time the right captured image data and the left captured image data are generated, the ground estimation unit 5 performs face detection processing on each of the right captured image data and the left captured image data stored in the RAM 9 (step S1). .

  If the face is not detected as a result of the face detection process (step S2: NO), the ground estimation unit 5 performs the process of step S1 again on the right captured image data and the left captured image data of the next frame, and the face is detected. If it is detected (step S2: YES), the process of step S3 is performed.

  For example, as shown in FIG. 4, when the user of the digital camera 100 determines the angle of view so that a person is included in the stereoscopic image being captured displayed on the display unit 11, right captured image data including the person is included. Then, a face is detected from the left captured image data, and the process of step S3 is performed.

  In step S3, the ground estimation unit 5 performs a process of extracting eyes and mouth from faces detected from each of the right captured image data and the left captured image data.

  When two eyes and one mouth are extracted in step S3 (step S4: YES), the ground estimation unit 5 performs the process of step S5. When two eyes and one mouth are not extracted in step S3 (step S4: NO), the ground estimation unit 5 performs the process of step S1 on the right captured image data and the left captured image data of the next frame.

  In step S5, the ground estimation unit 5 obtains a triangle 50 connecting the centers of the two eyes and one mouth extracted from each of the right captured image data and the left captured image data, as shown in FIG. The parallax between the triangle 50 obtained for the right captured image data and the triangle 50 obtained for the left captured image data, the position of both triangles 50 on the captured image data, the imaging units 1 and 2 of the digital camera 100 The coordinates of the triangle 50 in the three-dimensional space are calculated using design values (such as the convergence angle and the base line length).

  In step S5, as shown in FIG. 6, the ground estimation unit 5 coordinates (a1, b1, c1), (a2, b2, c2), (a3) of the three vertices of the triangle 50 in the three-dimensional space. , B3, c3).

  Next, the ground estimation unit 5 extracts the contour of the subject including the face detected from the right captured image data or the left captured image data from the right captured image data or the left captured image data by a known technique (step S6).

  For example, as illustrated in FIG. 7, the ground estimation unit 5 extracts a person's outline 71 (rectangular area) included in the right captured image data 70, and coordinates data ((y 1, z 1) of the four vertices of the outline 71. ), (Y2, z2), (y3, z3), (y4, z4)). In addition, the ground estimation unit 5 coordinates data ((y5, z5), (3) of the triangle 50 connecting the eyes and mouth included in the face of the person included in the right captured image data 70 extracted in step S3. y6, z6), (y7, z7)) are acquired.

  The coordinate data ((y5, z5), (y6, z6), (y7, z7)) converted into the coordinates of the three-dimensional space is the coordinate data ((a1, b1) of the triangle 50 shown in FIG. , C1), (a2, b2, c2), (a3, b3, c3)). The ground estimation unit 5 calculates coordinate data ((a1, b1, c1), (a2, b2, c2), (a3,) from the coordinate data ((y5, z5), (y6, z6), (y7, z7)). b3, c3)) is obtained, and the coordinate data ((y1, z1), (y2, z2), (y3, z3), (y4, z4)) are three-dimensionally obtained according to this coordinate transformation formula. The coordinates are converted into space coordinates (step S7).

  By the process of step S7, as shown in FIG. 8, the coordinates of the contour 71 in the three-dimensional space are obtained.

  Next, the ground estimation unit 5 includes the lower end of the contour 71 in the three-dimensional space shown in FIG. 8 (the lower side of the contour 71 in FIG. 8) and is perpendicular to the Z-axis direction. A simple plane G is estimated as the ground in the three-dimensional space (step S8).

  When the ground is estimated in step S8, the display control unit 10 reads the data of the virtual object from the virtual object DB 6, and displays the stereoscopic image in which the virtual object is arranged on the ground estimated in step S8 on the display unit 11. It is displayed (step S9). Since the stereoscopic video displayed on the display unit 11 is a reproduction of the three-dimensional space as a video, if the ground in the three-dimensional space can be estimated, the position of the ground in the stereoscopic video can also be estimated. Through the process of step S9, the user looking at the display unit 11 can see a stereoscopic video in which a virtual object is superimposed on the estimated ground in the stereoscopic video of the left image and the right image.

  FIG. 9 is a diagram illustrating an example of a screen displayed on the display unit 11, and an example of a screen in which a virtual object image 91 is superimposed on the estimated ground 90 in the stereoscopic video for the stereoscopic video being captured. Is shown. By viewing such a screen, the user can enjoy an augmented reality without any sense of incongruity. In FIG. 9, the ground 90 is illustrated, but this is for explanation, and the ground 90 is not actually displayed on the display unit 11.

  In the above description, as shown in FIG. 10A and FIG. 11A, it is assumed that shooting is performed such that the Z axis of the digital camera 100 is perpendicular to the ground. Actually, as shown in FIG. 10B and FIG. 11B, photographing may be performed in a state where the Z axis of the digital camera 100 is not perpendicular to the ground.

  When shooting is performed in the state shown in FIG. 10B or FIG. 11B, the ground estimation unit 5 uses the plane perpendicular to the Z axis of the digital camera 100 as the ground in step S8 of FIG. If estimated, the estimated ground is inclined with respect to the actual ground, and the user feels uncomfortable when the virtual object image is superimposed. A configuration of the digital camera for preventing such a sense of incongruity will be described below.

  FIG. 12 is a block diagram showing an internal configuration of a digital camera 200 which is a modification of the digital camera 100 shown in FIG. The digital camera 200 shown in FIG. 12 is the same as the block diagram shown in FIG. 2 except that the attitude sensor 20 is added.

  The attitude sensor 20 detects the attitude of the digital camera 200, and is composed of, for example, a gyro sensor. The posture sensor 20 detects the inclination of the Z-axis with respect to the ground passing through the center of gravity of the digital camera 200 (angle θ in FIGS. 10B and 11B), and notifies the ground estimation unit 5 of the detected angle information. To do. Hereinafter, the operation of the digital camera 200 in the augmented reality mode will be described.

  FIG. 13 is a flowchart for explaining the operation of the digital camera 200 shown in FIG. 12 in the augmented reality mode. In FIG. 13, the same processes as those shown in FIG.

  After performing the processing up to step S7, the ground estimation unit 5 acquires angle information indicating the tilt of the digital camera 200 with respect to the ground from the posture sensor 20 (step S20).

  Next, the ground estimation unit 5 uses the angle information acquired in step S20 to drop the vertical line of the digital camera 200 in the three-dimensional space (when the weight is attached to the center of gravity of the digital camera 200, the weight hangs down). Direction) is determined (step S21).

  Since the angle information acquired from the attitude sensor 20 is information on an angle with respect to the Z axis in the three-dimensional space, the vertical line direction of the digital camera 200 in the three-dimensional space can be determined by using this angle information.

  Next, the ground estimation unit 5 is a plane perpendicular to the vertical line direction determined in step S21 and includes an end in the vertical line direction in the outline of the three-dimensional space converted in step S7. Is estimated as the ground in the three-dimensional space (step S22).

  For example, as shown in FIG. 10B and FIG. 11B, when the contour in the three-dimensional space exists as indicated by reference numeral 110, the edge of the contour 110 in the vertical line direction of the digital camera 200 is shown. A plane G including 110T and perpendicular to the vertical direction is estimated as the ground.

  Note that when the user is shooting in a state as shown in FIGS. 10A and 11A, the inclination θ of the digital camera 200 is 0 °, and therefore Z in the three-dimensional space is set. The lower side in the axial direction coincides with the vertical direction of the digital camera 200. Therefore, in this case, the ground estimation unit 5 estimates the ground by the same process as the process of step S8 of FIG. 3 in step S22.

  After step S22, in step S9, a virtual object is superimposed and displayed on the estimated ground in the stereoscopic image being displayed on the display unit 11.

  As described above, according to the digital camera 200, since the ground can be accurately estimated even when the digital camera 200 is tilted with respect to the ground, it is possible to provide an augmented reality without any sense of incongruity. Is possible.

  Up to this point, the number of faces detected from the captured image data is assumed to be one. However, when a plurality of types of faces are detected from the captured image data (in step S4 in FIG. 13, 2). When the extraction of the eyes and mouths of more than one set of faces is successful, the ground estimation unit 5 selects one of the plurality of types of faces, and then performs step S5 in FIG. 13 for the selected face. The process of ~ S9 may be performed. Which one of the types of faces to select may be specified by the user, or a person having a high degree of coincidence with a pre-registered face after personal recognition is selected. Also good.

  In addition, when a plurality of types of faces are detected from the captured image data (when two or more sets of face eyes and mouths have been successfully extracted in step S4 in FIG. 13), the ground estimation unit 5 uses the plurality of types of faces. After selecting two of the faces, the processes of steps S5 to S22 in FIG. 13 are performed on each of the two selected faces, and the two ground planes obtained for each of the two selected faces are displayed. Using the coordinates, the ground having the highest correlation with the two grounds is estimated as the final ground (the ground on which the display control unit 10 superimposes the virtual object), so that the ground estimation accuracy can be increased.

  The digital cameras 100 and 200 obtain two captured image signals with parallax from the two imaging units of the imaging units 1 and 2, but if two captured image signals with parallax can be obtained, the imaging unit There may not be two.

  For example, an imaging unit that outputs two captured image signals with parallax using a single optical system and two imaging elements may be used. Alternatively, an imaging unit that can output two captured image signals with parallax using a single optical system and a single imaging device may be used. In an imaging unit capable of outputting two captured image signals with parallax using a single optical system and a single imaging device, a pixel pair capable of detecting a phase difference (for example, light-shielding film openings in opposite directions to each other) By using two-dimensionally arranged pixels), two captured image signals having parallax can be output. In addition, an image pickup unit of a type that acquires two picked-up image signals with parallax by performing image pickup a plurality of times while continuously changing the shooting range may be used.

  In the above description, the augmented reality providing device has an imaging unit capable of outputting two captured image signals with parallax. However, the augmented reality providing device may not include an imaging unit. Good. For example, two captured image signals with a parallax (still images, moving images) obtained by imaging with another imaging unit, downloaded from the Internet, or the like to a storage unit (recording medium 13) in the augmented reality providing device Two captured image signals with a parallax (still image, moving image) are stored in advance, and the ground estimation unit of the augmented reality providing device acquires the two captured image signals from the recording medium 13, and acquires the acquired two The processing shown in FIGS. 3 and 13 may be performed on the captured image signal.

  As described above, the following items are disclosed in this specification.

  The disclosed augmented reality providing device is an augmented reality providing device that provides augmented reality to a user, and is one of two captured image data generated from two captured image signals having parallax. A first image based on certain first imaged image data, and a display unit that compositely displays a second image based on the second imaged image data that is the other of the two imaged image data in a stereoscopic manner; A face detection process for detecting a face from the first captured image data and the second captured image data, and face data detected in common from the first captured image data and the second captured image data are used. The face coordinate calculation process for calculating the coordinates of the face in a three-dimensional space with the augmented reality providing device as the origin, the contour of the subject including the face from the first captured image data or the second captured image data The Contour extraction processing to be output, coordinate conversion processing for converting the two-dimensional coordinates of the contour extracted by the contour extraction processing into coordinates in the three-dimensional space based on the coordinates of the face calculated by the face coordinate calculation processing, and A ground estimation unit for performing ground estimation processing for estimating a plane perpendicular to the vertical line including an end of the outline in the augmented reality providing device of the contour in a three-dimensional space and perpendicular to the vertical line; and the display A display control unit that performs control to superimpose and display a virtual object on the ground estimated by the ground estimation unit in the stereoscopic video displayed on the unit.

  The disclosed augmented reality providing device includes a posture detecting unit that detects a posture of the augmented reality providing device, and the ground estimation unit is configured to detect an inclination of the augmented reality providing device detected by the posture detecting unit. Based on the information, the direction in which the vertical line extends is determined.

  In the disclosed augmented reality providing device, when there are a plurality of types of faces detected by the face detection process, the ground estimation unit selects one of the plurality of types of faces and includes the selected face The face is estimated by performing the face coordinate calculation process, the contour extraction process, the coordinate conversion process, and the ground estimation process on the subject.

  In the disclosed augmented reality providing device, when there are a plurality of types of faces detected by the face detection process, the ground estimation unit selects two of the plurality of types of faces and selects the selected two faces The ground coordinate is estimated for each of the two types of subjects by performing the face coordinate calculation process, the contour extraction process, the coordinate conversion process, and the ground estimation process for different types of subjects including A plane having the highest correlation with the two grounds obtained for each type of subject is estimated as a final ground on which the virtual object is superimposed, and the display control unit is configured to determine the final ground estimated by the ground estimation unit. The virtual object is superimposed on a smooth ground.

  The disclosed augmented reality providing apparatus includes an imaging unit capable of outputting two captured image signals having the parallax.

  The disclosed method for superimposing a virtual object includes a first image based on first captured image data that is one of two captured image data generated from two captured image signals having parallax, and the two imaging A second image based on the second captured image data, which is the other of the image data, is combined and displayed in a stereoscopic view, and a virtual object is superimposed on the stereoscopic image displayed in the stereoscopic view to provide augmented reality. A method of superimposing a virtual object in an augmented reality providing apparatus, wherein a face detection step of detecting a face from the first captured image data and the second captured image data, the first captured image data, and the A face coordinate calculation step of calculating the coordinates of the face in a three-dimensional space using the augmented reality providing device as an origin using face data commonly detected from second captured image data; The contour extracting step for extracting the contour of the subject including the face from the captured image data and the second captured image data, and the two-dimensional coordinates of the contour extracted by the contour extracting step are calculated by the face coordinate calculating step. A coordinate conversion step for converting into coordinates in the three-dimensional space based on the coordinates of the face; and an end of the contour in the three-dimensional space in the direction in which the vertical line of the augmented reality providing device extends and the vertical line A ground estimation step for estimating a vertical plane as a ground; and a display control step for performing control to superimpose and display a virtual object on the ground estimated by the ground estimation step in the stereoscopic image.

  The disclosed method for superimposing a virtual object includes a posture detection step of detecting a posture of the augmented reality providing device, and in the ground estimation step, the inclination of the augmented reality providing device detected by the posture detection step is detected. Based on the information, the direction in which the vertical line extends is determined.

  In the disclosed method for superimposing a virtual object, when there are a plurality of types of faces detected by the face detection step, one of the plurality of types of faces is selected, and the subject including the selected face is The ground is estimated by performing a face coordinate calculation step, the contour extraction step, the coordinate conversion step, and the ground estimation step.

  In the disclosed method for superimposing a virtual object, when there are a plurality of types of faces detected by the face detection step, two types are selected from the plurality of types of faces, and different types including each of the selected two faces For the subject, the face coordinate calculation step, the contour extraction step, the coordinate conversion step, and the ground estimation step are performed to estimate the ground for each of the two types of subjects and to obtain each of the two types of subjects. The plane having the highest correlation with the two grounds is estimated as the final ground on which the virtual object is superimposed, and the virtual object is superimposed on the final ground.

1, 2 Imaging unit 5 Ground estimation unit 200 Digital camera

Claims (9)

An augmented reality providing device that provides augmented reality to a user,
A first image based on first captured image data that is one of two captured image data generated from two captured image signals having parallax, and a second image that is the other of the two captured image data A display unit that synthesizes and displays a second image based on the image data in a stereoscopic manner;
A face detection process for detecting a face from the first captured image data and the second captured image data, and face data detected in common from the first captured image data and the second captured image data are used. The face coordinate calculation process for calculating the coordinates of the face in a three-dimensional space with the augmented reality providing device as the origin, the contour of the subject including the face from the first captured image data or the second captured image data A contour extraction process for extracting the two-dimensional coordinates of the contour extracted by the contour extraction process, and a coordinate conversion process for converting the coordinates of the face calculated by the face coordinate calculation process into coordinates in the three-dimensional space; Performing ground estimation processing for estimating a plane perpendicular to the vertical line that includes an end of the outline in the augmented reality providing device in the three-dimensional space in a direction in which the vertical line extends. And the ground estimation unit,
An augmented reality providing apparatus comprising: a display control unit that performs control to superimpose and display a virtual object on the ground estimated by the ground estimation unit in the stereoscopic image displayed on the display unit. The augmented reality providing device according to claim 1,
An attitude detection unit for detecting the attitude of the augmented reality providing device;
The augmented reality providing device, wherein the ground estimation unit determines a direction in which the vertical line extends based on information on an inclination of the augmented reality providing device detected by the posture detecting unit. The augmented reality providing device according to claim 1 or 2,
When there are a plurality of types of faces detected by the face detection process, the ground estimation unit selects one of the plurality of types of faces and performs the face coordinate calculation process for a subject including the selected face. An augmented reality providing apparatus that estimates the ground by performing the contour extraction process, the coordinate conversion process, and the ground estimation process. The augmented reality providing device according to claim 1 or 2,
When there are a plurality of types of faces detected by the face detection process, the ground estimation unit selects two of the plurality of types of faces and applies different types of subjects including each of the selected two faces. Then, the face coordinates calculation process, the contour extraction process, the coordinate conversion process, and the ground estimation process are performed to estimate the ground for each of the two types of subjects, and the two grounds obtained for each of the two types of subjects The plane most highly correlated with is estimated as the final ground on which the virtual object is superimposed,
The augmented reality providing apparatus in which the display control unit superimposes the virtual object on the final ground estimated by the ground estimation unit. The augmented reality providing device according to any one of claims 1 to 4,
An augmented reality providing apparatus including an imaging unit capable of outputting two captured image signals having parallax. A first image based on first captured image data that is one of two captured image data generated from two captured image signals having parallax, and a second image that is the other of the two captured image data The second image based on the image data is combined and displayed so as to be stereoscopically viewable, and the virtual object in the augmented reality providing device that provides the augmented reality by superimposing the virtual object on the stereoscopically displayed stereoscopic image is provided. A superposition method comprising:
A face detection step of detecting a face from the first captured image data and the second captured image data;
Face coordinates for calculating the coordinates of the face in a three-dimensional space with the augmented reality providing device as the origin using face data detected in common from the first captured image data and the second captured image data A calculation step;
A contour extracting step for extracting a contour of a subject including the face from the first captured image data and the second captured image data;
A coordinate conversion step of converting the two-dimensional coordinates of the contour extracted by the contour extraction step into coordinates in the three-dimensional space based on the coordinates of the face calculated by the face coordinate calculation step;
A ground estimation step for estimating a plane perpendicular to the vertical line including an end portion of the outline in the augmented reality providing device in the three-dimensional space in a direction in which the vertical line extends;
A virtual object superimposing method comprising: a display control step of performing control to superimpose and display a virtual object on the ground estimated by the ground estimation step in the stereoscopic video. The method of superimposing a virtual object according to claim 6,
A posture detecting step for detecting a posture of the augmented reality providing device;
In the ground estimation step, a virtual object superimposing method for determining a direction in which the vertical line extends based on information on an inclination of the augmented reality providing device detected in the posture detection step. The method of superimposing a virtual object according to claim 6 or 7,
When there are a plurality of types of faces detected by the face detection step, one of the plurality of types of faces is selected, and for the subject including the selected face, the face coordinate calculation step, the contour extraction step, A virtual object superimposing method for estimating the ground by performing the coordinate conversion step and the ground estimation step. The method of superimposing a virtual object according to claim 6 or 7,
When there are a plurality of types of faces detected by the face detection step, two of the plurality of types of faces are selected, and the face coordinate calculation is performed for different types of subjects including each of the selected two faces. Steps, the contour extraction step, the coordinate conversion step, and the ground estimation step are performed to estimate the ground for each of the two types of subjects and have the highest correlation with the two grounds obtained for the two types of subjects. A method of superimposing a virtual object by estimating a plane as a final ground on which the virtual object is superimposed, and superimposing the virtual object on the final ground.

Images