JP2012118948A - Extended reality presentation device, and extended reality presentation method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To freely represent foreground/background arrangement relation among a plurality of objects which are displayed on over another.SOLUTION: An extended reality presentation device 10 recognizes a specified part of a person based upon three-dimensional image data on the person which is acquired through photography, and positions the three-dimensional image data on the person and three-dimensional image data on the plurality of objects based upon a recognition result. Then the extended reality presentation device 10 arranges the three-dimensional image data on the plurality of objects outside the three-dimensional image data on the person according to a positioning result, and arranges and displays three-dimensional image data on a second object outside three-dimensional image data on a first object.

Description

  The present invention relates to a technology capable of synthesizing and presenting real-space video and computer graphics image data.

  In recent years, attempts to try on using AR (Augmented Reality) technology have become widespread. The technique disclosed in Non-Patent Document 1 is to wear a wristwatch-type marker on the wrist and superimpose and display the wristwatch image on the actual arm marker position captured by the camera. In addition to wristwatches, a mechanism to try on a tie using a tie-type marker has been put into practical use. The technique disclosed in Non-Patent Document 2 superimposes and displays an image of sunglasses on the face of a person captured by a camera without wearing a marker.

  Although these conventional techniques can easily try on virtual wearing objects, they are single-on fittings, and it is not envisaged to coordinate virtual wearing objects. For example, if a plurality of markers are attached and fit in one camera image, a virtual image corresponding to each marker is superimposed and displayed at the marker position. Since the marker is hidden and the position and shape are not recognized, the virtual image is not displayed. Like so-called layering, it was not possible to try on AR with a plurality of items stacked one after the other at the same time.

TISSOT REALITY, Internet <URL: http://www.tissot.ch/reality/> RAY-BAN Official Web Site, Internet <URL: http://www.ray-ban.com/france/science/virtual-mirror>

  Also, when considering layering with a shirt and jacket, such a display is not possible even if you want to compare the method of wearing the shirt collar inside the jacket with the method of putting the collar outside the jacket. There wasn't. In AR, a near object is displayed in front of a far object at a relative position from the camera. However, even if such a technique is used, for example, if the jacket is displayed on the front side of the shirt, the shirt image will be completely displayed on the back side of the jacket. I couldn't.

  Accordingly, an object of the present invention is to freely express the positional relationship between the front and back between a plurality of objects displayed in an overlapping manner.

A first aspect of the augmented reality presentation apparatus according to the present invention is an augmented reality presentation apparatus capable of combining and displaying a video of real space and image data, and is a method for displaying a 3 person obtained by photographing. Recognition means for recognizing a predetermined part of the person based on the three-dimensional image data, and alignment of the three-dimensional image data of the person and the three-dimensional image data of a plurality of objects based on the recognition result by the recognition means Positioning means for performing the positioning, and arranging the three-dimensional image data of the plurality of objects outside the three-dimensional image data of the person based on the result of the positioning by the positioning means, and the plurality of objects The 3D image data of the second object included in the plurality of objects is arranged outside the 3D image data of the first object included. And having a display control means for displaying on the device.
A second aspect of the augmented reality presentation device according to the present invention is an augmented reality presentation device capable of combining and displaying a video of real space and image data, and is for a person acquired by photographing. Recognition means for recognizing a predetermined part of the person based on the two-dimensional image data, and alignment of the two-dimensional image data of the person and the two-dimensional image data of a plurality of objects based on the recognition result by the recognition means And a predetermined portion of the two-dimensional image data of the second object included in the plurality of objects that are arranged on the first object included in the plurality of objects. Two-dimensional images of the plurality of objects on the two-dimensional image data of the person based on the result of alignment by the first processing means and the alignment means Display that displays on the display means such that the image data overlaps and the two-dimensional image data of the second object that has been processed by the first processing means overlaps the two-dimensional image data of the first object. And a control means.
A first aspect of the augmented reality presenting method of the present invention is an augmented reality presenting method executed by an augmented reality presenting apparatus capable of combining and displaying a video of real space and image data. A recognition step for recognizing a predetermined part of the person based on the three-dimensional image data of the person acquired by photographing, and the three-dimensional image data of the person and a plurality of objects based on the recognition result of the recognition step An alignment step for aligning with the three-dimensional image data, and arranging the three-dimensional image data of the plurality of objects outside the three-dimensional image data of the person based on the alignment result of the alignment step And included in the plurality of objects outside the three-dimensional image data of the first object included in the plurality of objects. That is characterized by having a display control step of displaying on the display unit by arranging the three-dimensional image data of the second object.
A second aspect of the augmented reality presenting method of the present invention is an augmented reality presenting method executed by an augmented reality presenting apparatus capable of combining and displaying video in real space and image data. A recognition step for recognizing a predetermined part of the person based on the two-dimensional image data of the person acquired by photographing, and the two-dimensional image data of the person and a plurality of objects based on the recognition result of the recognition step An alignment step for performing alignment with the two-dimensional image data, and a two-dimensional image of the second object included in the plurality of objects, which is arranged on the first object included in the plurality of objects. Based on the processing step of punching a predetermined portion of the image data and the alignment result of the alignment step, Two-dimensional image data of the plurality of objects overlaps with the two-dimensional image data, and the two-dimensional image data of the second object that has been cut out by the processing step on the two-dimensional image data of the first object. And a display control step for displaying on the display means so as to overlap each other.
According to a first aspect of the program of the present invention, there is provided a computer for executing an augmented reality presentation method executed by an augmented reality presentation apparatus capable of combining and displaying a video of real space and image data. A recognition step for recognizing a predetermined part of the person based on the three-dimensional image data of the person acquired by photographing, and the three-dimensional image data of the person based on the recognition result of the recognition step An alignment step for aligning the three-dimensional image data of the plurality of objects with the three-dimensional image data of the plurality of objects outside the three-dimensional image data of the person based on the alignment result of the alignment step Outside the 3D image data of the first object included in the plurality of objects in which image data is arranged In, characterized in that to execute a display control step of displaying on the display unit by arranging the three-dimensional image data of a second object included in the plurality of objects in a computer.
A second aspect of the program of the present invention is a program for causing a computer to execute an augmented reality presentation method executed by an augmented reality presentation device capable of combining and displaying a video of real space and image data. A recognition step for recognizing a predetermined part of the person based on the two-dimensional image data of the person acquired by photographing, and the two-dimensional image data of the person based on the recognition result of the recognition step An alignment step for aligning the two-dimensional image data of the plurality of objects with the second object included in the plurality of objects, which is arranged on the first object included in the plurality of objects. A processing step for cutting a predetermined portion of the two-dimensional image data of the object, and a position by the positioning step The two-dimensional image data of the plurality of objects is overlapped on the two-dimensional image data of the person and the two-dimensional image data of the first object is cut out by the processing step based on the result of matching. A display control step of causing the display unit to display the two-dimensional image data of the second object so as to overlap is executed by the computer.

  According to the present invention, it is possible to freely express the positional relationship between the front and back between a plurality of objects displayed in an overlapping manner.

1 is a diagram schematically showing a configuration of an AR display system according to a first embodiment of the present invention. It is a block diagram which shows the functional structure of the AR display system which concerns on the 1st Embodiment of this invention. It is a figure which shows the hardware constitutions of the augmented reality presentation apparatus in embodiment of this invention. It is a flowchart which shows operation | movement of AR display system which concerns on the 1st Embodiment of this invention. It is a figure for demonstrating concretely the arrangement | positioning process of each object in step S205. It is a figure for demonstrating in detail the blanking process in step S205. It is a figure for demonstrating concretely the punching process in step S206. It is a block diagram which shows the functional structure of the AR display system which concerns on the 2nd Embodiment of this invention. It is a flowchart which shows operation | movement of the AR display system which concerns on the 2nd Embodiment of this invention. It is a figure for demonstrating concretely the arrangement | positioning process of each object in step S407. It is a figure for demonstrating in detail the blanking process in the 2nd Embodiment of this invention. It is a figure which shows schematically the structure of the AR display system which concerns on the 3rd Embodiment of this invention.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments to which the invention is applied will be described in detail with reference to the accompanying drawings.

  First, a first embodiment of the present invention will be described. FIG. 1 is a diagram schematically showing the configuration of an AR display system according to the first embodiment of the present invention. As shown in FIG. 1, the AR display system according to the present embodiment inputs a 3D camera 20 that generates 3D image data of a subject by photographing, and 3D image data generated by the 3D camera 20. The augmented reality presentation device 10 is configured to display on the display and to synthesize and display the 3D image data of the object such as clothes at a position matching the 3D image data of the subject. The augmented reality presentation device 10 stores, for example, three-dimensional image data of objects such as shirts and T-shirts, and the three-dimensional image data 30 and 40 of these objects are stored as three-dimensional images of a person who is a subject. By combining and displaying the image data 50 at a position that matches the image data 50, it is possible to present an image in which a person is virtually trying on a shirt or T-shirt. In the following description, such a combination of computer graphics image data displayed on a real-space image to produce an expanded reality may be referred to as AR (Augmented Reality) display. The three-dimensional camera 20 and the augmented reality presentation device 10 are connected by a communication line such as a LAN.

  In the embodiments described below, a PC-type AR display device is cited as an example of an AR display device that is a device that performs AR display, but the present invention is not limited thereto. For example, a mobile terminal type AR display device equipped with a camera function and an image display function can also be applied.

  FIG. 2 is a block diagram showing a functional configuration of the AR display system according to the present embodiment. In FIG. 2, the three-dimensional camera 20 includes a three-dimensional image photographing unit 201 and an image data transmission unit 202. A three-dimensional image photographing unit 201 generates three-dimensional image data of a person by photographing. The image data transmission unit 202 transmits the 3D image data generated by the 3D image capturing unit 201 to the augmented reality presentation device 10.

  The augmented reality presentation device 10 includes an image data receiving unit 101, a human body recognition unit 102, an object storage unit 103, an object search unit 104, an alignment unit 105, an object placement unit 106, a punching processing unit 107, a color designation unit 108, and a display. A control unit 109 is provided.

  The image data receiving unit 101 receives the 3D image data of the person transmitted from the image data transmitting unit 202 of the 3D camera 20. The human body recognition unit 102 performs human body recognition processing such as recognition of a predetermined part of the human body such as the face and both shoulders and the size of the human body based on the three-dimensional image data of the person. The object storage unit 103 stores three-dimensional image data of an object such as a shirt or a T-shirt. Based on the recognition result of the human body recognition unit 102, the object search unit 104 searches for 3D image data of an object that matches the human body. The alignment unit 105 aligns the 3D image data of the person and the 3D image data of the object searched by the object search unit 104 based on the recognition result of the human body recognition unit 102. The object arrangement unit 106 arranges the three-dimensional image data of the object in the virtual three-dimensional space according to the result of the alignment performed by the alignment unit 105. As a result, the person's 3D image data and the 3D image data of each object are arranged at a matching position, and it looks as if the person tried on the object of each attachment.

  The color designation unit 108 corresponds to an operation unit that allows the user to designate an arbitrary color. The punching processing unit 107 punches out the 3D image data portion of each object to be superimposed on the portion corresponding to the color specified by the color specifying unit 108 in the 3D image data of the person. The punching process is to make a target part transparent. For example, in the virtual three-dimensional space, the first object and the second object are arranged so as to overlap in the depth direction (the first object is arranged in the back and the second object is arranged in the foreground). When a part of the second object is punched out, the three-dimensional image data of the part of the first object hidden in the part can be seen through. The display control unit 109 displays each 3D image data arranged in the virtual 3D space on the display 307.

  FIG. 3 is a diagram illustrating a hardware configuration of the augmented reality presentation device 10 according to the present embodiment. In FIG. 3, a CPU 301 comprehensively controls each device and controller connected to the system bus. The ROM 302 stores an operating system (OS) that is a control program for the CPU 301. The HDD 305 stores a program for realizing each function shown in FIG. The RAM 303 functions as a main memory, work area, temporary save area, and the like for the CPU 301. The communication I / F controller 304 executes communication control processing with an external device (for example, the three-dimensional camera 20) via a network. As an alternative to the HDD 305, an SDD (Solid State Drive) may be used. The display controller 306 controls image display on the display 307.

  The image data receiving unit 101 in FIG. 2 has a configuration corresponding to the communication I / F controller 304. The human body recognition unit 102, the object search unit 104, the alignment unit 105, the object placement unit 106, and the punching processing unit 107 are configured to correspond to programs stored in the CPU 301 and the HDD 305. The display control unit 109 has a configuration corresponding to the display controller 306. The object storage unit 103 has a configuration corresponding to the HDD 305. The color designation unit 108 has a configuration corresponding to an operation unit such as a mouse or a keyboard (not shown).

  FIG. 4 is a flowchart showing the operation of the AR display system according to the first embodiment. Hereinafter, the operation of the AR display system according to the present embodiment will be described in detail with reference to FIG.

  In step S101, the three-dimensional image photographing unit 201 photographs a person who is a subject and generates three-dimensional image data of the person. In step S <b> 102, the image data transmission unit 202 transmits the 3D image data of the person generated by the 3D image capturing unit 201 to the augmented reality presentation apparatus 10. In step S <b> 103, the three-dimensional camera 20 determines whether or not the user has performed an operation to end shooting. If there is an operation to end shooting, the process ends. On the other hand, when there is no photographing end operation, the process returns to step S101, and generation and transmission of the three-dimensional image data of the person are repeated.

  In step S <b> 201, the image data receiving unit 101 receives 3D image data of a person from the 3D camera 20. In step S202, the human body recognition unit 102 recognizes a predetermined part of the human body such as the face and both shoulders, the size of the human body, and the like based on the three-dimensional image data of the person. As a method for recognizing a predetermined part of a human body, for example, a method of recognizing these parts by preparing image data of a model face and both shoulders in advance and matching with three-dimensional image data of a person Etc.

  It is also possible to store human skeleton model information in a skeleton model database (not shown) so that unnatural part recognition is not performed on three-dimensional image data captured by a camera. For example, there is a possibility that the background is mistakenly recognized as a human body based on the face portion and background portion of the photographed person image. When misrecognition occurs, the relative position of the face and body shifts, and inconveniences such as the elbow and the knee being bent to the opposite side where they are not bent are judged. Therefore, as the skeleton model information of the human body, the relative position of the face and body, the bending direction and angle of the elbow / knee, etc. are retained, and the human body part suitable for the skeleton model information is specified based on the recognized face etc. It is good to make it.

  In addition, these parts may be recognized by displaying three-dimensional image data of a person on a display and causing the user to input an instruction such as a face or both shoulders. In addition, since the site | part recognized here is used in order to match the position of the person's 3D image data and the object's 3D image data, it is not limited to the face and both shoulders. For example, when the object is pants, the waist, ankle, etc. may be recognized. Examples of the method for recognizing the size of the human body include a method for recognizing based on the subject distance from the person and the size of the photographed image of the person. The size of the human body may be input by the person himself using an operation means such as a keyboard.

  In step S203, the object search unit 104 searches the object storage unit 103 for 3D image data of an object that matches the person based on the 3D image data of the person. Here, the three-dimensional image data of the object matching the size of the human body recognized in step S202 is searched. In addition, since this embodiment represents the wearing state close | similar to the actual layering by AR display, in step S203, the 3D image data of the at least 2 or more object is searched about the said person. To do.

  In step S <b> 204, the alignment unit 105 performs alignment between the 3D image data of the person and the 3D image data of each object searched by the object search unit 104 based on the 3D image data of the person. That is, the alignment unit 105 determines the position of the three-dimensional image data of each object based on the part of the human body recognized by the human body recognition unit 102. For example, when both shoulders are recognized, the arrangement position and orientation of the shirt and the T-shirt are determined so that the shoulders coincide with both shoulder portions of the shirt and the T-shirt object. When the face is recognized, it is determined in which direction the face is facing from the position of the eyes or mouth, and the arrangement position and orientation of the hat are determined so as to match the face orientation.

  In step S <b> 205, the object placement unit 106 places the three-dimensional image data of each object in the virtual three-dimensional space according to the result of the positioning performed by the positioning unit 105. Thereby, it will be in the arrangement state as if a person tried on each object.

  FIG. 5 is a diagram for specifically explaining the arrangement processing of each object in step S205. In FIG. 5, reference numeral 501 represents a state in which only three-dimensional image data of a person is arranged in the virtual three-dimensional space. Reference numeral 502 denotes a state in which the three-dimensional image data of the shirt (first object) is arranged so as to match the three-dimensional image data of the person according to the alignment result of the alignment unit 105. Reference numeral 503 denotes a state in which the 3D image data of the T-shirt (second object) is arranged so as to be aligned with the 3D image data of the person according to the alignment result of the alignment unit 105. At this time, as shown at 503, the three-dimensional image data of the shirt is arranged inside the three-dimensional image data of the T-shirt, but the shirt collar such that the collar portion of the shirt appears outside the T-shirt. It may be a shape. Even in this state, a state close to actual layering is expressed, but some users may want to see a state in which the shirt collar is inside the T-shirt. In such a case, when the user gives an instruction to that effect, the punching processing unit 107 punches out a portion of the shirt collar that overlaps the T-shirt. Thereby, the collar portion of the shirt that has been outside the T-shirt is made transparent, and the shirt collar enters the inside of the T-shirt.

  FIG. 6 is a diagram for explaining the above-described punching process in detail. In FIG. 6, reference numeral 601 denotes the positional relationship between the shirt three-dimensional image data and the T-shirt three-dimensional image data. A broken line 601 represents the position of the three-dimensional image data of the T-shirt. Reference numeral 602 denotes a portion of the shirt collar that overlaps with the T-shirt, that is, a portion of the collar that protrudes outside the T-shirt, and a portion surrounded by an alternate long and short dash line corresponds thereto. The portion surrounded by the alternate long and short dash line is punched out (transparent), so that the portion of the shirt located behind it is displayed transparently. As shown at 504 in FIG. 5, the collar of the shirt is the inner side of the T-shirt. It becomes the state that entered.

  In step S <b> 206, the punching processing unit 107 includes the color designation unit 108 in the human three-dimensional image data in a state where the human three-dimensional image data and the three-dimensional image data of each object are arranged in the virtual three-dimensional space. The part of the three-dimensional image data of each object to be superimposed on the part corresponding to the color designated by the above is processed. For example, when the hair color is designated by the color designation unit 108, the part of the 3D image data of each object to be superimposed on the part of the same color as the hair color (hair part) in the 3D image data of the person Is punched. Thus, when the 3D image data of each object is arranged outside the 3D image data of the person, the hair of the person is always displayed so as to be positioned outside the object.

  FIG. 7 is a diagram for specifically explaining the punching process in step S206. When the 3D image data of the shirt and the 3D image data of the T-shirt are arranged outside the person, a shirt or a T-shirt may appear on the outside of the hair as indicated by 701. . In an actual wearing state, there is almost no shirt or T-shirt appearing on the outside of the hair, so it is desirable to process the three-dimensional image data so as to approximate the wearing state of the actual clothes. Therefore, in the present embodiment, when the hair color is designated by the color designation unit 108, the three-dimensional image data of shirts and T-shirts that are superimposed on the same color portion as the designated color among the three-dimensional image data of the person. The part 7011 is cut and processed. As a result, as shown at 702, the hair portion hidden in the three-dimensional image data of the shirt or T-shirt can be seen, and the hair appears on the outside of the shirt or T-shirt.

  Further, for a specific object, the punching process in step S206 may be invalidated. Specifically, for example, for objects such as hats and hoods that are to be displayed on the hair, the punching process is not executed even if the objects are arranged at positions that overlap the hair. As a result, it is possible to express a state of wearing a hat, a hood, or the like on the head.

  In step S207, the display control unit 109 displays the three-dimensional image data of the person and each object on the display 307. At this time, since the display 307 has a planar shape, conversion processing from three-dimensional image data to two-dimensional image data is performed.

  In step S <b> 208, the augmented reality presentation device 10 determines whether or not there has been a shooting end operation by the user. This determination process may be linked to step S103, or may be performed by the augmented reality presentation device 10 alone. If there is an operation to end shooting, the process ends. On the other hand, if there is no operation for ending photographing, the process returns to step S201, and AR display based on the 3D image data received from the 3D camera 20 is repeated.

  Next, a second embodiment of the present invention will be described. The schematic configuration of the AR display system according to the second embodiment is substantially the same as the configuration shown in FIG. 1, and the three-dimensional camera 20 is replaced with the two-dimensional camera 10, and the augmented reality presentation device 10 is a two-dimensional image. It differs from the first embodiment in having a function corresponding to data.

  FIG. 8 is a block diagram showing a functional configuration of the AR display system according to the present embodiment. In FIG. 8, the two-dimensional camera 80 includes a two-dimensional image capturing unit 801 and an image data transmission unit 802. A two-dimensional image photographing unit 801 generates two-dimensional image data of a person by photographing. The image data transmission unit 802 transmits the 2D image data generated by the 2D image capturing unit 801 to the augmented reality presentation device 90.

  The augmented reality presentation device 90 includes an image data receiving unit 901, a human body recognition unit 902, an object storage unit 903, an object search unit 904, a conversion unit 905, a positioning unit 906, a first blanking unit 907, and an object placement unit 908. , A second punching unit 909, a color designation unit 910, and a display control unit 911.

  The image data receiving unit 901 receives the two-dimensional image data of the person transmitted by the image data transmitting unit 802 of the two-dimensional camera 80. The human body recognition unit 902 performs recognition processing related to the human body, such as recognition of a predetermined part of the person such as the face and both shoulders and the size of the human body based on the two-dimensional image data of the person. The object storage unit 903 stores three-dimensional image data of an object such as a shirt or a T-shirt. Based on the recognition result of the human body recognition unit 902, the object search unit 904 searches for three-dimensional image data of an object that matches the human body. The alignment unit 906 aligns the 3D image data of the person and the 3D image data of each object based on the recognition result of the human body recognition unit 902. The conversion unit 905 converts the three-dimensional image data of each object searched by the object search unit 904 into two-dimensional image data according to the alignment result by the alignment unit 906. The first punching processing unit 907 extracts a predetermined portion of the two-dimensional image data of another object (second object) arranged so as to be superimposed on the two-dimensional image data of a certain object (first object). Process to process. The object arrangement unit 908 arranges the two-dimensional image data of the object according to the result of the alignment performed by the alignment unit 906. As a result, the two-dimensional image data of the person and the two-dimensional image data of each object are arranged at a matching position, and it looks as if the person tried on the object of each attachment.

  The color designation unit 910 corresponds to an operation unit that allows the user to designate an arbitrary color. The second blanking unit 909 cuts and processes the part of the two-dimensional image data of each object to be superimposed on the part corresponding to the color designated by the color designation unit 910 in the two-dimensional image data of the person. The two-dimensional image data blanking process is the same as the three-dimensional image data blanking process. For example, in a state where the first object and the second object are arranged so as to overlap each other in the depth direction (a state where the first object is located at the back and the second object is located at the front), the second object When a certain part is punched, the two-dimensional image data of the part of the first object hidden in the part can be seen through. The display control unit 911 displays each arranged two-dimensional image data on the display 912.

  Note that the hardware configuration of the augmented reality presentation device 90 in the second embodiment is the same as the configuration shown in FIG. That is, the image data receiving unit 901 in FIG. 8 has a configuration corresponding to the communication I / F controller 304. The human body recognition unit 902, the object search unit 904, the conversion unit 905, the alignment unit 906, the first blanking unit 907, the object placement unit 908, and the second blanking unit 909 are stored in programs stored in the CPU 301 and the HDD 305. Corresponding configuration. The display control unit 911 has a configuration corresponding to the display controller 306. The object storage unit 903 has a configuration corresponding to the HDD 305. The color designation unit 910 has a configuration corresponding to an operation unit such as a mouse or a keyboard (not shown).

  FIG. 9 is a flowchart showing the operation of the AR display system according to the second embodiment. Hereinafter, the operation of the AR display system according to the present embodiment will be described in detail with reference to FIG.

  In step S301, the two-dimensional image capturing unit 801 captures a person who is a subject and generates two-dimensional image data of the person. In step S <b> 302, the image data transmission unit 802 transmits the two-dimensional image data of the person generated by the two-dimensional image photographing unit 801 to the augmented reality presentation device 90. In step S <b> 303, the two-dimensional camera 80 determines whether or not there has been an operation for ending shooting by the user. If there is an operation to end shooting, the process ends. On the other hand, when there is no photographing end operation, the process returns to step S301, and generation and transmission of the two-dimensional image data of the person are repeated.

  In step S <b> 401, the image data receiving unit 901 receives two-dimensional image data of a person from the two-dimensional camera 80. In step S402, the human body recognition unit 902 recognizes a predetermined part of the human body such as the face and both shoulders, the size of the human body, and the like based on the two-dimensional image data of the person. As a method for recognizing a part of a human body, the same method as in the first embodiment can be mentioned. Moreover, it is the same as that of 1st Embodiment that the site | part to recognize is not limited to a face and both shoulders. Further, the human body size recognition method is the same as that in the first embodiment.

  In step S403, the object search unit 904 searches the object storage unit 903 for 2D image data of an object that matches the person based on the 2D image data of the person. The object search method here is the same as in the first embodiment. Also in this embodiment, since the wearing state close to the actual layering is expressed by AR display, in step S403, two-dimensional image data of at least two or more objects is searched for the person. And

  In step S <b> 404, the alignment unit 906 performs alignment between the two-dimensional image data of the person and the two-dimensional image data of each object searched by the object search unit 904 based on the two-dimensional image data of the person. That is, the alignment unit 906 determines the position of the three-dimensional image data of each object based on the part of the human body recognized by the human body recognition unit 902. For example, when both shoulders are recognized, the arrangement position and orientation of the shirt and the T-shirt are determined so that the shoulders coincide with both shoulder portions of the shirt and the T-shirt object. When the face is recognized, it is determined in which direction the face is facing from the position of the eyes or mouth, and the arrangement position and orientation of the hat are determined so as to match the face orientation.

  In step S405, the conversion unit 905 converts the 3D image data of each object searched by the object search unit 904 in accordance with the arrangement position and orientation of the 3D image data of each object determined by the alignment unit 906. Convert to 2D image data. That is, the conversion unit 905 converts the three-dimensional image data into the two-dimensional image data so that each object can be seen in the above-described direction at the arrangement position on the display 307.

  In step S <b> 406, the first punching processing unit 907 determines predetermined two-dimensional image data of another object (second object) that is arranged on the two-dimensional image data of a certain object (first object). The process of punching out the part is performed. In step S407, the object arrangement unit 908 arranges the two-dimensional image data of each object according to the result of the alignment by the alignment unit 906. The two-dimensional image data arranged here includes two-dimensional image data of the object that has been subjected to the punching process by the first punching unit 907.

  FIG. 10 is a diagram for specifically explaining the arrangement processing of each object in step S407. In FIG. 10, reference numeral 1001 denotes a state in which only two-dimensional image data of a person is arranged. Reference numeral 1002 denotes a state in which the two-dimensional image data of the shirt (first object) is arranged so as to be aligned with the two-dimensional image data of the person in accordance with the alignment result of the alignment unit 906. Reference numeral 1003 denotes a state in which the two-dimensional image data of the T-shirt (second object) is arranged so as to be aligned with the two-dimensional image data of the person according to the alignment result of the alignment unit 906. In the state shown in 1003, since the 2D image data of the T-shirt is simply superimposed on the 2D image data of the shirt, the 2D image data of the T-shirt is superimposed on the 2D image data of the shirt. All the parts to do are hidden. Accordingly, it is uncomfortable for a user who assumes that the shirt collar is inside the T-shirt and the shirt collar is outside the T-shirt in an actual layered state. AR display is displayed. Therefore, in this embodiment, when the user gives an instruction to view the layered shirt with the shirt collar placed outside the T-shirt, the first punching section 907 displays the shirt collar portion. Of these, the part that overlaps the T-shirt is punched out. As a result, the portion of the shirt that was inside the T-shirt is made transparent, and the shirt collar comes out of the outside of the T-shirt. When considering the relationship between the shirt and the jacket, if the two-dimensional image data of the jacket is simply placed in front of the two-dimensional image data of the shirt, most of the two-dimensional image data of the shirt is hidden in the two-dimensional image data of the jacket. It will not be in a state of being layered. Therefore, when the punching process as in the present embodiment is performed on the corresponding portion of the two-dimensional image data of the jacket, the two-dimensional image data of the back shirt can be seen, and it is in a state of being layered.

  FIG. 11 is a diagram for explaining the above-described punching process in detail. In FIG. 11, reference numeral 1101 denotes the positional relationship between the shirt 3D image data and the T shirt 3D image data. Simply overlay the T shirt 2D image data on the shirt 2D image data. It represents the state. Note that a broken line 1101 represents a collar portion of the shirt hidden behind the T-shirt. Reference numeral 1102 denotes a portion where a T-shirt and a shirt collar overlap, that is, a portion of a T-shirt to be punched, and a portion surrounded by an alternate long and short dash line corresponds thereto. The portion surrounded by the alternate long and short dash line is punched out (transparent), so that the collar portion of the shirt is transparently displayed, and as shown by 1004 in FIG. 10, the collar of the shirt comes out of the T-shirt. It will be in such a state.

  In step S408, the second blanking unit 909 is designated by the color designation unit 910 out of the two-dimensional image data of the person in a state where the two-dimensional image data of the person and the two-dimensional image data of each object are arranged. The portion of the two-dimensional image data of each object that overlaps the portion corresponding to the selected color is processed. Thereby, as demonstrated using FIG. 7, two-dimensional image data can be processed into the wearing state close | similar to an actual layering. Also in the second embodiment, similarly to the first embodiment, the blanking process in step S206 may be invalidated for a specific object.

  In step S409, the display control unit 911 displays the two-dimensional image data of the person and each object on the display 307. In step S <b> 410, the augmented reality presentation device 90 determines whether or not the user has performed a shooting end operation. This determination process may be linked to step S303, or may be performed by the augmented reality presentation device 90 alone. If there is an operation to end shooting, the process ends. On the other hand, if there is no operation for ending photographing, the process returns to step S401, and AR display based on the two-dimensional image data received from the two-dimensional camera 80 is repeated.

  Next, a third embodiment of the present invention will be described. In the first and second embodiments, the augmented reality presentation device automatically searches for an object suitable for the person (human body) based on the person's image data (3D image data, 2D image data). However, the present invention is not limited to this. That is, a person's image data (3D image data, 2D image data) is transmitted to an external communication terminal, and a person who uses the external device recommends an object suitable for the person (human body) based on the image data. You may make it do.

  FIG. 12 is a diagram schematically showing a configuration of an AR display system according to the third embodiment. As shown in FIG. 12, the AR display system according to the present embodiment uses a 3D camera 1220 that generates 3D image data of a person who is a subject by photographing, and 3D image data generated by the 3D camera 1220. A store PC 1230 that receives the augmented reality presentation device 1210 and returns the three-dimensional image data of the object recommended from the store side based on the three-dimensional image data to the augmented reality presentation device 1210, and the person stores the store PC 1230. Purchased product management table 1240 for managing the history of products (objects) purchased at the store where the product is installed, 3D image data generated by the 3D camera 1220, 3D image data of each object received from the store PC 1230, And an augmented reality presentation device 1210 for composing and displaying the image on the display. It is. The augmented reality presentation device 1210 and the store PC 1230 are connected via the Internet.

  The operation of the AR display system according to the third embodiment will be described below. The three-dimensional image data of the person generated by the three-dimensional camera 1220 is transmitted to the store PC 1230 via the augmented reality presentation device 1210. Store PC 1230 displays the three-dimensional image data of the person on the display. On the store side, referring to the 3D image data, a history of objects that the person has purchased so far is searched from the purchased product management table 1240 connected to the store PC 1230, and an object that suits (applies to) the person. Select. When the object is selected, the store PC 1230 acquires the three-dimensional image data of each selected object from the purchased product management table 1240 and transmits it to the augmented reality presentation device 1210. Upon receiving the 3D image data of each object, the augmented reality presentation device 1210 synthesizes and displays the 3D image data of each object at a position that matches the 3D image data of the person. The process for the composite display is the same as steps S201 to S208 in FIG. Here, the description has been made on the three-dimensional image data, but when the two-dimensional image data is the target, the same processing as that in steps S401 to S410 in FIG. 9 is applied to the processing for composite display. The image data of the person transmitted from the three-dimensional camera or the two-dimensional camera to the store PC may be recorded or real-time.

  As another embodiment, an object that suits the person is not recommended only from an object purchased at a single store, but an object that suits the person may be recommended from objects purchased at a plurality of stores. . In this case, for example, attachments purchased at a plurality of stores are stored in the closet, and the RFID of each attachment can be read by attaching an RFID to a hanger that suspends each attachment in the closet. Keep it. Then, by sending the read ID of each attachment, the image data of each attachment corresponding to the ID, and the image data of the person to the friend's PC, the attachment can be recommended from the friend. . In addition, the management location of the image data of the attachment corresponding to each ID may not be the person side. When the friend's PC receives each ID, the corresponding image data may be acquired from an external site using the ID as a key.

  According to the above-described embodiment, it is possible to freely express the positional relationship between the front and back between a plurality of objects displayed in an overlapping manner. More specifically, when the objects of a plurality of wearing objects are overlapped and displayed in AR, it can be expressed in a state close to the actual wearing, and a part of the wearing object can be represented by another wearing object. It is possible to freely select whether to display in front of or behind the object. In the above-described embodiment, clothes are exemplified as the attachments. However, the present invention can also be applied to cars as attachments for people in a broad sense. Thereby, the state which test-rided the desired vehicle can be browsed by AR display.

  Although detailed description is omitted in the above-described embodiment, the shape of the object to be superimposed can be changed in accordance with the movement of the person by creating the image data of the object to be superimposed by using so-called polygon data or the like. Can do. For example, when a person moves from a state where his arms are stretched to a state where his elbows are bent, the superimposed object such as a shirt can be deformed into a shape where his elbows are bent. As a result, it is possible to eliminate the unnaturalness such as the sleeve of the shirt being stretched and displayed in a superimposed manner even though the person is actually bending the elbow.

  DESCRIPTION OF SYMBOLS 10: Augmented reality presentation apparatus, 101: Image data receiving part, 102: Human body recognition part, 103: Object storage part, 104: Object search part, 105: Position alignment part, 106: Object arrangement part, 107: Extraction process part , 108: color designation unit, 109: display control unit, 20: three-dimensional camera, 201: three-dimensional image photographing unit, 202: image data transmission unit, 80: two-dimensional camera, 801: two-dimensional image photographing unit, 802: Image data transmission unit, 90: augmented reality presentation device, 901: image data reception unit, 902: human body recognition unit, 903: object storage unit, 904: object search unit, 905: conversion unit, 906: alignment unit, 907 : First punching section, 908: object placement section, 909: second punching section, 910: color designation section, 911: display control section

Claims (17)

An augmented reality presentation device capable of combining and displaying real space video and image data,
Recognition means for recognizing a predetermined part of the person based on the three-dimensional image data of the person acquired by photographing;
Alignment means for aligning the three-dimensional image data of the person and the three-dimensional image data of a plurality of objects based on the recognition result by the recognition means;
Based on the result of the alignment by the alignment means, the 3D image data of the plurality of objects is arranged outside the 3D image data of the person, and the first object included in the plurality of objects An augmented reality presentation device comprising: display control means for arranging and displaying on a display means three-dimensional image data of a second object included in the plurality of objects outside the three-dimensional image data. Processing means for cutting a predetermined portion of the three-dimensional image data of the second object;
The display means displays the 3D image data of the second object that has been punched and processed by the processing means, arranged outside the 3D image data of the first object. Augmented reality presentation device described in 1.   3. The predetermined portion is a part of a portion where the three-dimensional image data of the first object and the three-dimensional image data of the second object overlap each other. Augmented reality presentation device. It further has a designation means capable of designating an arbitrary color,
2. The processing unit according to claim 1, wherein the processing unit extracts and processes a part of the three-dimensional image data of the plurality of objects to be superimposed on a part of the three-dimensional image data of the person having a color specified by the specifying unit. The augmented reality presentation device according to any one of 1 to 3.   5. The search unit according to claim 1, further comprising: a search unit that searches the plurality of objects in which the positioning unit performs positioning of the three-dimensional image data based on information relating to the person. Augmented reality presentation device described in 1. Receiving means for receiving information on the plurality of objects recommended based on the information on the person from an external device;
5. The alignment unit according to claim 1, wherein the alignment unit performs alignment between the 3D image data of the person and the 3D image data of the plurality of objects received by the reception unit. The augmented reality presentation apparatus according to item 1.   The augmented reality presentation apparatus according to claim 5, wherein the information related to the person is three-dimensional image data of the person. An augmented reality presentation device capable of combining and displaying real space video and image data,
Recognizing means for recognizing a predetermined part of the person based on the two-dimensional image data of the person acquired by photographing;
Alignment means for aligning the two-dimensional image data of the person and the two-dimensional image data of a plurality of objects based on the recognition result by the recognition means;
First processing means for extracting and processing a predetermined portion of the two-dimensional image data of the second object included in the plurality of objects, which is arranged on the first object included in the plurality of objects; ,
Based on the result of the alignment by the alignment means, the 2D image data of the plurality of objects overlaps the 2D image data of the person, and the 2D image data of the first object overlaps the 2D image data of the first object. An augmented reality presentation apparatus, comprising: display control means for causing the display means to display the two-dimensional image data of the second object that has been punched and processed by one processing means.   9. The predetermined part is a part of a part where two-dimensional image data of the first object and two-dimensional image data of the second object overlap. Augmented reality presentation device. A designation means that can designate any color;
The image processing apparatus further includes second processing means for extracting and processing the two-dimensional image data portions of the plurality of objects to be superimposed on the two-dimensional image data portion of the person having the color specified by the specifying means. The augmented reality presentation device according to claim 8 or 9.   11. The apparatus according to claim 8, further comprising a search unit that searches the plurality of objects in which the alignment unit performs alignment of two-dimensional image data based on information relating to the person. Augmented reality presentation device described in 1. Receiving means for receiving information on the plurality of objects recommended based on the information on the person from an external device;
11. The alignment unit according to claim 8, wherein the alignment unit performs alignment between the two-dimensional image data of the person and the two-dimensional image data of the plurality of objects received by the reception unit. The augmented reality presentation apparatus according to item 1.   The augmented reality presentation apparatus according to claim 11, wherein the information related to the person is two-dimensional image data of the person. An augmented reality presentation method executed by an augmented reality presentation device capable of combining and displaying a video of real space and image data,
A recognition step for recognizing a predetermined part of the person based on the three-dimensional image data of the person acquired by photographing;
An alignment step for aligning the 3D image data of the person and the 3D image data of a plurality of objects based on the recognition result of the recognition step;
Based on the result of the alignment in the alignment step, the 3D image data of the plurality of objects is arranged outside the 3D image data of the person, and the first object included in the plurality of objects An augmented reality presentation method comprising: a display control step of arranging and displaying on a display means three-dimensional image data of a second object included in the plurality of objects outside the three-dimensional image data. An augmented reality presentation method executed by an augmented reality presentation device capable of combining and displaying a video of real space and image data,
A recognition step for recognizing a predetermined part of the person based on the two-dimensional image data of the person acquired by photographing;
An alignment step of aligning the two-dimensional image data of the person and the two-dimensional image data of a plurality of objects based on the recognition result of the recognition step;
A processing step of punching out a predetermined portion of the two-dimensional image data of the second object included in the plurality of objects, which is arranged on the first object included in the plurality of objects;
Based on the result of the alignment in the alignment step, the two-dimensional image data of the plurality of objects overlaps the two-dimensional image data of the person, and the processing is performed on the two-dimensional image data of the first object. An augmented reality presentation method comprising: a display control step of causing the display means to display the two-dimensional image data of the second object that has been punched and processed in steps. A program for causing a computer to execute an augmented reality presentation method executed by an augmented reality presentation device capable of combining and displaying a video of real space and image data,
A recognition step for recognizing a predetermined part of the person based on the three-dimensional image data of the person acquired by photographing;
An alignment step for aligning the 3D image data of the person and the 3D image data of a plurality of objects based on the recognition result of the recognition step;
Based on the result of the alignment in the alignment step, the 3D image data of the plurality of objects is arranged outside the 3D image data of the person, and the first object included in the plurality of objects A program for causing a computer to execute a display control step in which 3D image data of a second object included in the plurality of objects is arranged outside the 3D image data and displayed on a display unit. A program for causing a computer to execute an augmented reality presentation method executed by an augmented reality presentation device capable of combining and displaying a video of real space and image data,
A recognition step for recognizing a predetermined part of the person based on the two-dimensional image data of the person acquired by photographing;
An alignment step of aligning the two-dimensional image data of the person and the two-dimensional image data of a plurality of objects based on the recognition result of the recognition step;
A processing step of punching out a predetermined portion of the two-dimensional image data of the second object included in the plurality of objects, which is arranged on the first object included in the plurality of objects;
Based on the result of the alignment in the alignment step, the two-dimensional image data of the plurality of objects overlaps the two-dimensional image data of the person, and the processing is performed on the two-dimensional image data of the first object. A program for causing a computer to execute a display control step of causing the display means to display the two-dimensional image data of the second object that has been punched and processed in steps.

Images