JP5597087B2 - Virtual object manipulation device - Google Patents

Description

  The present invention relates to a virtual object manipulation device that manipulates a virtual object arranged in a virtual three-dimensional space constructed using computer graphics technology.

  In general, at product exhibitions and briefing sessions, product mock-up models and clay models are exhibited, and prototypes of products are exhibited to allow visitors to experience. Therefore, there is a problem that it takes time to create an exhibit and it takes a lot of time to prepare the exhibit.

  For this type of problem, a virtual object arranged in a virtual three-dimensional space constructed using computer graphics technology may be presented instead of an exhibit. If a virtual object is used instead of an exhibit, it is not necessary to create an actual exhibit, so that it is possible to reduce the time and effort required for preparation. In order to create such a virtual object, three-dimensional shape data is often generated by using software for three-dimensional CAD.

  In addition, as a technique for generating three-dimensional shape data, a technique using image data obtained by capturing an object is also known (see, for example, Patent Document 1). In the technique described in Patent Document 1, three-dimensional shape data regarding an object is generated using a plurality of image data captured from various directions regarding the object.

  On the other hand, it is considered to create a real solid model by processing a real space workpiece based on such three-dimensional shape data (see, for example, Patent Document 2). Patent Document 2 also describes that deformation, correction, integration, and processing of three-dimensional shape data generated by the technique described in Patent Document 1 are performed.

  Patent Document 2 does not specifically describe what kind of work is performed in order to perform deformation, correction, integration, and processing of three-dimensional shape data. However, if it is a normal technique, it is estimated that the graphic software provided with the three-dimensional function is used, and the operation is performed on the three-dimensional shape data by operating a pointing device such as a mouse.

  By the way, it is considered that a moving image of a distance image having a pixel value as a distance to an object is generated, a specific part of a person is extracted from the distance image, and a time change of the specific part is recognized as a gesture (for example, (See Patent Document 3). In the technique described in Patent Document 3, the control target device is controlled by the gesture by giving the control target device a control output associated with the gesture.

Japanese Patent Application No. 10-124704 JP 2001-166809 A JP 2006-99749 A

  The three-dimensional shape data can be generated using image data obtained by capturing an object as in the technique described in Patent Document 1, and can be generated using three-dimensional CAD software. In addition, as described in Patent Document 2, it is possible to perform deformation, correction, integration, and processing of three-dimensional shape data. For this work, a virtual three-dimensional space displayed on the monitor device is used. It is common to operate a pointing device such as a mouse or digitizer.

  A normal monitor device has only a two-dimensional plane display area, and a pointing device can only indicate a position on the plane. Accordingly, it is difficult to match the work on the virtual three-dimensional space displayed on the monitor device with the sense of operation of the pointing device, and it is necessary to be skilled in performing the above-described work on the three-dimensional shape data as desired. is necessary.

  Similarly, when a motion is given to a virtual object represented by three-dimensional shape data in the virtual three-dimensional space, it is difficult to give an instruction with a pointing device.

  On the other hand, the technique described in Patent Document 3 recognizes the movement of a specific part of a person as a gesture, and controls a control target device according to the content of the gesture. However, in Patent Document 3, the content of the gesture is used as a switch for the device to be controlled, and is not considered to be used for operations such as transformation, correction, integration, and processing of three-dimensional shape data. The configuration used for this work is also not shown. That is, Patent Document 3 describes that the movement of a specific part of a person is reflected in the operation of the device to be controlled, but does not describe a technique for reflecting it in a virtual object existing in a virtual three-dimensional space. Absent.

  An object of the present invention is to provide a virtual object manipulating apparatus in which an action caused by movement of an object in real space is reflected on a virtual object in a virtual three-dimensional space.

The present invention includes a distance image generating means for generating a distance image of the object present in the real space, the distance and the input information obtaining means for extracting a specific portion of the object from the image, a virtual three-dimensional movement of the specific portion Interaction means for acting on a virtual object in space and reflecting the result on the display on the display means , wherein the interaction means uses the pixel value of the specific part extracted by the input information acquisition means. Projecting means for associating with an index arranged in a virtual three-dimensional space constructed by a computer, object arranging means for arranging a virtual object in the virtual three-dimensional space, and the object arranging means arranged in the virtual three-dimensional space Tool providing means for providing a processing tool that exerts a predetermined action on the virtual object, and processing provided by the tool providing means for the index extracted by the input information acquisition means Tool combining means for combining tools, and the display means according to the movement of the index in a state in which a processing tool is combined with the index associated with the virtual three-dimensional space by the projection means by the tool combining means. Display processing means for reflecting the result of the action by the processing tool on the virtual object displayed on the object, the object placement means has a function of placing the virtual object in the virtual three-dimensional space, and a background on the virtual object. And a function of arranging a first virtual object having a prescribed shape and a second virtual object covering the virtual object in the virtual three-dimensional space as the virtual object. And the tool providing means, as a processing tool, pastes a predetermined opaque texture onto the virtual object existing at the passage site. , An erasing tool for erasing the second virtual object present at the passage site, and a deformation tool for adsorbing the index to the surface of the virtual object and deforming the virtual object according to the direction of movement of the index The tool combination means selects a processing tool to be combined with the index .

  In this case, a pattern classification unit that classifies the motion pattern of the specific part using the pixel value of the specific part extracted by the input information acquisition unit is provided, and the tool combination unit includes a pattern classification in which the processing tool is associated with the motion pattern. Preferably, a processing tool corresponding to the motion pattern classified by the means is combined with the index.

  The display processing means may change the display state of the penetrated virtual object from the display state of other virtual objects when the deformation tool penetrates the virtual object when the virtual object is deformed by the deformation tool.

  The object is a human body, the specific part includes at least an extremity and a head, and the virtual object is a three-dimensional model including the extremity and the head, and the tool providing means uses the movement of the extremity and the head of the human body as a processing tool. You may make it provide the interlocking | linkage tool linked with the movement of the limb and head of a three-dimensional model.

  The specific part is a part whose position in the real space changes in accordance with the movement of at least one of the limbs and the head of the human body, and the input information acquisition means determines the distance and the specific part from the display means of the specific part. The direction of movement of the part is extracted, and the display processing means changes the range of the virtual three-dimensional space displayed on the display means so as to follow the position of the specific part extracted by the input information acquisition means, and the display means The range of the virtual three-dimensional space and the direction of the coordinate axis displayed on the screen may be changed so as to follow the direction of movement of the specific part extracted by the input information acquisition unit.

  The object is a human body and the specific part is the head, the input information acquisition means extracts the position and orientation of the head, and the display processing means inputs the range of the virtual three-dimensional space displayed on the display means. The head position extracted by the information acquisition means is changed to follow the viewpoint determined by the position of the head, and the orientation of the coordinate axis of the virtual three-dimensional space displayed on the display means is changed to the position of the head extracted by the input information acquisition means. You may make it change according to the direction of the gaze determined by direction.

  The display means preferably performs stereoscopic display of the virtual three-dimensional space.

  According to the configuration of the present invention, a specific part of an object in the real space is combined with a processing tool that has a predetermined action on the virtual object in the virtual three-dimensional space, and the action caused by the movement of the object in the real space is reflected in the virtual object. Can be made. As a result, the virtual object in the virtual three-dimensional space can be operated intuitively without using a pointing device, and there is an advantage that the virtual object can be operated without requiring skill. .

It is a block diagram which shows embodiment. It is a figure which shows an example of the human body model used for the same as the above. It is a block diagram which shows the structural example of the distance image generation means used for the same as the above. It is a figure which shows the usage example same as the above. It is a figure which shows the other usage example same as the above. It is a figure which shows the other usage example same as the above. It is a figure which shows another example of use same as the above. It is a figure which shows another usage example same as the above.

  In the embodiments described below, “objects” in real space include human bodies and articles that humans can wear. In addition, a pointing rod, a writing instrument, a glove and the like are assumed as articles worn by a person. When the object is a human body, the “specific part” of the object includes a hand, extremities, a head, and the like, and an article such as a pointing stick or a writing instrument means a tip portion. Hereinafter, a case where the object is a human body and the specific part is one of the left and right hands will be exemplified.

  As shown in FIG. 1, the present embodiment includes a distance image generation unit 10, and the distance image generation unit 10 generates a distance image having a pixel value as a distance from an object existing in a predetermined space area in real space. To do. The predetermined spatial region is determined within a range in which the distance image generating unit 10 can detect the distance to the object. The configuration of the distance image generation unit 10 will be described later.

  The distance image generated by the distance image generation means 10 is given to the input information acquisition means 21, and the input information acquisition means 21 extracts a human hand from the distance image. In order to extract the human hand from the distance image, the input information acquisition unit 21 first extracts a region where the human body exists based on the distance difference from the background. As will be described later, the distance image generation means 10 detects the position of the object in the real space using an orthogonal coordinate system set in the real space. Therefore, a region in which a human body exists is extracted by defining a reference plane in this orthogonal coordinate system and using a distance from the reference plane.

  For example, an orthogonal coordinate system is defined by two coordinate axes defining a plane corresponding to the floor surface and one coordinate axis orthogonal to the floor surface, and the plane corresponding to the wall surface is used as the reference plane described above. With respect to the distance from the reference plane to the object, the outline of the object corresponding to the human body can be extracted by performing the edge enhancement process in the same manner as the outline extraction process in the grayscale image. For edge enhancement, an evaluation value that evaluates the magnitude of the local change in the distance value is obtained using the pixel values around the pixel of interest in the same manner as in the differentiation of the grayscale image, and the pixel having the maximum evaluation value is edged. Let it be the upper pixel.

  If the outline of the human body is extracted, each part of the human body can be separated by applying the extracted outline to an appropriate human body model 30 as shown in FIG. The illustrated human body model 30 includes a segment 31 corresponding to a bone and a joint 32 corresponding to a joint. Here, the human body model 30 using 15 joints is illustrated, but a human body model 30 having another configuration may be used.

  The human body model 30 can be applied to the human body existing in the real space by extracting a plurality of feature points appropriately determined based on the contour line obtained from the distance image. Thus, by applying to the human body model 30, each part of the human body existing in the real space can be separated and recognized. Here, the limbs of the human body, the head, and the torso are recognized separately, and the hand is recognized by applying the arm to the human body model 30.

  It is desirable that the input information acquisition means 21 also recognizes the hand shape. In order to recognize the shape of the hand, it is necessary to apply the outline of the hand extracted from the distance image to the skeleton model of the hand. Therefore, it is desirable to track the position of the hand and enlarge the region including the hand. Here, as the shape of the hand, it is only necessary to recognize whether the hand is open or gripped and which finger is bent.

  As described above, the input information acquisition unit 21 recognizes the hand in the real space by applying the human body existing in the real space to the human body model 30. Further, the input information acquisition unit 21 obtains the position of the hand in the orthogonal coordinate system set in the real space by the distance image generation unit 10. The coordinate position of the hand in the real space thus obtained is associated with the coordinate position of the virtual three-dimensional space constructed by the computer. An appropriate index is defined in the virtual three-dimensional space, and a hand is associated with the index by the projection means 22. In other words, the projection means 22 associates the hand in the real space with the index in the virtual reality space, so that the index in the virtual three-dimensional space can be moved by moving the hand in the real space.

  Incidentally, an appropriate virtual object is arranged in the virtual three-dimensional space by the object arrangement means 23. The virtual three-dimensional space is displayed on the display means 25 through the display processing means 24. As the display means 25, a configuration for performing a three-dimensional stereoscopic display may be employed in addition to a configuration of a normal monitor device that performs a two-dimensional planar display. The display means 25 that performs stereoscopic display uses a configuration using goggles with a shutter that opens and closes in synchronization with the displayed video, a display that allows stereoscopic viewing when viewed with the left and right eyes without using goggles, and goggles There is a head-mounted display with a built-in indicator. An arrangement example of the virtual object will be described later.

  By the way, in the projection means 22, since the hand of real space is matched with the parameter | index prescribed | regulated to virtual three-dimensional space, the parameter | index in virtual three-dimensional space can be moved by moving a hand. Therefore, the hand can be used in place of the mouse by handling the index in the same manner as the mouse cursor. However, no action can be exerted on the virtual three-dimensional space simply by moving the index following the movement of the hand.

  Therefore, the tool providing means 27 is provided to provide a processing tool that exerts a predetermined action on the virtual object arranged in the virtual three-dimensional space, and the processing tool provided from the tool providing means 27 is obtained by the tool combining means 28 in hand. Are connected. With this configuration, when the hand is moved in the real space, the processing tool coupled to the hand acts on the virtual object placed in the virtual three-dimensional space, and the processing determined for the processing tool is performed.

  Although the tool providing means 27 may be configured to provide only one type of processing tool, it is usually preferable to provide a plurality of types of processing tools. Moreover, it is preferable that the tool coupling | bonding means 28 selects the processing tool couple | bonded with a parameter | index (corresponding to a hand) from a plurality of types of processing tools.

  In order to select a processing tool to be combined with a hand from a plurality of types of processing tools, a tool palette in which processing tools are arranged is displayed on the screen of the display means 25 as in the case of ordinary graphic software, and the processing tool is selected from the tool palette. Just choose.

  Further, instead of using the tool palette, the processing tool may be selected using a motion pattern (which may include a shape pattern) of a specific part of the human body such as a hand detected by the projection unit 22. In this case, a movement pattern is registered in advance in the pattern classification unit 29, and when the registered movement pattern is detected by the pattern classification unit 29, the tool combination unit 28 selects a processing tool corresponding to the movement pattern as an index (hand). (Corresponding to) may be combined. The shape pattern means a pattern such as whether the hand is open or gripped and which finger is bent.

  The pattern classification unit 29 uses the pixel value of the hand (specific part) acquired by the input information acquisition unit 21 to classify the hand movement pattern based on the temporal change in the position of the representative point (for example, the center of gravity) obtained from the pixel value. To do. A plurality of types of hand movement patterns are registered in advance, and a code is assigned to each type. In the tool combination means 28, by supplying this code to the tool providing means 27, a processing tool corresponding to the code can be combined with the index.

  If a code is also given to the processing tool in the above-described tool palette and is matched with the code of the pattern classification means 29, the selection of the processing tool using the hand movement pattern can be selected from the tool palette. It becomes the same processing. That is, the same code is given to the instruction of the same processing tool regardless of whether the instruction is based on the hand movement pattern or the tool palette, and the processing tool to be combined with the index is selected in the same procedure. be able to.

  The result of the action on the virtual object by the processing tool is reflected in the display content on the display means 25 by the display processing means 24. That is, in a state where the processing tool is coupled to the index (corresponding to the hand) associated with the virtual three-dimensional space by the projecting means 22, the processing tool acts on the virtual object according to the movement of the hand, and the result is displayed. This is reflected in the display of the means 25.

  With the above-described configuration, the input information acquisition unit 21 acquires the movement of the human body existing in the real space, and the movement is applied to the virtual object arranged in the virtual three-dimensional space, and the result is displayed on the display unit 25. It can be reflected. Therefore, the projecting means 22, the object arranging means 23, the display processing means 24, the tool providing means 27, and the tool combining means 28 are the movements of the human body in the real space and the virtual objects in the virtual three-dimensional space acquired by the input information acquiring means 21. It functions as an interactive means of interactively combining The input information acquisition unit 21 and the dialogue unit are configured using a computer.

  More specific description will be given below. The distance image generated by the distance image generating means 10 is an image having a pixel value as a distance value, and various techniques for generating a distance image are known. As the passive type distance image generating means 10, a stereo image method is known in which the distance to an object is obtained based on the parallax of a plurality of imaging devices. Further, as the active type distance image generating means 10, a light cutting method for obtaining the distance to the object based on the principle of the triangulation method, and the time until the light is projected and the reflected light from the object is received is measured. The time-of-flight method is widely adopted. There is also known a distance image generation means 10 that projects a plurality of types of light patterns having different spatial phases and obtains the three-dimensional shape of the surface of the object from the positional relationship of the light patterns formed on the surface of the object.

  In the embodiment described below, there is no particular limitation on the technology for generating a distance image, but a case of using a time-of-flight method will be described as an example. Hereinafter, the time-of-flight method is abbreviated as “TOF method”. Various configurations of the distance image generation means 10 using the TOF method are known. Here, the time from when the intensity-modulated light, which is the intensity of light modulated by a modulated signal with a fixed period, is projected to the target space, which is real space, and the intensity-modulated light reflected by the object existing in the target space is received Is detected as a phase difference between light projection and reception of intensity-modulated light, and this phase difference is converted into a distance to an object.

  The light projected into the target space is preferably light that is reflected by the surface of the object without passing through many objects and is not perceived by humans. For this reason, it is desirable to use near infrared rays as the light to be projected. However, visible light can be used when it is desirable to be perceived by a person, such as when adjusting the imaging region.

  The waveform of the intensity-modulated light is assumed to be a sine wave, but a triangular wave, a sawtooth wave, a square wave, or the like can be used. When a sine wave, a triangular wave, or a sawtooth wave is used, the period of the intensity modulated light is set to a constant period. In addition, when using a square wave, in addition to making the period of intensity modulated light constant, the ratio of the on period (light emitting source light emitting period) to the off period (light emitting source non-light emitting period) is randomized. It is also possible to adopt changing technology. In other words, the on period and the off period are irregularly changed so that the probability of occurrence of the on period is 50% in a sufficiently long time with respect to the on period and the off period, and the light reception accumulated in the sufficiently long time. An amount may be used.

In addition, when the intensity-modulated light has a fixed period, for example, the light to be projected is modulated by a 20 MHz modulation signal, and the received light amount of about 10,000 periods is accumulated to reduce the influence of shot noise. Even when the on period and the off period are changed randomly, for example, the unit period is set to a period (5 × 10 ⁻⁸ s) corresponding to one cycle of 20 MHz, and the on period is in the range of several times the unit period. And the off period are changed, and the received light amount of about 10,000 times the unit period is accumulated. With this operation, the amount of received light after accumulation can be handled in the same manner as when the amount of received light is accumulated using intensity-modulated light having a fixed period.

  The intensity-modulated light reflected by the object is received by an imaging device in which a plurality of pixels are two-dimensionally arranged. The imaging apparatus includes an imaging element for capturing a grayscale image and a light receiving optical system that limits a range in which light enters the light receiving surface of the imaging element. As the image pickup element, an image pickup element having a well-known configuration for picking up a gray image provided as a CCD image sensor or a CMOS image sensor can be used. However, the image pickup element is specially designed to have a structure suitable for the distance image generation unit 10. It is desirable to use an image sensor.

  In the following description, the following configuration is assumed as an example of the distance image generation unit 10, but not limited to this configuration, but the modulation waveform of intensity-modulated light, the configuration of the image sensor, the control of the image sensor, and the like are well known. The various configurations provided in the various distance image generation means 10 can be adopted.

  As shown in FIG. 3, the distance image generation means 10 used in the following description includes a light emitting element 11 that projects light into a target space, and an imaging element 12 that receives light from the target space. As the light emitting element 11, an element such as a light emitting diode or a laser diode that can obtain an optical output proportional to an instantaneous value of input is used. Further, the light emitted from the light emitting element 11 is projected through the light projecting optical system 13. An appropriate number of light emitting elements 11 are provided to ensure light output. A light receiving optical system 14 that determines the field of view is disposed in front of the image sensor 12.

  The intensity-modulated light emitted from the light emitting element 11 is projected onto a desired spatial region through the light projecting optical system 13. The image sensor 12 receives light from the target space through the light receiving optical system 14. The light projecting optical system 13 and the light receiving optical system 14 are arranged close to each other with the light projecting and receiving directions (the direction of the optical axis) in parallel. Here, it is assumed that the distance between the light projecting optical system 13 and the light receiving optical system 14 can be substantially ignored with respect to the visual field region.

  The distance image generation unit 10 includes a modulation signal generation unit 15 that generates a modulation signal to be given to the light emitting element 11 in order to emit intensity modulated light. In addition, the distance image generation unit 10 includes a timing control unit 16 that generates a light reception timing signal that defines the light reception timing at the image sensor 12 from the modulation signal in order to control the timing at which the image sensor 12 receives light from the target space. . The charge corresponding to the amount of light received by the image sensor 12 is read from the image sensor 12 and input to the arithmetic processing unit 17. The arithmetic processing unit 17 obtains the distance to the object existing in the target space from the relationship between the light reception timing and the light reception amount. As will be described later, the arithmetic processing unit 17 also has a function of converting the distance and direction to the object into coordinate values of an orthogonal coordinate system set in the real space and outputting the same.

  The modulation signal generator 15 generates a modulation signal whose output voltage changes with a sine waveform having a constant frequency (for example, 20 MHz). The light emitting element 11 is driven by this modulation signal, and intensity modulated light whose light output changes in a sine wave shape is emitted from the light emitting element 11.

  The image sensor 12 used in the present embodiment generates an electric charge according to the light reception intensity only during a period synchronized with the light reception timing signal by using an electronic shutter technique. The generated charges are transferred to a light-shielded accumulation area, accumulated in an accumulation period corresponding to a plurality of periods (for example, 10000 periods) of a modulation signal in the accumulation area, and then received and output to the outside of the image sensor 12. As taken out.

  The timing control unit 16 generates a light reception timing signal synchronized with the modulation signal. Here, the timing control unit 16 generates four types of light reception timing signals having a light reception period having a certain time width for each of four phases having different modulation signals. In addition, one type of light reception timing signal selected from the four types of light reception timing signals is supplied to the image sensor 12 for each accumulation period described above.

  That is, by supplying one type of light reception timing signal to the image sensor 12 in one accumulation period, charges in the light reception period corresponding to a specific phase period of the modulation signal are generated in each pixel of the image sensor 12. The accumulated charge is taken out from the image sensor 12 as a light reception output. When the light receiving timing signals different for each accumulation period are given to the image sensor 12 and the operation of taking out the electric charges generated by the image sensor 12 as light reception outputs is repeated, the light reception corresponding to four types of light reception timing signals in four accumulation periods. An output is obtained from the image sensor 12.

Now, four types of light reception timing signals are set to phases different by 90 degrees in one cycle of the modulation signal, and the light reception outputs (charge amounts) output from the image sensor 12 corresponding to the respective light reception timing signals are respectively A0. , A1, A2, A3. At this time, using the relationship of the trigonometric function, the phase difference ψ [rad] between when the intensity-modulated light is projected and received can be expressed in the form of the following equation.
ψ = (A0−A2) / (A1−A3)
Since the frequency of the modulation signal is constant, the phase difference ψ can be converted into a time difference from light projection to light reception, and since the speed of light is known, the distance to the object can be obtained if the time difference is obtained.

  That is, the distance to the object can be obtained from the four types of light reception outputs (charge amounts) A0 to A3. The light receiving period can be set as appropriate so that an appropriate amount of light is obtained in each pixel (for example, a light receiving period corresponding to a quarter cycle of the modulation signal can be used). . The time width of each light receiving period needs to be equal to each other.

  The arithmetic processing unit 17 obtains the phase difference ψ based on the light reception outputs (charge amounts) A0 to A3, and performs the processing described in the following embodiment in addition to the above processing for converting to the distance. The arithmetic processing unit 17 is configured by using a digital signal processing device selected from a microcomputer, a DSP, an FPGA, and the like, and the above-described processing is realized by executing a program in the digital signal processing device. Further, not only the arithmetic processing unit 17 but also the configuration excluding the light emitting element 11 and the imaging element 12 can be realized by using the above-described digital signal processing apparatus.

  In the above-described operation example, four types of light reception timing signals are used. However, the phase difference ψ can be obtained with three types of light reception timing signals, and two types of light reception are performed in an environment where there is no ambient light or ambient light. The phase difference ψ can also be obtained from the timing signal.

  Furthermore, in the above-described operation, charges corresponding to one type of light reception timing signal are accumulated for one pixel, so that four types of light reception outputs (charge amounts) A0 to A3 are extracted four times from the image sensor 12. An accumulation period is required. On the other hand, if charges corresponding to two types of light reception timing signals are accumulated for one pixel, it is possible to read light reception outputs corresponding to the two types of light reception timing signals from the image sensor 12 at a time. Similarly, if the charge corresponding to the four types of light reception timing signals can be accumulated for one pixel, the light reception outputs corresponding to the four types of light reception timing signals can be read out once.

  Since the above-described distance image generation means 10 uses an image sensor in which a plurality of pixels are two-dimensionally arranged as a light receiving element for receiving light from the target space, a distance value is obtained as a pixel value of each pixel. As a result, a distance image is generated. That is, the light receiving surface of the image sensor becomes a virtual projection surface for projecting the visual field region of the distance image generating means 10.

  Since the above-described distance image generation means 10 projects light from the light emitting element 11 into the target space and images the field of view of the imaging element 12 as the target space, the shape of the target space is a distance image generation with the distance image generation means 10 as a vertex. The further away from the means 10, the wider the shape. For example, when the light projecting optical system 13 and the light receiving optical system 14 are formed isotropically around the optical axis, the shape of the target space is a pyramid with the distance image generating means 10 as the apex.

  Therefore, the positions of the pixels arranged on the virtual projection plane described above correspond to the direction in which the target space is viewed from the distance image generation unit 10, and the pixel value of each pixel is the distance to the object existing in the direction. Will be expressed. In other words, the distance image generated by the distance image generating means 10 represents the position of the object in the polar coordinate system. Such a distance image is called a polar coordinate system distance image.

  The above-mentioned distance image in the polar coordinate system is highly convenient when distance information from the distance image generating means 10 is necessary, but the correspondence with each position in the real space that is the target space is difficult to understand, and the real space It is inconvenient to specify an area based on an existing object. Therefore, the arithmetic processing unit 17 performs coordinate conversion for generating an image having each coordinate value of the orthogonal coordinate system from the distance image of the polar coordinate system. Hereinafter, the image after the coordinate conversion is referred to as a coordinate axis-specific image.

  A procedure for generating the coordinate axis-specific image from the polar coordinate system distance image will be briefly described. In order to generate an image for each coordinate axis from a polar coordinate system distance image, first, the polar coordinate system distance image is converted into a three-dimensional image of a camera coordinate system, which is an orthogonal coordinate system, and this three-dimensional image is defined as an orthogonal coordinate defined in the target space. Convert to a global coordinate system 3D image. When a three-dimensional image of the global coordinate system is obtained, it can be decomposed into images for each coordinate axis.

  In order to generate a three-dimensional image in the camera coordinate system from the distance image in the polar coordinate system, the following calculation is performed. Here, the horizontal direction and the vertical direction on the light receiving surface of the image sensor 12 are defined as the u direction and the v direction. The image sensor 12 and the light receiving optical system 14 are arranged so that the optical axis of the light receiving optical system 14 passes through the pixel at the center position of the light receiving surface of the image sensor 12. Further, the light receiving surface of the image sensor 12 is positioned at the focal point of the light receiving optical system 14. In this positional relationship, the coordinates (unit: pixel) of the pixel at the center position of the light receiving surface of the image sensor 12 are (uc, vc), and the pitch between the u direction and the v direction of the pixel of the image sensor 12 (unit: mm). ) Is (su, sv). Further, the focal length of the light receiving optical system 14 is f [mm], and light receiving optics is used for an object that exists in the direction corresponding to the position (u, v) of the coordinates (unit: pixel) of each pixel on the light receiving surface of the image sensor 12. The distance from the center of the system 14 to the object is d [mm]. These values become known values by associating the distance d to the object with the position of each pixel in the distance image generating means 10.

The coordinate values (X1, Y1, Z1) in the camera coordinate system for the object are obtained as follows. The unit of each component of the coordinate values (X1, Y1, Z1) is mm. The origin of the coordinate system set on the light receiving surface of the image sensor 12 is the position of one corner of the light receiving surface of the image sensor 12 which is rectangular, and the origin of the orthogonal coordinate system is the center of the light receiving optical system 14.
X1 = u1 · d / R
Y1 = v1 · d / R
Z1 = f · d / R
However,
u1 = su (u-uc)
v1 = sv (v−vc)
R = (u1 ² + v1 ² + f ² ) ^1/2
Here, in order to simplify the description, the influence of the optical distortion of the light receiving optical system 14 is omitted. However, optical distortion can be corrected by using a distortion correction formula for correcting the distance R from the optical center.

  As described above, after the distance image in the polar coordinate system is converted into a three-dimensional image in the camera coordinate system, the coordinate conversion to the global coordinate system is performed. Here, it is assumed that the distance image generation means 10 is arranged so that the vertical direction (that is, the v direction) on the light receiving surface of the image sensor 12 is parallel to the indoor wall surface and floor surface.

  Now, let the angle of depression of the optical axis of the light receiving optical system 14 in the distance image generating means 10 be θ. Further, the viewing angle of the distance image generating means 10 is assumed to be φ. For conversion from the camera coordinate system to the global coordinate system, rotation corresponding to the depression angle θ is performed around the Y axis with respect to the coordinate values (X1, Y1, Z1) in the camera coordinate system. Thereby, a three-dimensional image of a global coordinate system having a coordinate axis orthogonal to the indoor wall surface and floor surface is generated.

Hereinafter, the coordinate values in the global coordinate system are assumed to be (X, Y, Z). In the illustrated example, the direction away from the wall in the direction orthogonal to the wall is defined as the positive direction in the X direction, and the downward direction orthogonal to the floor is defined as the positive direction in the Z direction. A right-hand system is used as the coordinate system. The rotation of the coordinate system with respect to the depression angle θ is performed by the following calculation.
X = X1 · cos (90 ° −θ) + Z1 · sin (90 ° −θ)
Y = Y1
Z = −X1 · sin (90 ° −θ) + Z1 · cos (90 ° −θ)
The image by coordinate axis is not the image itself mapped to the global coordinate system, but an image in which the X value, the Y value, and the Z value are individually associated with the coordinate position (u, v) of each pixel of the distance image. is there. In other words, each coordinate position (u, v) is an image in which X (u, v), Y (u, v), and Z (u, v) are associated with each other. On the other hand, three coordinate axis-specific images are generated. In order to obtain an image for each coordinate axis from a polar coordinate system distance image, the above calculation may be performed. However, if a table for converting a polar coordinate system distance image into a coordinate value (X, Y, Z) is prepared, Processing load can be reduced.

  Of the images by coordinate axes, an image having an X value as a pixel value is referred to as an X image, an image having a Y value as a pixel value is referred to as a Y image, and an image having a Z value as a pixel value is referred to as a Z image. In the above configuration, three types of tables for performing coordinate conversion are required: an X conversion table for converting to an X image, a Y conversion table for converting to a Y image, and a Z conversion table for converting to a Z image.

  The X image, the Y image, and the Z image described above are stored in a memory of a computer and represent a three-dimensional virtual space corresponding to a target space that is a real space. Therefore, setting the condition in the virtual space is equivalent to setting the condition in the target space.

  The X image, Y image, and Z image generated by the distance image generating means 10 are given to the input information acquiring means 21 as shown in FIG. Here, when a distance image to a human body existing in a predetermined space area in the real space is generated by the distance image generating means 10, the virtual object existing in the virtual three-dimensional space corresponds to the position of the hand as described above. It becomes possible to act.

  Below, the example of the processing tool couple | bonded with a hand is demonstrated. FIG. 4 shows an example in which a virtual object 41 having a transparent texture that transmits the background is arranged in a virtual three-dimensional space. The virtual object 41 is, for example, an automobile, but is initially arranged in a state where it cannot be visually recognized on the screen of the display means 25 as shown in FIG. A broken line in FIG. 4A indicates that there is a transparent virtual object 41 that is not displayed on the display means 25.

  In this state, as shown in FIG. 4B, when the indicator 42 displayed on the screen of the display unit 25 is overlaid on the area where the virtual object 41 exists, an opaque texture is mapped to the virtual object 41, and the virtual object 41 becomes visible on the screen of the display means 25. That is, in the illustrated example, a case where a pasting tool that acts on the virtual object 41 and maps an opaque texture is provided as a processing tool is shown.

  When the pasting tool is coupled to the hand, an opaque texture is gradually formed on the transparent virtual object 41 by performing an operation of rubbing the virtual object 41 with a hand as shown in FIG. To be mapped. In the illustrated example, a hand-shaped index 42 is displayed on the screen of the display means 25 so that the position of the index in the virtual three-dimensional space can be confirmed. Then, by moving the hand in the real space, the texture is mapped to the virtual object 41 existing at the passage part of the index 42.

  By using the pasting tool as described above, the virtual object gradually appears on the screen of the display means 25 with the impression that the virtual object 41 appears from a space where nothing exists in accordance with the movement of the hand in the real space. 41 is displayed. As a result, an impressive presentation can be performed when the virtual object 41 appears in an exhibition or a briefing session.

  In the example shown in FIG. 5, two types of virtual objects 43 and 44 are arranged in a virtual three-dimensional space. The first virtual object 43 is, for example, an automobile, and the second virtual object 44 is disposed so as to cover the first virtual object 43. FIG. 5A shows a state in which the second virtual object 44 completely covers the first virtual object 43. In FIG. 5A, the first virtual object 43 is indicated by a broken line, but in reality, the first virtual object 43 is not visible on the screen of the display means 25.

  If the second virtual object 44 is deleted in this state, the first virtual object 43 can be displayed on the screen of the display means 25. Therefore, in the illustrated example, a case is shown in which an erasing tool that operates only on the second virtual object 44 and erases the second virtual object 44 is provided as a processing tool. When the erasing tool is coupled to the hand, as shown in FIG. 5B, the first virtual object 43 is covered by performing an operation of rubbing the surface of the second virtual object 44 with the hand. The second virtual object 44 is erased gradually as if it were erased with an eraser. In the illustrated example, a hand-shaped index 42 is displayed on the screen of the display means 25 so that the position of the index in the virtual three-dimensional space can be confirmed. Then, by moving the hand in the real space, it is possible to erase the second virtual object 44 existing at the passage portion of the index 42.

  By using the erasing tool as described above, the first virtual object 44 is gradually displayed on the screen of the display means 25 with the impression that the second virtual object 44 is scraped in accordance with the movement of the hand in the real space. A virtual object 43 is displayed. By using the erasing tool, as in the case of using the pasting tool, an impressive presentation can be performed when the first virtual object 43 appears at an exhibition or a briefing session.

  The tool providing means 27 also provides a deformation tool that deforms a virtual object as a processing tool. The deformation tool is a surface of a virtual object in a state in which a virtual object having a prescribed shape such as a shape similar to an actual object or a geometric shape such as a cylinder, cube, or sphere is placed in a virtual three-dimensional space. Has the function of adsorbing the indicator.

  The deformation tool adds deformation that causes the virtual object to be recessed when the index is moved in a direction to push the virtual object into the virtual object surface, and the index is moved in the direction in which the virtual object is pulled by the index. And a function to add deformation to inflate virtual objects. Furthermore, the deformation tool can also pinch and twist the adsorption part with the virtual object, and the virtual object can be deformed as if clay is being formed.

  Here, when the deformation | transformation which makes a virtual object dent using a deformation | transformation tool is added, possibility that a deformation | transformation tool penetrates a virtual object is considered. Therefore, when the deformation tool penetrates the virtual object, the display state of the virtual object penetrated by the deformation tool is made different from other virtual objects. When the virtual object is composed of a plurality of parts, it is desirable to notify the presence or absence of penetration by the deformation tool in units of parts.

  By changing the display state in this way, the user who is deforming the virtual object can be alerted. As the display state, the color of the virtual object, the blinking display of the virtual object, or the like is used.

  Since the deformation tool is provided as described above, a desired deformation can be applied based on the virtual object displayed on the display means 25 without producing a mock-up model or clay model of the product. This makes it easier to create a virtual object. It is also possible to deform a virtual object while discussing with a plurality of people.

  In the above-described operation, the input information acquisition unit 21 extracts the human hand existing in the real space as the specific part. However, as illustrated in FIG. 6, even if the head 34 of the human body 33 is extracted as the specific part. Good. Further, when the input information acquisition unit 21 extracts the head 34 as a specific part, the input information acquisition unit 21 extracts the position and orientation of the head 34. When the position and orientation of the head 34 are extracted, the position and orientation in which the human body in the real space is looking at the screen of the display means 25 can be estimated.

  Therefore, it is desirable for the display processing unit 24 to change the range of the virtual three-dimensional space displayed on the display unit 25 according to the position and orientation of viewing the screen of the display unit 25. That is, the position of the viewpoint is determined by the position of the head 34 and the display range on the display unit 25 is adjusted, and the direction of the line of sight is determined by the position of the head 34 and the coordinate axis of the virtual three-dimensional space displayed on the display unit 25. Adjust the orientation. These adjustments are performed by following the position and orientation of the head 34 extracted by the input information acquisition unit 21.

  FIG. 6A shows a state in which the head 34 of the human body 33 faces the screen of the display means 25, and a virtual object 45 is displayed on the display means 25. Here, as shown in FIG. 6B, when the head 34 is turned downward, the line of sight becomes downward, so that the orientation of the coordinate axes of the virtual three-dimensional space is changed so that the upper surface of the virtual object 45 can be seen. Here, if the orientation of the virtual object 45 changes when the head 34 is raised to a position facing the display means 25, the human body 33 as the user cannot see the state of the virtual object 45 in detail. Therefore, when returning the position of the head 34 to the original position, it is desirable not to change the orientation of the coordinate axes. That is, the direction of the coordinate axis is changed according to the direction in which the head 34 is directed from the position facing the display unit 25, and then the direction of the coordinate axis is not changed while returning to the position facing the display unit 25. Is desirable.

  By the above-described operation, the human body 33 as the user moves the head 34, whereby the orientation of the virtual object 45 displayed on the screen of the display means 25 can be changed. Therefore, the human body 33 who is the user who is viewing the display means 25 simply moves the head 34 so that the line of sight is directed to the part desired to be seen with respect to the virtual object 45 (at this time, the human body 33 moves the position relative to the display means 25). The viewpoint of viewing the virtual object 45 can be changed.

  Although the operation shown in FIG. 6 uses the head of the human body 33, the display of the virtual object on the display means 25 may be changed according to the position of the hand 35 of the human body 33 in the real space. For example, as shown in FIG. 7, the range of the virtual three-dimensional space presented on the display unit 25 may be changed according to the distance between the screen of the display unit 25 and the hand 35. The display processing unit 24 changes the display range.

  In the illustrated example, the virtual three-dimensional space displayed on the display means 25 shows a case where the viewpoint is located above the virtual object 46 like a map or an aerial photograph. In this case, as shown in FIG. 7A, when the distance from the screen of the display unit 25 to the hand 35 is large, it is assumed that the viewpoint is located above and the display range is widened. On the other hand, as shown in FIG. 7B, when the distance from the screen of the display unit 25 to the hand 35 is small, the display range is narrowed assuming that the viewpoint is located below. The range of the virtual three-dimensional space displayed on the display means 25 may be changed by following the movement of the hand 35.

  As described above, the display range of the virtual three-dimensional space on the display unit 25 can be changed depending on the position of the hand 35 relative to the display unit 25. This operation can be combined with the operation shown in FIG. 6, and the display range may be changed using the distance between the display means 25 and the head 34 instead of the hand 35. Alternatively, the display range of the virtual three-dimensional space is moved to the left and right by following the left and right movements of the hand 35, or the panning of the display range of the virtual three-dimensional space by following the direction of the hand 35 (the direction of the palm) Tilt (that is, change of the direction of the coordinate axis around the viewpoint) may be performed.

  Furthermore, as long as the position of the real space changes not only with the head 34 and the hand 35 but also with the movement of at least one position of the limb of the human body 33 and the head, the display means 25 is moved according to the movement of other positions. You may change the direction and range of the coordinate axes to be displayed. The part where the position of the real space changes with at least one movement of the limbs and the head is not only the limbs and the entire head, but also a part of the limbs and the head, together with the limbs and the head It may be a moving article, such as a pointing stick or a writing instrument held in the hand.

  The example shown in FIG. 8 is still another operation example, in which a three-dimensional model 47 having limbs and a head as virtual objects is arranged in a virtual three-dimensional space. For the three-dimensional model 47 shown in the figure, for example, a character can be used. The input information acquisition unit 21 extracts the movements of the limbs and the head of the human body 33 in the real space. The tool providing means 27 provides an interlocking tool that links the movements of the limbs and the head of the human body 33 in the real space with the movements of the limbs and the head in the three-dimensional model 47.

  Therefore, the human body 33 as a user can move the limbs and the head of the three-dimensional model 47 such as the character displayed on the display unit 25 in conjunction with each other by moving the limbs and the head. In this operation, since the user's three-dimensional avatar (incarnation) is set by the three-dimensional model 47 in the virtual three-dimensional space, the user can manipulate the avatar in the virtual three-dimensional space according to the movement of the limbs. it can. Further, as shown in the figure, when the left and right of the human body 33 in the real space and the three-dimensional model 47 in the virtual three-dimensional space are reversed, the three-dimensional model 47 can be moved like a mirror image reflected in a mirror.

  Each operation described above is an example, and can be used in an appropriate combination, and the contents of the virtual three-dimensional space displayed on the display means 25 are not limited to the example of the embodiment described above. . As described above, the technique described in the present embodiment can reflect the movement of an object (particularly, a human body) in a real space on a virtual object displayed on the screen of the display unit 25. Therefore, when designing a product, it is possible to design the product by applying various deformations to the virtual object without forming a model in the real space. In addition, when a virtual object placed in a virtual three-dimensional space is subjected to an action by a processing tool, it is not necessary to operate an input device and it is not necessary to attach an interface device to the human body. Even if it exists, the virtual object can be operated. Therefore, it can be applied to educational games for infants.

DESCRIPTION OF SYMBOLS 10 Distance image generation means 21 Input information acquisition means 22 Projection means 23 Object arrangement means 24 Display processing means 25 Display means 27 Tool provision means 28 Tool connection means 29 Pattern classification means 30 Human body model 31 Segment 32 Joint 33 Human body 34 Head 35 Hand 42 index 43 first virtual object 44 second virtual object 45 virtual object 46 virtual object 47 3D model

Claims (7)

Distance image generation means for generating a distance image of an object existing in real space;
Input information acquisition means for extracting a specific part of the object from the distance image;
Interaction means for causing the movement of the specific part to act on a virtual object in a virtual three-dimensional space and reflecting the result on the display on the display means ;
The interaction means includes
Projection means for associating the specific part with an index arranged in a virtual three-dimensional space constructed by a computer using the pixel value of the specific part extracted by the input information acquisition unit;
Object placement means for placing a virtual object in the virtual three-dimensional space;
Tool providing means for providing a processing tool that the object placement means exerts a predetermined action on the virtual object placed in the virtual three-dimensional space;
Tool coupling means for coupling a processing tool provided by the tool providing means to the index extracted by the input information acquisition means;
In a state where the processing tool is combined with the index associated with the virtual three-dimensional space by the projecting means, the processing tool is applied to the virtual object displayed on the display means according to the movement of the index. Display processing means for reflecting the result of the action by
The object placement means includes
A function of arranging the virtual object in the virtual three-dimensional space;
A function of giving the virtual object a transparent texture that transmits a background;
The virtual object has a function of arranging a first virtual object having a prescribed shape and a second virtual object covering the virtual object in the virtual three-dimensional space;
The tool providing means is a processing tool,
A pasting tool for mapping a predetermined opaque texture to the virtual object present at the passage site;
An erasing tool for erasing the second virtual object present at the passage site;
Providing a deformation tool for adsorbing the index to the surface of the virtual object and deforming the virtual object according to the direction of movement of the index;
The virtual object operating apparatus , wherein the tool combining means selects a processing tool to be combined with the index . Pattern classification means for classifying the movement pattern of the specific part using the pixel value of the specific part extracted by the input information acquisition means, and the tool combining means, the processing tool is associated with the movement pattern, 2. The virtual object manipulation device according to claim 1 , wherein a processing tool corresponding to the motion pattern classified by the pattern classification unit is coupled to the index . When the deformation tool penetrates the virtual object when the deformation tool deforms the virtual object with the deformation tool, the display processing means changes the display state of the penetrated virtual object from the display state of other virtual objects. The virtual object operating device according to claim 1 or 2 , characterized in that The object is a human body, the specific part includes at least an extremity and a head, and the virtual object is a three-dimensional model including an extremity and a head, and the tool providing means includes a human extremity and a head as a processing tool. The virtual object operating device according to any one of claims 1 to 3, further comprising an interlocking tool that interlocks the movement of the unit with the movement of the limbs and the head of the three-dimensional model . The specific part is a part whose position in real space changes with movement of at least one of the extremities and the head of the human body, and the input information acquisition unit is configured to display the specific part from the display unit. And the direction of movement of the specific part is extracted, and the display processing means determines the range of the virtual three-dimensional space displayed on the display means as the position of the specific part extracted by the input information acquisition means. The range of the virtual three-dimensional space displayed on the display unit and the direction of the coordinate axis are changed to follow the direction of movement of the specific part extracted by the input information acquisition unit. The virtual object operating device according to any one of claims 1 to 4 , wherein The object is a human body, the specific part is a head, the input information acquisition means extracts the position and orientation of the head, and the display processing means is the virtual three-dimensional displayed on the display means. The space range is changed by following the viewpoint determined by the position of the head extracted by the input information acquisition means, and the direction of the coordinate axes of the virtual three-dimensional space displayed on the display means is changed to the input 5. The virtual object operating device according to claim 1 , wherein the virtual object operating device is changed so as to follow the direction of the line of sight determined by the orientation of the head extracted by the information acquisition unit . The display means, the virtual object manipulation apparatus according to any one of claims 1 to 6, characterized in that the three-dimensional display of the virtual three-dimensional space.

Images