JP5307060B2 - Image processing apparatus, image processing method, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To allow a user to change, through a simple operation, the way of viewing an object placed inside a virtual place. <P>SOLUTION: In an image processor 300, a storage unit 301 stores the position and attitude of an object placed in a virtual space, the position of a virtual viewpoint and the direction of a virtual line of sight, when the virtual space is viewed from the position of the virtual viewpoint. A detection unit 302 detects the position and attitude of a first body part of a user and the position and attitude of a second body part of the user in a real space. A change unit 303 changes the position and attitude of the object based on the detected position and attitude of the first body part and changes the position of the virtual viewpoint and the direction of the virtual line of sight, based on the detected position and attitude of the second body part relative to the first body part. An image generation unit 304 generates an image, representing an appearance of the object seen from the position of the virtual viewpoint in the direction of the virtual line of sight. A display unit 305 displays the generated image. <P>COPYRIGHT: (C)2011,JPO&amp;INPIT

Description

  The present invention relates to an image processing apparatus, an image processing method, and a program suitable for enabling a user to change the way of viewing objects arranged in a virtual space with a simple operation.

  There is a so-called motion capture technique in which a marker is attached to a model such as a human and the movement of the marker is detected to digitize and capture the movement of the model in real space. In addition, the movement of the model is photographed with one or a plurality of cameras, and the distance to the model (also referred to as “depth”) is measured using the time of flight of the infrared rays irradiated to the surroundings and the phase difference of the reflected waves. There is also a technology to capture the movement of the model by doing.

  These techniques for capturing the state of the real space are used, for example, to move the characters in the screen more naturally in a game in a virtual space. Patent Document 1 discloses an apparatus that allows a player to enjoy a more varied game by photographing a player and tracking a predetermined part of the player (for example, the head and both eyes). .

  On the other hand, systems such as CAD (Computer Aided Design) that can view and design objects such as products and parts arranged in a three-dimensional virtual space from various directions are widely used.

Japanese Patent No. 4117682

  By the way, in the conventional CAD system, when the user wants to change the position and orientation of the object in the virtual space, or the position (virtual viewpoint) and direction (virtual line of sight) of viewing the object, the user can use a predetermined keyboard or mouse. The input device has to be adjusted to a desired angle, and there is a problem that the operation is troublesome or the operation is complicated and difficult to handle.

  The present invention solves such a problem, and an image processing apparatus, an image processing method, and an image processing apparatus suitable for enabling a user to change the way of viewing an object arranged in a virtual space with a simple operation. The purpose is to provide a program.

  In order to achieve the above object, the following invention is disclosed in accordance with the principle of the present invention.

An image processing apparatus according to a first aspect of the present invention includes a storage unit, a detection unit, a change unit, an image generation unit, and a display unit.
The storage unit stores the position and orientation of the object arranged in the virtual space, the position of the virtual viewpoint arranged in the virtual space, and the direction of the virtual line of sight of viewing the virtual space from the position of the virtual viewpoint. Is done.
The detection unit detects the position and posture of the first part of the user in the real space and the position and posture of the second part of the user.
The changing unit changes the position and posture of the stored object based on the detected position and orientation of the first part, and detects the relative position of the detected second part with respect to the first part. And the position of the stored virtual viewpoint and the stored direction of the virtual line of sight are changed based on the orientation and the posture.
The image generation unit generates an image representing a state in which the object is viewed from the position of the virtual viewpoint in the direction of the virtual line of sight.
The display unit displays the generated image on a monitor screen in the real space.

An object whose position and orientation are variable is arranged in the virtual space. An arbitrary solid can be an object. For example, the object is a character appearing in a game, a solid model of an airplane, a product, a part, a three-dimensional topographic map, and the like.
In the real space, a monitor for displaying an image representing the state of the object, a camera for photographing the user, and a sensor for detecting the position and posture of the predetermined part of the user are arranged. The predetermined part is, for example, a user's hand, finger, head, face, foot, or torso.

The image processing device detects the position and posture of the first part of the user in the real space. The first part is, for example, a hand. Further, the image processing device detects the position and posture of the user's second part in the real space. The second part is, for example, the head. The position and posture of the first part and the position and posture of the second part are repeatedly detected at regular timing.
By analyzing the pattern and feature points of the captured image, the positions and shapes of the first part and the second part in the image are determined. Further, the reflected wave from the object of the irradiation wave (for example, infrared rays) emitted from the image processing apparatus is detected, and the flight time and phase difference of the irradiation wave are analyzed, so that the first part and the second part are detected from the image processing apparatus. The distance to the part ("depth" from the sensor) is determined. Based on the result of analyzing the image and the detected depth, the positions and postures of the first part and the second part in the real space are specified.

  In the present invention, the movement of the object in the virtual space is interlocked with the movement of the first part and the second part of the user in the real space. When the user changes the position and posture of the first part, the position and posture of the object change accordingly. When the user changes the position and orientation of the second part, the position of the viewpoint (virtual viewpoint) for viewing the object and the direction of the line of sight (virtual viewpoint) for viewing the object change accordingly.

  Therefore, according to the present invention, when the user wants to view the object arranged in the virtual space from different angles, the user can freely change the position and posture of the object by changing the position and posture of the first part. Alternatively, the position of the virtual viewpoint and the direction of the virtual line of sight may be freely changed by changing the position and posture of the second part. At that time, the user does not need to use an input device such as a controller, a keyboard, or a mouse, but only moves a part of his / her body. The user can change the view of the object arranged in the virtual space with a simple operation.

The changing unit may estimate the position and orientation of the line of sight (hereinafter referred to as “user line of sight”) when viewing the real world from the user's viewpoint based on the detected position and orientation of the second part. .
Then, when the user line of sight intersects the monitor screen within a predetermined time after the position or orientation of the user line of sight starts changing, the image generation unit stores before the change starts while the user line of sight intersects the monitor screen. Based on the position and orientation of the object, the position of the virtual viewpoint, and the direction of the virtual line of sight stored in the section, an image representing the appearance of the object is generated.

According to the present invention, the image processing apparatus can estimate whether or not the user is looking at the monitor screen and can change the display mode according to the estimation result. For example, when the user wants to view an object in the virtual space from a different angle, the user first sets the first part to a desired position and posture. Thereafter, the user turns away from the first part and faces the monitor screen. Then, the state immediately before facing the monitor screen is displayed on the monitor screen.
That is, when it is assumed that the user is not looking at the monitor screen (the user is looking at the first part instead), the movement of the object in the virtual space is the same as the first part of the user in the real space. It is linked to the movement of 2 parts. On the other hand, when it is assumed that the user is looking at the monitor screen (instead, the user is not looking at the first part), the movement of the object in the virtual space and the first part and the first part of the user in the real space are The movement of the second part stops and the state of the object is displayed as if the previous replay image is displayed.
Therefore, the user does not have to turn his eyes to the monitor screen immediately after setting the position and posture of the first part and the position and posture of the second part to the desired position and posture. In addition, the user does not have the inconvenience of setting the position and posture of the first part and the position and posture of the second part to the desired position and posture and looking at the monitor screen with a horizontal eye. That's it.

The storage unit may store an object position and orientation history, a virtual viewpoint position history, and a virtual line-of-sight direction history.
Then, the image generation unit is configured to start the change based on the stored history of the position and orientation of the object, the stored history of the virtual viewpoint position, and the stored history of the virtual line-of-sight. An image representing the appearance of the object may be generated.

  According to the present invention, since a history of various parameters necessary for displaying an image of viewing an object is stored, the image processing apparatus can display an image of viewing the object at an arbitrary past time. .

  The changing unit changes the stored position of the virtual viewpoint and the direction of the virtual line of sight by a predetermined multiple of the detected amount of change in the position and orientation of the first part, and changes the direction of the detected second part. The stored position and orientation of the object may be changed by a predetermined multiple of each relative change in position and orientation.

  That is, the amount of change in the position of the first part, the amount of change in the posture of the first part, the amount of change in the position of the second part, and the amount of change in the position of the second part are the changes in the position of the object. It is not necessary to faithfully reproduce the amount, the amount of change in the posture of the object, the amount of change in the position of the virtual viewpoint, and the amount of change in the direction of the virtual line of sight. According to the present invention, for example, it is possible to realize a specification that if the first part is tilted by 90 degrees, the object is tilted by 180 degrees which is twice that of the first part. The predetermined multiple is not limited to double, and the predetermined multiple can be set to an arbitrary value.

  The changing unit obtains a plane that approximates the shape of the surface of the first part based on the detected position and orientation of the first part, and determines the position and orientation of the object using the obtained plane. May be.

  According to the present invention, since the image processing apparatus does not have to determine the exact shape of the first part, the image processing is further simplified.

  The changing unit obtains a plane that approximates the shape of the surface of the second part based on the detected position and orientation of the second part, and uses the obtained plane to determine the position of the virtual viewpoint and the virtual line of sight The direction may be determined.

  According to the present invention, since the image processing apparatus does not have to determine the exact shape of the second part, the image processing is further simplified.

The first part may be the user's hand, and the second part may be the user's head.
The detection unit may detect the position and posture of the user's hand in the real space and the position and posture of the user's head.
The changing unit changes the position and posture of the stored object based on the detected hand position and posture, and based on the detected relative position and posture of the head to the hand. The stored virtual viewpoint position and the stored virtual visual line direction may be changed.

  That is, the user can see the object viewed from various angles simply by moving the hand and head. At that time, the user does not need to handle an input device such as a controller, a keyboard, and a mouse, and only needs to perform an intuitive operation of moving a hand and a head.

An image processing method according to another aspect of the present invention is an image processing method executed by an image processing apparatus having a storage unit, a detection unit, a change unit, an image generation unit, and a display unit, and includes a detection step and a change step. An image generation step and a display step.
The storage unit stores the position and orientation of the object arranged in the virtual space, the position of the virtual viewpoint arranged in the virtual space, and the direction of the virtual line of sight of viewing the virtual space from the position of the virtual viewpoint. Is done.
In the detection step, the detection unit detects the position and posture of the first part of the user in the real space and the position and posture of the second part of the user.
In the changing step, the changing unit changes the position and posture of the stored object based on the detected position and posture of the first part, and detects the detected second part with respect to the first part. Based on the relative position and orientation, the position of the stored virtual viewpoint and the stored direction of the virtual line of sight are changed.
In the image generation step, the image generation unit generates an image representing a state where the object is viewed from the position of the virtual viewpoint in the direction of the virtual line of sight.
In the display step, the display unit displays the generated image on a monitor screen in the real space.

  According to the present invention, the user can change how to view an object placed in the virtual space with a simple operation.

A program according to another aspect of the present invention causes a computer to function as a storage unit, a detection unit, a change unit, an image generation unit, and a display unit.
The storage unit stores the position and orientation of the object arranged in the virtual space, the position of the virtual viewpoint arranged in the virtual space, and the direction of the virtual line of sight of viewing the virtual space from the position of the virtual viewpoint. Is done.
The detection unit detects the position and posture of the first part of the user in the real space and the position and posture of the second part of the user.
The changing unit changes the position and posture of the stored object based on the detected position and orientation of the first part, and detects the relative position of the detected second part with respect to the first part. And the position of the stored virtual viewpoint and the stored direction of the virtual line of sight are changed based on the orientation and the posture.
The image generation unit generates an image representing a state in which the object is viewed from the position of the virtual viewpoint in the direction of the virtual line of sight.
The display unit displays the generated image on a monitor screen in the real space.

According to the present invention, it is possible to cause a computer to function as an image processing apparatus that operates as described above.
The program of the present invention can be recorded on a computer-readable information storage medium such as a compact disk, flexible disk, hard disk, magneto-optical disk, digital video disk, magnetic tape, and semiconductor memory.
The above program can be distributed and sold via a computer communication network independently of the computer on which the program is executed. The information storage medium can be distributed and sold independently from the computer.

  According to the present invention, it is possible to provide an image processing apparatus, an image processing method, and a program suitable for enabling a user to change the way of viewing an object placed in a virtual space with a simple operation. .

It is a figure which shows schematic structure of the typical information processing apparatus with which the image processing apparatus of this invention is implement | achieved. It is a figure showing typically the appearance of an information processor. It is a figure for demonstrating the functional structure of an image processing apparatus. (A) It is a figure for demonstrating the object arrange | positioned in virtual space, a virtual viewpoint, and a virtual eyes | visual_axis. (B) It is a figure showing a mode that the object was seen from the point on the extension of virtual eyes | visual_axis. (A) It is a figure for demonstrating the 1st site | part and 2nd site | part in real space. (B) It is a figure showing the approximate plane of a 1st site | part. (C) It is a figure showing the approximate plane of a 2nd site | part. (A) It is a figure showing the object in an initial state. (B) It is a figure showing a mode that the position of an object is moved. (C) It is a figure showing a mode that the attitude | position of an object is changed. (A), (b) is a figure which shows the relationship between the variation | change_quantity of the attitude | position of a 1st site | part, and the variation | change_quantity of the attitude | position of an object. (A) It is a figure showing the virtual viewpoint and virtual eyes | visual_axis in an initial state. (B) It is a figure showing a mode that a virtual viewpoint is moved. (C) It is a figure showing a mode that a virtual eyes | visual_axis is moved. (A), (b) is a figure which shows the relationship between the variation | change_quantity of a relative attitude | position with respect to the 1st site | part of a 2nd site | part, and the variation | change_quantity of a virtual gaze. It is a flowchart for demonstrating image processing. It is a figure which shows the structural example of the log | history of the direction of a user gaze. (A), (b) is a figure which represents typically the positional relationship of a user's 1st site | part, 2nd site | part, and monitor in real space. (A), (b), (c) is a figure for demonstrating the relationship between a user's 1st site | part and 2nd site | part in real space, and a mode that the object arrange | positioned in virtual space was seen. It is.

  An embodiment of the present invention will be described. In the following, for ease of understanding, an embodiment in which the present invention is realized using an information processing apparatus for games will be described. However, the following embodiment is for explanation, and the scope of the present invention There is no limit. Therefore, those skilled in the art can employ embodiments in which each or all of these elements are replaced with equivalent ones, and these embodiments are also included in the scope of the present invention.

(Embodiment 1)
FIG. 1 is a schematic diagram showing a schematic configuration of a typical information processing apparatus 100 that functions as an image processing apparatus of the present invention.

  The information processing apparatus 100 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, a hard disk 104, an interface 105, an external memory 106, an input device 107, and the like. And a DVD-ROM (Digital Versatile Disk-Read Only Memory) drive 108, an image processing unit 109, an audio processing unit 110, and a NIC (Network Interface Card) 111.

  A DVD-ROM storing a game program and data is loaded into the DVD-ROM drive 108 and the information processing apparatus 100 is turned on to execute the program, thereby realizing the image processing apparatus of the present embodiment. The

  The CPU 101 controls the overall operation of the information processing apparatus 100 and is connected to each component to exchange control signals and data. The CPU 101 uses an ALU (Arithmetic Logic Unit) (not shown) to perform arithmetic operations such as addition, subtraction, multiplication, and division of data stored in a high-speed accessible storage area called a register, logical sum, logical product, Logical operations such as logical negation, bit operations such as bit sum, bit product, bit inversion, bit shift, and bit rotation can be performed. Further, the CPU 101 includes a coprocessor that can perform saturation calculation such as addition / subtraction / multiplication / division for multimedia processing, vector calculation such as trigonometric function at high speed.

  The ROM 102 records an IPL (Initial Program Loader) that is executed immediately after the power is turned on. When the IPL is executed by the CPU 101, the program recorded on the DVD-ROM is read to the RAM 103, and the startup process by the CPU 101 is started.

  The RAM 103 is for temporarily storing data and programs, and holds, for example, programs and data read from a DVD-ROM, and other data necessary for game progress and chat communication. Further, the CPU 101 provides a variable area in the RAM 103 and performs an operation by directly operating the ALU on the value stored in the variable, or temporarily stores the value stored in the RAM 103 in the register. Perform operations such as performing operations on registers and writing back the operation results to memory.

  The hard disk 104 stores an operating system (OS) program necessary for operation control of the entire information processing apparatus 100, various game data, and the like. The CPU 101 can rewrite information stored in the hard disk 104 at any time.

  The external memory 106 detachably connected via the interface 105 stores data indicating game play status (past results, etc.), data indicating the progress of the game, and communication with other devices using the network. Log (record) data and the like are stored. The CPU 101 can rewrite information stored in the external memory 106 at any time. In addition, the information processing apparatus 100 can connect an additional hard disk via the interface 105.

  As shown in FIG. 2, the input device 107 is installed near a monitor 250 on which a game screen is displayed. The input device 107 includes a camera that captures a user's situation. The CPU 101 analyzes image data representing an image photographed by the camera, and determines a user part (for example, a user's hand, foot, face, etc.) included in the image. Image analysis methods include, for example, analysis by pattern recognition, analysis by extraction of feature points, analysis by calculation of spatial frequency, and the like. Shooting by the camera is continuously performed during the game.

  The input device 107 includes a depth sensor that measures the distance from the input device 107 to the user (or any part of the user). For example, the input device 107 irradiates the surroundings with infrared rays and detects the reflected waves of the infrared rays. Then, the input device 107 determines the irradiation wave from the irradiation wave emission port based on the phase difference between the irradiation wave and the reflected wave and the time (flight time) from when the infrared ray is emitted until the reflected light is detected. The distance (hereinafter also referred to as “depth”) to the object that reflects is obtained. Depth detection by the depth sensor is repeatedly performed at predetermined time intervals in each of the directions in which infrared rays can be emitted.

  By providing the depth sensor, the information processing apparatus 100 can grasp in more detail the three-dimensional position and shape of the object arranged in the real space. Specifically, as a result of image analysis of the first image data acquired at the first time and the second image data acquired at the second time by the CPU 101, the first image data and the second image data Assume that it is determined that a part representing the user's head is included in both of the image data. The CPU 101 determines in which direction the user's head is up, down, left, or right as viewed from the camera based on the change in the position of the head in the first image data and the position of the head in the second image data. In addition to being able to determine whether it has moved to some extent, the head of the user as viewed from the camera from the change in the depth of the head in the first image data and the depth of the head in the second image data It is also possible to determine how far the camera has moved in either the forward or backward direction (how close or away from the camera).

  As described above, the CPU 101 uses the image captured by the camera included in the input device 107 and the distance (depth) measured by the depth sensor included in the input device 107 as in the case of so-called motion capture. The user's three-dimensional movement in the space can be digitized and grasped.

  For example, in a bowling game, when the player makes a motion of throwing a ball in front of the monitor 250 screen (that is, in front of the input device 107), the CPU 101 can recognize that the player has made a motion of throwing the ball. Then, the CPU 101 can proceed with the game according to the recognized motion. That is, the player can input a desired instruction by freely moving his / her body without having a touchpad controller or the like. The input device 107 serves as a so-called “controller” that receives an instruction input from a player.

  Digital image data representing a captured image is a set of a plurality of pixels. Typically, a value representing the intensity of the three primary colors (R, G, B) is associated with each pixel. The fact that the depth in each direction is measured by the image depth sensor means that, in effect, one pixel has a depth (D) in addition to red (R), green (G), and blue (B). It is expressed using one dimension.

  A DVD-ROM mounted on the DVD-ROM drive 108 stores in advance a program for realizing a game, image data and sound data associated with the game, and the like. The DVD-ROM drive 108 reads programs and data recorded on the attached DVD-ROM under the control of the CPU 101. The CPU 101 temporarily stores the read program and data in the RAM 103 or the like.

  The image processing unit 109 processes data read from the DVD-ROM by the CPU 101 or an image arithmetic processor (not shown) included in the image processing unit 109, and then performs a frame memory (not shown) included in the image processing unit 109. Record). Image information recorded in the frame memory is converted into a video signal at a predetermined synchronization timing, and is output to the monitor 250 connected to the image processing unit 109.

  The image calculation processor can execute a two-dimensional image overlay calculation, a transmission calculation such as α blending, and various saturation calculations at high speed. Further, the image arithmetic processor renders polygon information arranged in the virtual three-dimensional space and added with various pieces of texture information by the Z buffer method, so that the polygon arranged in the virtual three-dimensional space from a predetermined viewpoint position is rendered. It is also possible to perform a high-speed execution of a calculation for obtaining a rendering image viewed from a predetermined line of sight.

  Furthermore, by the cooperative operation of the CPU 101 and the image arithmetic processor, it is possible to draw a character string as a two-dimensional image on a frame memory or draw it on the surface of each polygon according to font information that defines the character shape. .

  In addition, the CPU 101 and the image arithmetic processor can write the image data stored in advance in the DVD-ROM into the frame memory, thereby displaying the state of the game on the screen. By repeatedly performing the process of writing and displaying the image data in the frame memory at a regular timing (typically the timing of the vertical synchronization interrupt (VSYNC)), an animation can be displayed on the monitor 250.

  The audio processing unit 110 converts audio data read from the DVD-ROM into an analog audio signal and outputs it from a speaker. Further, under the control of the CPU 101, sound data such as sound effects and music to be generated during the progress of the game is generated, and various sound is output from the speaker by decoding the generated sound data.

  When the audio data recorded on the DVD-ROM is MIDI data, the audio processing unit 110 refers to the sound source data included in the audio processing unit 110 and converts the MIDI data into PCM data. In addition, when the audio processing unit 110 is audio data compressed in an ADPCM (Adaptive Differential Pulse Code Modulation) format, an Ogg Vorbis format, or the like, the audio processing unit 110 converts the compressed audio data into PCM data. The PCM data can be output by performing D / A (Digital / Analog) conversion at a timing corresponding to the sampling frequency and outputting it to a speaker.

  The NIC 111 connects the information processing apparatus 100 to a computer communication network such as the Internet. The NIC 111 is based on, for example, a 10BASE-T / 100BASE-T standard used when configuring a LAN (Local Area Network), an analog modem for connecting to the Internet using a telephone line, ISDN (Integrated Services). A digital network (ADSL) modem, an ADSL (Asymmetric Digital Subscriber Line) modem, a cable modem for connecting to the Internet using a cable television line, and an interface (not shown) that mediates between these and the CPU 101 Composed.

  Next, a functional configuration and the like of the image processing apparatus 300 of the present embodiment realized by the information processing apparatus 100 having the above configuration will be described. The image processing apparatus 300 displays on the monitor 250 an image in which the position and orientation of the object arranged in the virtual space are changed according to the position and orientation of the user in the real space. The user can view the object arranged in the virtual space from various angles as if he / she looked while holding a real plastic model in his / her hand. This will be described in detail below.

  FIG. 3 is a diagram illustrating a functional configuration of the image processing apparatus 300. The image processing apparatus 300 includes a storage unit 301, a detection unit 302, a change unit 303, an image generation unit 304, and a display unit 305.

  The storage unit 301 includes (1) the position and orientation of the object arranged in the three-dimensional virtual space, (2) the position of the virtual viewpoint arranged in the virtual space, and (3) the virtual viewpoint from the position of the virtual viewpoint. The direction of the virtual line of sight to see the space is stored. The RAM 103 functions as the storage unit 301.

  FIG. 4A schematically shows an object 400, a virtual viewpoint 450, and a virtual line of sight 470 arranged in the virtual space. A global coordinate system (XYZ coordinate system) is defined in advance in the virtual space. In FIG. 4, a Cartesian coordinate system is used as the global coordinate system, but the global coordinate system may be defined using other types of coordinate systems such as a polar coordinate system.

  The position of the object 400 in the virtual space is represented by coordinate values in the global coordinate system. As the position of the object 400, the coordinate value of the representative point 401 of the object 400 is used. The representative point 401 is the center of gravity of the object 400, for example.

  FIG. 4B shows an example of a state where the object 400 is viewed from a point on the extension of the virtual line of sight 470. The posture of the object 400 in the virtual space is represented by a vector 402 in the global coordinate system and an inclination angle α from the reference direction 403. In the present embodiment, the reference direction 403 is the Z-axis direction.

  The position of the virtual viewpoint 450 is represented by coordinate values in the global coordinate system. The position of the virtual viewpoint 450 is a position where a virtual camera that captures the virtual space is arranged, and can be changed by an instruction from the user or by a predetermined algorithm.

  The direction of the virtual visual line 470 is represented by a vector in the global coordinate system. The direction of the virtual line of sight 470 is the direction of the virtual camera that captures the virtual space, and can be changed by an instruction from the user or by a predetermined algorithm.

  The CPU 101 updates the position and orientation of the object stored in the RAM 103, the position of the virtual viewpoint 450, and the orientation of the virtual line of sight 470 as needed according to an instruction from the user or according to a predetermined algorithm.

  The detection unit 302 detects the position and posture of the first part of the user in the real space and the position and posture of the second part of the user. The CPU 101 and the input device 107 cooperate to function as the detection unit 302.

  FIG. 5A shows a first part 510 and a second part 520 of the user in the real space. In the present embodiment, the first part 510 of the user is a “hand” (which may be either a left hand or a right hand), and the second part 520 of the user is a “head”.

  The CPU 101 determines the image analysis result of the image data acquired by the input device 107 and the depth of the first part 510 detected by the input device 107 (the distance from the infrared emission port provided in the input device 107 to the user's hand). Based on the above, it is determined in what position and in what posture the first part 510 is arranged in the real space.

  The position of the first part 510 is represented by coordinate values in a coordinate system (xyz coordinate system) defined in the real space.

  The posture of the first part 510 is specified using an approximate plane as described below. FIG. 5B is a diagram for explaining the posture of the first part 510.

  First, the CPU 101 obtains a plane (approximate plane) 512 that approximates the shape of the surface of the user's hand. For example, the CPU 101 extracts a predetermined number of points on the surface of the palm (or the back of the hand) determined by image analysis or the like, and acquires coordinate values in the real space of the extracted points. Then, the CPU 101 calculates an approximate plane 512 by each extracted point using a least square method or the like.

  In the coordinate system (xyz coordinate system) defined in the real space, the approximate plane 512 is expressed by [Equation 1] using parameters a1, b1, c1, and r1.

      a1 · x + b1 · y + c1 · z = r1 [Equation 1]

  Next, the CPU 101 obtains a vector 511 that is a unit vector on the approximate plane 512 and that is a direction in which the finger of the hand determined by image recognition or the like extends. The direction of the vector 511 is the direction of the first part 510.

  Further, the CPU 101 obtains an angle α formed by the vertical direction 513 in the real space and the normal vector 514 of the approximate plane 512. The vertical direction 513 and the direction of one axis in the coordinate system defined in the real space (in this embodiment, the z-axis direction) are preferably matched.

  The normal vector 514 is defined as [Expression 2] using parameters a1, b1, and c1 expressed by [Expression 1].

      Normal vector 514: (a1, b1, c1) (Equation 2)

  The direction of the vector 511 represents “how the hand is pointing”, and the angle α represents “how much the hand is tilted”. Therefore, when the vector 511 and the angle α are obtained, the posture of the first part 510 is specified.

  Similarly, the CPU 101 determines the image analysis result of the image data acquired by the input device 107 and the depth of the second part 520 detected by the input device 107 (from the infrared emission port included in the input device 107 to the user's head). And the position of the second part 520 in the real space and in what posture.

  The position of the second part 520 is represented by a coordinate value in a coordinate system defined in the real space.

  The posture of the second part 520 is specified by using an approximate plane, similarly to the method for obtaining the posture of the first part 510. FIG. 5C is a diagram for explaining the posture of the second part 520.

  First, the CPU 101 obtains an approximate plane 522 representing the shape of the surface of the user's face. For example, the CPU 101 extracts a predetermined number of points on the surface of the face determined by image analysis or the like, and acquires coordinate values in the real space of the extracted points. Then, the CPU 101 calculates an approximate plane 522 by each extracted point using a least square method or the like.

  In the coordinate system (xyz coordinate system) defined in the real space, the approximate plane 522 is expressed by [Equation 3] using parameters a2, b2, c2, and r2.

      a2 * x + b2 * y + c2 * z = r2 [Formula 3]

  Next, the CPU 101 obtains the orientation of the second part 520. The direction of the second part 520 is represented by the direction of the normal vector 521 of the approximate plane 522. The normal vector 521 is defined as [Expression 4] using parameters a2, b2, and c2 expressed by [Expression 4].

      Normal vector 521: (a2, b2, c2) [Equation 4]

  The orientation of the second part 520, that is, the orientation of the user's head (or face) is estimated as the orientation of the user's line of sight (hereinafter referred to as “user's line of sight”) in real space.

  Further, the CPU 101 obtains an angle β formed by the straight line 523 representing the horizontal direction in the real space and the straight line 524 passing through the left eye and the right eye determined by image analysis or the like. The straight line 524 may be a straight line passing through the left ear and the right ear determined by image analysis or the like.

  The direction of the normal vector 521 represents “which face is facing”, and the angle β represents “how much the head is tilted”. Therefore, when the normal vector 521 and the angle β are obtained, the posture of the second part 520 is specified.

  The method for specifying the posture of the first part 510 and the posture of the second part 520 described here is merely an example, and these postures may be specified using other methods. For example, instead of obtaining the approximate planes 512 and 522, an approximate curved surface that approximates the shape of the hand and an approximate curved surface that approximates the shape of the face may be obtained. In this case, the normal vector at the centroid position of the approximate curved surface may be the hand direction or the head direction.

  Next, the changing unit 303 changes the position and posture of the object 400 stored in the storage unit 301 based on the position and posture of the first part 510 detected by the detecting unit 302. The changing unit 303 also determines the position of the virtual viewpoint 450 stored in the storage unit 301 based on the relative position and posture of the second part 520 with respect to the first part 510 detected by the detecting unit 302. The direction of the virtual line of sight 470 is changed. The CPU 101 and the RAM 103 work together to function as the changing unit 303.

  First, a process in which the changing unit 303 changes the position and orientation of the object 400 will be described.

  In the initial state, the CPU 101 places the object 400 in a predetermined posture at a predetermined position in the virtual space. For example, the CPU 101 arranges the object 400 in the horizontal direction at the position of the origin in the global coordinate system defined in the virtual space.

  For example, as illustrated in FIG. 6A, the CPU 101 sets the representative point position 401 of the object 400 in the initial state as the position of the origin in the global coordinate system, and a vector 402 representing the orientation of the object 400 in the initial state. The direction is the positive direction of the X axis in the global coordinate system. In this example, the object 400 is a model of an airplane, but the CPU 101 is arranged so that the wings of the airplane are horizontal in the initial state. Further, the CPU 101 associates the approximate plane 512 in the real space with the XY plane in the global coordinate system defined in the virtual space.

  When the detected position of the first part 510 (in this embodiment, the user's hand) changes, the CPU 101 changes the position of the first part 510 in accordance with the amount of change in the position of the first part 510 as shown in FIG. The representative point 401 of the object 400 in the virtual space is moved. That is, when the user translates the hand in the real space, the object 400 also translates in the virtual space.

  The CPU 101 increases the movement amount ΔPvirtual of the representative point 401 in the virtual space as the change amount ΔPreal of the position of the first part 510 in the real space is larger. Typically, the change amount ΔPreal and the movement amount ΔPvirtual are in a proportional relationship.

  In addition, when the detected posture of the first part 510 changes, the CPU 101 changes the posture of the object 400 in the virtual space according to the amount of change in the posture of the first part 510.

  For example, when the approximate plane 512 of the first part 510 is tilted by the angle α, the CPU 101 tilts the object 400 by the angle α as shown in FIG. That is, when the user tilts his / her hand in the real space, the object 400 tilts by the same amount in the virtual space. When the user wants to see the state when the object 400 in the virtual space is tilted, the user only needs to tilt the hand by the same amount.

  Here, the CPU 101 may set the amount by which the object 400 is tilted when the approximate plane 512 of the first part 510 is tilted by the angle α as an amount obtained by multiplying the angle α by a predetermined amount instead of the angle α. That is, the relationship of [Equation 5] is established between the inclination αreal of the approximate plane 512 of the first part 510 in the real space and the inclination αvirtual of the object 400 in the virtual space.

αvirtual = C · αreal (5)
However, C represents a magnification and is a non-zero real number.

  When C = 1, as shown in FIG. 6C, the object 400 in the virtual space is tilted by the same amount as the hand tilt amount in the real space.

  In the case of 0 <C <1, the inclination amount of the object 400 in the virtual space is smaller than the inclination amount of the hand in the real space as shown in FIG. That is, the sensitivity of posture change becomes duller.

  In the case of C> 1, as shown in FIG. 7B, the inclination amount of the object 400 in the virtual space is larger than the inclination amount of the hand in the real space. That is, the sensitivity of posture change becomes more sensitive.

  In most cases, it is difficult for a human to twist his hand near 360 degrees (or more than 360 degrees). Therefore, when the specification is such that the object 400 can be easily viewed from any direction, “C = 2”. It is desirable to make it about. In this case, if the hand is rotated halfway, the object 400 is rotated once.

  FIG. 6C shows a state where the object 400 is rotated with the Y axis as the rotation axis, that is, when the user tilts his / her hand so that the fingertip is on the upper side. It can be an axis, and any point can be the center point of rotation.

  Next, processing in which the changing unit 303 changes the position of the virtual viewpoint 450 and the direction of the virtual visual line 470 will be described.

  In the initial state, the CPU 101 arranges the virtual viewpoint 450 at a predetermined position in the virtual space, and sets the predetermined direction to the direction of the virtual visual line 470.

  For example, as shown in FIG. 8A, the CPU 101 sets the position (X0, Y0, Z0) as the position of the virtual viewpoint 450 in the initial state, and the position of the object 400 in the initial state from the position of the virtual viewpoint 450 in the initial state. The direction toward the representative point 401 is the direction of the virtual visual line 470 in the initial state.

  When the relative position of the detected second part 520 (in this embodiment, the user's head) with respect to the first part 510 changes, as shown in FIG. The position of the virtual viewpoint 450 in the virtual space is moved according to the amount of change in the relative position 520. That is, when the user moves the relative position with respect to the head hand in the real space, the virtual viewpoint 450 moves in the virtual space.

  The CPU 101 increases the movement amount ΔQvirtual of the virtual viewpoint 450 in the virtual space as the change amount ΔQreal of the relative position of the second portion 520 in the real space with respect to the first portion 510 is larger. Typically, the change amount ΔQreal and the movement amount ΔQvirtual are in a proportional relationship.

  Further, when the detected posture of the second part 520 changes, the CPU 101 changes the virtual line of sight 470 in the virtual space according to the amount of change in the relative posture of the second part 520 with respect to the first part 510. Change posture.

  For example, when the approximate plane 522 of the second part 520 is tilted by the angle β, the CPU 101 tilts the direction of the virtual line of sight 470 by the angle β as shown in FIG. That is, when the user tilts his / her head in the real space, the virtual line of sight 470 tilts by the same amount in the virtual space. When the user wants to view the object 400 in the virtual space from different angles, the user only has to tilt his / her head by the same amount.

  Here, the CPU 101 may set the amount by which the virtual visual line 470 is inclined when the approximate plane 522 of the second portion 520 is inclined by the angle β as an amount obtained by multiplying the angle β by a predetermined amount instead of the angle β. That is, the relationship of [Equation 6] is established between the inclination βreal of the approximate plane 522 of the second part 520 in the real space and the inclination βvirtual of the virtual visual line 470 in the virtual space.

βvirtual = D · βreal (Equation 6)
However, D represents a magnification and is a non-zero real number.

  In the case of D = 1, as shown in FIG. 8C, the virtual line of sight 470 in the virtual space is tilted by the same amount as the tilt amount of the head in the real space.

  When 0 <D <1, as shown in FIG. 9A, the inclination amount of the virtual visual line 470 in the virtual space is smaller than the inclination amount of the head in the real space. That is, the sensitivity of the change in the virtual visual line 470 becomes duller.

  In the case of D> 1, as shown in FIG. 9B, the inclination amount of the virtual line of sight 470 in the virtual space is larger than the inclination amount of the head in the real space. That is, the sensitivity of changes in the virtual line of sight 470 becomes more sensitive.

  Next, the image generation unit 304 generates an image representing a state in which the object 400 is viewed from the position of the virtual viewpoint 450 in the direction of the virtual visual line 470. The CPU 101 and the image processing unit 109 cooperate to function as the image generation unit 304.

  The display unit 305 displays the image generated by the image generation unit 304 on the monitor 250. The CPU 101 and the image processing unit 109 cooperate to function as the display unit 305.

  As described above, the position of the object 400, the posture of the object 400, the position of the virtual viewpoint 450, and the orientation of the virtual line of sight 470 are the positions and postures of the user's first part 510 in the real space, and the second part 520. Since it changes according to the change in position and posture, the user can see the object 400 viewed in various directions from various positions. The user may move his hand and head so that the desired angle is obtained. For example, when the object 400 is a model of an airplane, when the user wants to see the airplane from the bottom (from the side where the wheels come out), change the posture of the hand so that the hand is turned over, or keep the hand stationary and keep the hand from the bottom. You just have to look into it. Also, for example, when the user wants to see the airplane from the front (from the cockpit side), change the posture of the hand so that the hand extends toward the face, or look into the toe while keeping the hand stationary. That's fine.

  Next, image processing executed by the above-described units of the image processing apparatus 300 will be described with reference to the flowchart of FIG.

  First, the CPU 101 sets initial values for the position and orientation of the object 400 in the virtual space, the position of the virtual viewpoint 450, and the orientation of the virtual line of sight 470 (step S1001).

  The CPU 101 controls the input device 107 to acquire the position and posture of the user's first part 510 and the position and posture of the user's second part 520 in the real space (step S1002).

  The CPU 101 determines whether or not the position or orientation of the first part 510 acquired in step S1002 has changed compared to the position or orientation of the first part 510 acquired last time (step S1003).

  For example, the CPU 101 controls the input device 107 to repeatedly detect the position and posture of the first part 510 and the position and posture of the second part 520 at a predetermined periodic timing, and a predetermined number of times. A history of detection results for minutes is stored in the RAM 103. The CPU 101 detects the position and posture of the first part 510 and the position and posture of the second part 520 that are detected at the first timing and stored in the RAM 103, and the second timing after the first timing. The position and posture of the first part 510 and the position and posture of the second part 520 stored in the RAM 103 are compared.

  When it is determined that the position or orientation of the first part 510 has not changed (step S1003; NO), the CPU 101 proceeds to the process of step S1005 described later. On the other hand, when it is determined that the position or orientation of the first part 510 has changed (step S1003; YES), the CPU 101 determines that the object 400 is based on the position and orientation of the first part 510 acquired in step S1002. The position or posture is changed (step S1004). The CPU 101 stores information indicating the changed position of the object 400 and information indicating the changed posture of the object 400 in the RAM 103.

  Next, the CPU 101 determines the relative position of the second part 520 acquired in step S1002 with respect to the first part 510 (hereinafter referred to as “relative position”), or the second part 520 acquired in step S1002. It is determined whether or not the relative posture with respect to the first part 510 (hereinafter referred to as “relative posture”) has changed compared to the relative position or relative posture of the second part 520 acquired previously (step S1005). ).

  When it is determined that the relative position or the relative posture of the second part 520 has not changed (step S1005; NO), the CPU 101 proceeds to the process of step S1007 described later. On the other hand, when it is determined that the relative position or relative posture of the second part 520 has changed (step S1005; YES), the CPU 101 determines the position of the virtual viewpoint 450 based on the relative position and the relative posture acquired this time. The direction of the virtual line of sight 470 is changed (step S1006). The CPU 101 stores information indicating the changed position of the virtual viewpoint 450 and information indicating the changed direction of the virtual visual line 470 in the RAM 103.

  Based on the position and orientation of the object 400, the position of the virtual viewpoint 450, and the direction of the virtual visual line 470, the CPU 101 represents a state in which the object 400 is viewed from the position of the virtual viewpoint 450 to the virtual visual line 470. An image is generated (step S1007).

  Then, the CPU 101 displays the image generated in step S1007 on the monitor 250 (step S1008).

  The CPU 101 repeats the processes of steps S1002 to S1008. As a result, when the user moves his / her hand or head, the state in which the position and posture of the object 400 change accordingly is displayed on the monitor 250.

  According to the present embodiment, the image processing apparatus 300 allows the user to easily and freely move and view the position and posture of the object 400 arranged in the virtual space. For example, the user can move and view the object 400 arranged in the virtual space as if he / she viewed the model in the real space from various angles. At that time, the user can intuitively change the position and orientation of his / her hand as if it were the object 400 without requiring a troublesome operation such as pressing a button on the controller.

(Embodiment 2)
Next, other embodiments of the present invention will be described. In the above embodiment, the position and orientation of the object 400, the position of the virtual viewpoint 450, and the orientation of the virtual line of sight 470 are the position change and posture change of the user's first part 510, and the second part 520. The monitor 250 displays a state of immediately changing according to the change in position and the change in posture. By the way, depending on the positional relationship between the first part 510, the second part 520, and the monitor 250 in the real space, it is difficult for the user to view the monitor 250 while moving the first part 510 and the second part 520. There is a possibility. Therefore, in the present embodiment, the image processing apparatus 300 does not immediately change the position and orientation of the object 400 and the position of the virtual viewpoint 450 and the direction of the virtual line of sight 470, but changes the position of the first part 510. Change the position and orientation of the object 400, and the position of the virtual viewpoint 450 and the orientation of the virtual line of sight with a time difference after the change of the posture and the change of the position of the second part 520 occur. To be able to. This will be described in detail below.

  Based on the position and orientation of the second part 520 detected by the detection unit 302, the changing unit 303 estimates the position and orientation of the line of sight (user line of sight) when viewing the real world from the user's viewpoint.

  As described above, the CPU 101 estimates the direction of the normal vector 521 of the approximate plane 522 representing the shape of the surface of the second part 520 as the direction of the user's line of sight.

  Further, the CPU 101 estimates the coordinates of the midpoint of the line segment connecting the left eye and the right eye determined by image recognition or the like as the position of the user's line of sight in the real space.

  When the user line of sight intersects the image display area of the monitor 250 within a predetermined time after the position or orientation of the user line of sight begins to change, the image generation unit 304 changes the user line of sight while the user line of sight intersects the monitor 250. Based on the position and orientation of the object 400, the position of the virtual viewpoint 450, and the direction of the virtual line of sight 470 stored in the storage unit 301 before starting, an image representing the appearance of the object 400 is generated. .

  The CPU 101 stores, in the RAM 103, a history of determination results for the most recent predetermined number of times for each of the position and orientation of the object 400, the position of the virtual viewpoint 450, and the direction of the virtual line of sight 470.

FIG. 11 is a diagram illustrating a configuration example of the history of the direction of the virtual line of sight 470.
12A and 12B are diagrams schematically showing the positional relationship among the first part 510, the second part 520, and the monitor 250 of the user in the real space. This figure shows a state when looking down from directly above the user (over the head).

  In FIG. 11, the RAM 103 stores a history of discrimination results for the latest n times (n is an integer of 2 or more). The time is the time when the position and posture of the second part 520 are detected by the detection unit 302. The direction of the user's line of sight is represented by an angle γ formed by the line segment 1240 connecting the position 1210 of the user's line of sight and the center 1230 of the display area of the monitor 250 and the direction of the normal vector 521 of the approximate plane 522.

For example, the displacements γ ₁ −γ ₀ , γ ₂ −γ ₁ ,..., Γ _n−2− γ _{n−3 in} the direction until time T (n−2) are all smaller than a predetermined threshold, When the displacement γ _n−1 −γ _{n−2 in} the direction from time T (n−2) to time T (n−1) is greater than a predetermined threshold, the CPU 101 determines that the time T (n−1) It is determined that the user has started to move the second part 520. That is, the time T (n−1) is estimated as the time when the direction of the user's line of sight begins to change.

  When the user's line of sight intersects with the image display area of the monitor 250, as shown in FIG. 12B, a half line 1250 extending from the position 1210 of the user's line of sight toward the normal vector 521 intersects the monitor 250. If you want to.

  The CPU 101 stores the coordinate values of the four corners of the monitor 250 in the real space in the RAM 103 in advance. Then, the CPU 101 includes the user's line of sight in the image display area of the monitor 250 when the intersection of the half line 1250 and the plane including the surface of the monitor 250 falls within the rectangle formed by the four corners of the monitor 250 ( That is, it is determined that the user is looking at an image displayed on the monitor. In other cases, the CPU 101 determines that the user's line of sight is not included in the image display area of the monitor 250 (that is, the user does not see the image displayed on the monitor).

  In step S1007 described above, when the user's line of sight intersects the image display area of the monitor 250, the CPU 101 changes from the position of the virtual viewpoint 450 before the change starts while the user's line of sight intersects the monitor 250. An image representing a state in which the object 400 arranged at the position and posture before the change is started is generated in the direction of the virtual visual line 470 before the start. In step S1008, the CPU 101 displays the generated image on the monitor 250.

For example, the displacements γ ₁ −γ ₀ , γ ₂ −γ ₁ ,..., Γ _n−2− γ _{n−3 in} the direction until time T (n−2) are all smaller than a predetermined threshold, When the displacement γ _n−1 −γ _{n−2 in} the direction from time T (n−2) to time T (n−1) is larger than a predetermined threshold, the CPU 101 determines that the user Information indicating the position and orientation of the object 400, information indicating the position of the virtual viewpoint 450, and virtual line of sight, which are stored in the RAM 103 in association with the time T (n−1) when it is determined that the 520 has started to move. An image is generated based on the information indicating the orientation of 470.

If the position and posture of the object 400 are changed so as to be immediately linked to the change in the position and posture of the hand, the change in the position of the head and the change in posture, If the hand is far away, it may be difficult for the user to see the monitor 250 while moving the hand.
For example, as shown in FIG. 13A, when the user places the first part 510 and the second part 520 in the real space and the user's line-of-sight direction or the arrow YA direction, the object 400 It is assumed that it is arranged at the center of the display area of the monitor 250. Here, when the user turns to look at the monitor 250 from the front and the line-of-sight direction is the arrow YB direction, the position or posture of the second part 520 changes, and thus the virtual viewpoint 450 or the virtual line-of-sight 470 changes. . Then, as shown in FIG. 13B, the monitor 250 shows a state after the virtual viewpoint 450 or the virtual line of sight 470 is changed, and as a result, the user displays the object 400 in the center of the monitor 250. You will not be able to see what you are doing. In order to avoid this problem, it is a heavy burden on the user to keep the monitor 250 flickering from the side, and there is a limit to such an action.
Therefore, when the user's line-of-sight direction enters the image display area from outside the image display area of the monitor 250, the image processing apparatus 300 according to the present embodiment allows the user to monitor as shown in FIG. The image displayed immediately before turning in the direction 250 is displayed. That is, even after the user moves the line-of-sight direction in the direction of the arrow YB, the user can view an image when the line-of-sight direction is the direction of the arrow YA. The user can see an image created by the position and posture of the “slightly forward” hand and head.
When the CPU 101 determines that the user is looking at the monitor 250, as shown in FIG. 13B, “the direction of the vector indicating the direction of the line of sight is determined from the position 1305 of the second part. This is a case where the four corners 1301 to 1304 of the display area of the monitor 250 fall within a quadrangular prism-shaped area surrounded by four connecting line segments.

  Note that the CPU 101 represents a state in which the object 400 arranged at the position and posture before the change starts from the position of the virtual viewpoint 450 before the change starts to the direction of the virtual visual line 470 before the change starts. If the user's line of sight no longer intersects the monitor 250 after displaying the image, it is assumed that the user has looked away from the monitor 250, and the hand position change, posture change, head position change, posture change An image in which the position and orientation of the object 400 is changed in synchronization with the change is generated and displayed.

  In other words, if it is estimated that the user is not facing the monitor 250, the object 400 is displayed “linked” with the hand or head, and if the user is estimated to be facing the monitor 250, Interlocking with the head and the head is temporarily interrupted, and a “replay” image is displayed.

  In this embodiment, the CPU 101 displays an image representing the object 400 immediately before the user turns to the monitor 250, but the image representing the object 400 at any past timing stored in the RAM 103. Can also be displayed. For example, when it is estimated that the user has turned to the monitor 250, the CPU 101 uses the history of the latest past predetermined time (for example, from X seconds before the present to the present, etc.) to use the object 400. May be generated and displayed. In addition, the predetermined time may be set freely by the user.

  The present invention is not limited to the above-described embodiments, and various modifications and applications are possible. Moreover, it is also possible to freely combine the constituent elements of the above-described embodiments.

  A program for operating a computer as all or part of the image processing apparatus 300 is stored and distributed in a computer-readable recording medium such as a memory card, CD-ROM, DVD, or MO (Magneto Optical disk). This may be installed in another computer and operated as the above-described means, or the above-described steps may be executed.

  Furthermore, the program may be stored in a disk device or the like included in a server device on the Internet, and may be downloaded onto a computer by being superimposed on a carrier wave, for example.

  As described above, according to the present invention, an image processing apparatus, an image processing method, and a program suitable for enabling a user to change the way of viewing an object arranged in a virtual space with a simple operation. Can be provided.

100 Information processing apparatus 101 CPU
102 ROM
103 RAM
104 Hard disk 105 Interface 106 External memory 107 Input device 108 DVD-ROM drive 109 Image processing unit 110 Audio processing unit 111 NIC
250 Monitor 300 Image processing device 301 Storage unit 302 Detection unit 303 Change unit 304 Image generation unit 305 Display unit 400 Object 450 Virtual viewpoint 470 Virtual line of sight 510 First part 511 Vector (direction of the first part)
512 approximate plane (approximate plane representing the first part)
513 Vertical direction 514 normal vector (normal vector of the approximate plane representing the first part)
520 Second part 521 normal vector (normal vector of the approximate plane representing the second part)
522 approximate plane (approximate plane representing the second part)
523 Horizontal line 524 Straight line 1210 User line of sight position 1230 Center (center of display area of monitor)
1240 line segment 1250 half straight line

Claims (13)

A storage unit that stores the position and orientation of an object arranged in the virtual space, the position of the virtual viewpoint arranged in the virtual space, and the direction of the virtual line of sight of viewing the virtual space from the position of the virtual viewpoint; ,
A detection unit that detects the position and posture of the first part of the user in the real space and the position and posture of the second part of the user;
Based on the detected position and orientation of the first part, the position and orientation of the stored object are changed, and the relative position of the detected second part with respect to the first part is changed. And a changing unit that changes the position of the stored virtual viewpoint and the stored direction of the virtual line of sight based on the orientation and the posture,
An image generation unit that generates an image representing a state in which the object is viewed in the direction of the virtual line of sight from the position of the virtual viewpoint;
A display unit for displaying the generated image on a monitor screen in the real space;
With
The changing unit estimates the position and orientation of a line of sight (hereinafter referred to as “user line of sight”) when viewing the real world from the viewpoint of the user, based on the detected position and posture of the second part. ,
When the user line of sight intersects the monitor screen within a predetermined time after the position or orientation of the user line of sight begins to change, the image generation unit changes the change while the user line of sight intersects the monitor screen. An image representing the appearance of the object is generated based on the position and orientation of the object, the position of the virtual viewpoint, and the direction of the virtual line of sight stored in the storage unit before starting the operation. To
An image processing apparatus. The image processing apparatus according to claim 1 ,
The storage unit stores a history of the position and orientation of the object, a history of the position of the virtual viewpoint, and a history of the direction of the virtual line of sight,
The image generation unit, based on the stored history of the position and orientation of the object, the stored history of the position of the virtual viewpoint, and the stored history of the direction of the virtual line of sight, Generating an image showing how the object was viewed before starting the change,
An image processing apparatus. The image processing apparatus according to claim 1 , wherein:
The changing unit changes the stored position of the virtual viewpoint and the direction of the virtual line of sight by a predetermined multiple of each detected change amount of the position and orientation of the second part, and detects the detected first position. Changing the position and posture of the stored object by a predetermined multiple of the relative change amounts of the position and posture of one part;
An image processing apparatus. A storage unit that stores the position and orientation of an object arranged in the virtual space, the position of the virtual viewpoint arranged in the virtual space, and the direction of the virtual line of sight of viewing the virtual space from the position of the virtual viewpoint; ,
A detection unit that detects the position and posture of the first part of the user in the real space and the position and posture of the second part of the user;
Based on the detected position and orientation of the first part, the position and orientation of the stored object are changed, and the relative position of the detected second part with respect to the first part is changed. And a changing unit that changes the position of the stored virtual viewpoint and the stored direction of the virtual line of sight based on the orientation and the posture,
An image generation unit that generates an image representing a state in which the object is viewed in the direction of the virtual line of sight from the position of the virtual viewpoint;
A display unit for displaying the generated image on a monitor screen in the real space;
With
The changing unit obtains a plane that approximates the shape of the surface of the first part based on the detected position and posture of the first part, and uses the obtained plane to determine the position of the object. Determine the posture,
An image processing apparatus. A storage unit that stores the position and orientation of an object arranged in the virtual space, the position of the virtual viewpoint arranged in the virtual space, and the direction of the virtual line of sight of viewing the virtual space from the position of the virtual viewpoint; ,
A detection unit that detects the position and posture of the first part of the user in the real space and the position and posture of the second part of the user;
Based on the detected position and orientation of the first part, the position and orientation of the stored object are changed, and the relative position of the detected second part with respect to the first part is changed. And a changing unit that changes the position of the stored virtual viewpoint and the stored direction of the virtual line of sight based on the orientation and the posture,
An image generation unit that generates an image representing a state in which the object is viewed in the direction of the virtual line of sight from the position of the virtual viewpoint;
A display unit for displaying the generated image on a monitor screen in the real space;
With
The changing unit obtains a plane that approximates the shape of the surface of the second part based on the detected position and posture of the second part, and uses the obtained plane to determine the position of the virtual viewpoint. And the direction of the virtual line of sight,
An image processing apparatus. An image processing apparatus according to any one of claims 1 to 5 ,
The first part is the user's hand, the second part is the user's head;
The detection unit detects the position and posture of the user's hand in the real space and the position and posture of the user's head;
The changing unit changes the position and posture of the stored object based on the detected position and posture of the hand, and changes the position and posture of the detected head relative to the hand. Based on the stored virtual viewpoint position and the stored virtual visual line direction,
An image processing apparatus. The image processing apparatus according to any one of claims 1 to 6 ,
When the relative position of the detected second part relative to the first part or the relative posture of the detected second part relative to the first part is changed Changing the position of the stored virtual viewpoint or the stored direction of the virtual line of sight,
An image processing apparatus. An image processing method executed by an image processing apparatus having a storage unit, a detection unit, a change unit, an image generation unit, and a display unit,
The storage unit includes a position and orientation of an object arranged in the virtual space, a position of a virtual viewpoint arranged in the virtual space, and a direction of a virtual line of sight viewing the virtual space from the position of the virtual viewpoint. Remembered,
A detecting step for detecting the position and posture of the first part of the user in the real space and the position and posture of the second part of the user;
The changing unit changes the position and posture of the stored object based on the detected position and posture of the first part, and the first part of the detected second part. The position of the stored virtual viewpoint is changed based on the relative position with respect to the image, and the stored virtual line of sight is determined based on the relative posture of the detected second part with respect to the first part. A change step that changes the orientation of the
An image generation step in which the image generation unit generates an image representing a state in which the object is viewed in the direction of the virtual line of sight changed in the change step from the position of the virtual viewpoint changed in the change step;
A display step in which the display unit displays the generated image on a monitor screen in the real space; and
Equipped with a,
In the changing step, the changing unit is based on the detected position and posture of the second part, and the position of the line of sight when viewing the real world from the viewpoint of the user (hereinafter referred to as “user line of sight”). And direction,
In the image generation step, when the user line of sight intersects the monitor screen within a predetermined time after the position or orientation of the user line of sight begins to change, the user line of sight intersects the monitor screen. The object was viewed based on the position and orientation of the object, the position of the virtual viewpoint, and the direction of the virtual line of sight stored in the storage unit before starting the change. Generate an image of the situation,
An image processing method. Computer
A storage unit that stores the position and orientation of an object arranged in the virtual space, the position of the virtual viewpoint arranged in the virtual space, and the direction of the virtual line of sight of viewing the virtual space from the position of the virtual viewpoint;
A detection unit for detecting the position and posture of the first part of the user in the real space and the position and posture of the second part of the user;
Based on the detected position and orientation of the first part, the position and orientation of the stored object are changed, and the relative position of the detected second part with respect to the first part is changed. And changing the position of the stored virtual viewpoint, and changing the direction of the stored virtual line of sight based on the relative posture of the detected second part with respect to the first part. Change part,
An image generation unit that generates an image representing a state in which the object is viewed in the direction of the virtual line of sight changed by the change unit from the position of the virtual viewpoint changed by the change unit;
A display unit for displaying the generated image on a monitor screen in the real space;
To function as,
The changing unit estimates the position and orientation of a line of sight (hereinafter referred to as “user line of sight”) when viewing the real world from the viewpoint of the user, based on the detected position and posture of the second part. ,
When the user line of sight intersects the monitor screen within a predetermined time after the position or orientation of the user line of sight begins to change, the image generation unit changes the change while the user line of sight intersects the monitor screen. An image representing the appearance of the object is generated based on the position and orientation of the object, the position of the virtual viewpoint, and the direction of the virtual line of sight stored in the storage unit before starting the operation. To
A program characterized by that.   An image processing method executed by an image processing apparatus having a storage unit, a detection unit, a change unit, an image generation unit, and a display unit,
  The storage unit includes a position and orientation of an object arranged in the virtual space, a position of a virtual viewpoint arranged in the virtual space, and a direction of a virtual line of sight viewing the virtual space from the position of the virtual viewpoint. Remembered,
  A detecting step for detecting the position and posture of the first part of the user in the real space and the position and posture of the second part of the user;
  The changing unit changes the position and posture of the stored object based on the detected position and posture of the first part, and the first part of the detected second part. A change step for changing the position of the stored virtual viewpoint and the direction of the stored virtual line of sight based on the relative position and orientation with respect to
  An image generation step for generating an image representing a state in which the image generation unit views the object in the direction of the virtual line of sight from the position of the virtual viewpoint;
  A display step in which the display unit displays the generated image on a monitor screen in the real space; and
  With
  In the changing step, the changing unit obtains a plane that approximates the shape of the surface of the first part based on the detected position and posture of the first part, and uses the obtained plane. Determining the position and orientation of the object;
  An image processing method.   Computer
  A storage unit that stores the position and orientation of an object arranged in the virtual space, the position of the virtual viewpoint arranged in the virtual space, and the direction of the virtual line of sight of viewing the virtual space from the position of the virtual viewpoint;
  A detection unit for detecting the position and posture of the first part of the user in the real space and the position and posture of the second part of the user;
  Based on the detected position and orientation of the first part, the position and orientation of the stored object are changed, and the relative position of the detected second part with respect to the first part is changed. And a changing unit that changes the position of the stored virtual viewpoint and the stored direction of the virtual line of sight based on the orientation and the posture,
  An image generation unit that generates an image representing a state in which the object is viewed in the direction of the virtual line of sight from the position of the virtual viewpoint;
  A display unit for displaying the generated image on a monitor screen in the real space;
  Function as
  The changing unit obtains a plane that approximates the shape of the surface of the first part based on the detected position and posture of the first part, and uses the obtained plane to determine the position of the object. Determine the posture,
  A program characterized by that.   An image processing method executed by an image processing apparatus having a storage unit, a detection unit, a change unit, an image generation unit, and a display unit,
  The storage unit includes a position and orientation of an object arranged in the virtual space, a position of a virtual viewpoint arranged in the virtual space, and a direction of a virtual line of sight viewing the virtual space from the position of the virtual viewpoint. Remembered,
  A detecting step for detecting the position and posture of the first part of the user in the real space and the position and posture of the second part of the user;
  The changing unit changes the position and posture of the stored object based on the detected position and posture of the first part, and the first part of the detected second part. A change step for changing the position of the stored virtual viewpoint and the direction of the stored virtual line of sight based on the relative position and orientation with respect to
  An image generation step for generating an image representing a state in which the image generation unit views the object in the direction of the virtual line of sight from the position of the virtual viewpoint;
  A display step in which the display unit displays the generated image on a monitor screen in the real space; and
  With
  In the changing step, the changing unit obtains a plane that approximates the shape of the surface of the second part based on the detected position and posture of the second part, and uses the obtained plane. Determining the position of the virtual viewpoint and the direction of the virtual line of sight;
  An image processing method.   Computer
  A storage unit that stores the position and orientation of an object arranged in the virtual space, the position of the virtual viewpoint arranged in the virtual space, and the direction of the virtual line of sight of viewing the virtual space from the position of the virtual viewpoint;
  A detection unit for detecting the position and posture of the first part of the user in the real space and the position and posture of the second part of the user;
  Based on the detected position and orientation of the first part, the position and orientation of the stored object are changed, and the relative position of the detected second part with respect to the first part is changed. And a changing unit that changes the position of the stored virtual viewpoint and the stored direction of the virtual line of sight based on the orientation and the posture,
  An image generation unit that generates an image representing a state in which the object is viewed in the direction of the virtual line of sight from the position of the virtual viewpoint;
  A display unit for displaying the generated image on a monitor screen in the real space;
  Function as
  The changing unit obtains a plane that approximates the shape of the surface of the second part based on the detected position and posture of the second part, and uses the obtained plane to determine the position of the virtual viewpoint. And the direction of the virtual line of sight,
  A program characterized by that.

Images