JP5474169B2 - Image processing apparatus and image processing method - Google Patents

Description

  The present invention relates to a technique for synthesizing a real space image and a virtual space image.

  By presenting the computer graphics (CG) superimposed on the live-action scene and presenting it to the experiencer, this experience makes the experience as if a virtual object existed on the spot. Mixed Reality (MR) technology has been proposed.

  In order to give the experience-rich experience to the experience using MR technology, not only simply superimposing and displaying the CG on the live-action scenery, but also the virtual object that the experience is actually drawn by CG Interactions such as touching and operating (making you feel as you do) are important. In order to realize such an interaction, it is necessary to display the hand (subject) of the user who operates the virtual object in front of the virtual object (foreground). This is because if the subject that should be in front of the virtual object is hidden by the virtual object, the sense of distance from the virtual object and the sense of reality will break down, impairing the sense of reality.

  In order to solve such problems, the applicant of the present application has proposed a technique in Patent Document 1 that prevents an image of a subject to be the foreground from being hidden by a virtual object. In such a technique, the background and the subject are acquired as a live-action image, and the “subject to be displayed in front of the virtual object (area having color information as subject detection information) previously registered in the system manually from the live-action image. "Is extracted as a subject area. Then, drawing of a virtual object is prohibited in the subject area. With such a technique, the subject that should be the foreground is displayed so as to be in front of the virtual object without being hidden by the virtual object, and it is possible to perform a mixed reality experience with high presence.

  FIG. 1 is a diagram illustrating an example of a composite image in which a virtual space image is superimposed on a real space image, a virtual space image, and a real space image.

In FIG. 1, reference numeral 101 denotes a real space image. The real space image 101 includes a hand region 150 as a subject. Reference numeral 102 denotes a virtual space image to be superimposed on the real space image 101. Reference numeral 103 denotes a composite image in which the virtual space image 102 is superimposed on the real space image 101. When generating the composite image 103, the virtual space image 102 is not superimposed on the hand region 150 on the real space image 101. As a result, the hand region 150 remains on the composite image 103 as it is. Has been drawn.
JP 2003-296759 A

  The mixed reality experience system disclosed in Patent Document 1 operates well when the subject viewed by the experience person is monochromatic. However, when the subject has a plurality of different colors, only the area having a certain color can be prohibited from drawing CG, so that only a part of the subject is displayed as floating in the CG, and the experience of the user is felt. Could be damaged.

  FIG. 2 is a diagram illustrating an example of a real space image including a subject having a plurality of colors, a virtual space image, and a composite image in which the virtual space image is superimposed on the real space image.

  In FIG. 2, reference numeral 201 denotes a real space image. The real space image 201 includes a hand region 150a and an arm region 150b as subjects. The respective areas 150a and 150b are areas having different colors. Reference numeral 202 denotes a virtual space image to be superimposed on the real space image 201. Reference numeral 203 denotes a composite image obtained by superimposing the virtual space image 202 on the real space image 201. Here, since only the region having the color of the hand region 150a is excluded from the superimposition target of the virtual space image 202, as shown in FIG. The image 202 has been drawn.

  From such a technical background, the following is desired. That is, after the hands of the experience person and the designated area are extracted from the live-action image, areas (such as the hands of the experience person) attached to the extracted area are further extracted. Then, the virtual space image is superimposed on the real space image so that the virtual space image is not superimposed on the subject region (hand and arm) obtained by merging the extracted regions.

  The present invention has been made in view of the above problems, and an object thereof is to provide a technique for appropriately setting a region in which a virtual space image is not superimposed in a real space image.

  In order to achieve the object of the present invention, for example, an image processing apparatus of the present invention comprises the following arrangement.

That is, a means for acquiring a real space image composed of a plurality of frames,
In the frame of interest of the real space image, a first setting means for setting a region composed of a predetermined pixel satisfying a first region,
Within the frame of interest, and acquisition means you get a motion vector of another region other than the first region of the target frame the motion vector of the first region,
First determination means for determining whether or not the magnitude of a motion vector of the acquired other region is greater than or equal to a threshold ;
When the first determination unit determines that the magnitude of the motion vector of the other area is equal to or greater than a threshold, the motion vector of the other area is similar to the motion vector of the first area. A second judging means for judging whether or not
A second region that sets the other region as a second region when the second determination unit determines that the motion vector of the other region is similar to the motion vector of the first region; Setting means,
To the first region and the region other than the second region of the target frame that have been pre-Symbol set, and wherein the obtaining Bei a synthesizing means for synthesizing the virtual space image.

  According to the configuration of the present invention, it is possible to appropriately set a region in which a virtual space image is not superimposed in a real space image.

It is a figure which shows an example of the synthesized image which superimposed the virtual space image on the real space image, the virtual space image, and the real space image. It is a figure which shows an example of the synthetic | combination image which superimposed the virtual space image on the real space image containing the to-be-photographed object which has a some color, a virtual space image, and the said real space image. It is a block diagram which shows the function structural example of the system which concerns on the 1st Embodiment of this invention. It is a figure which shows a mode that the user observes via HMD the composite image on which the image of the virtual space was superimposed on the image of the real space. 12 is a flowchart of a series of processes for the image processing apparatus 300 to generate an image of mixed reality space and send the generated mixed reality space image to the display unit 309 included in the HMD 390. It is a flowchart which shows the detail of the process in step S502. It is a flowchart which shows the detail of the process in step S601. It is a flowchart which shows the detail of the process in step S604. It is a figure which shows the example of the result of having clustered only the feature of the key area on the feature space. It is a figure which shows the characteristic which belongs to the class of a key area | region, and another class. It is a flowchart which shows the detail of the process in step S605. It is a flowchart which shows the detail of the production | generation process of the image of mixed reality space in step S505. It is a figure which shows an example of the image of the mixed reality space produced | generated by the 1st Embodiment of this invention. It is a flowchart of the process in step S502 performed in the 3rd Embodiment of this invention. 2 is a diagram illustrating an example of a hardware configuration of a computer applicable to the image processing apparatus 300. FIG. It is a flowchart of the process performed by the 4th Embodiment of this invention in step S602. It is a figure which shows the principle which calculates the positional change motion vector Tv projected on the image plane.

  Preferred embodiments of the present invention will be described below with reference to the accompanying drawings. The embodiment described below shows an example when the present invention is specifically implemented, and is one of the specific examples of the configurations described in the claims.

[First Embodiment]
In this embodiment, when a virtual space image is superimposed on a real space image, if the “hand” region and the “arm” region are included in the real space image, the respective regions are merged. Thus, a single subject area (composite area) is generated. Then, the superimposition process is controlled so that the subject area is always displayed in front of the image in the virtual space. As will be described in detail later, the subject area is not limited to the merged “hand” area and “arm” area, and any area may be merged to form the subject area. good. That is, in the following description, any subject area may be used as long as the subject area is displayed with a plurality of different pixel values.

  FIG. 4 is a diagram illustrating a state in which the user observes the composite image in which the virtual space image is superimposed on the real space image via the HMD.

  As shown in FIG. 4, a user 401 superimposes a virtual space image generated based on a measurement result by the sensor 404 on a real space image captured by the image capturing unit 301 via the HMD 390 ( The image of the mixed reality space) is observed. If the user's hand 405 or his / her arm 406 enters the visual field range 409 of the imaging unit 301 during such observation, the hand 405 and the arm 406 are displayed in the mixed reality space image displayed on the HMD 390. That is, the image of the virtual space (a part of the virtual object 408) is not superimposed on the region of the hand 405 and the arm 406 on the image of the real space that is the foreground. In order to realize this, the real subject to be the foreground is “the region of the hand 405 and the arm 406 of the user 401”, and the real object to be the background is “the background real object 407 such as a wall or potted plant”. And so on.

  FIG. 3 is a block diagram illustrating a functional configuration example of the system according to the present embodiment.

  As shown in FIG. 3, the system according to the present embodiment includes an HMD 390, a position / orientation measurement unit 306, and an image processing apparatus 300. The HMD 390 and the position / orientation measurement unit 306 are each connected to the image processing apparatus 300.

  First, the HMD 390 will be described.

  The HMD 390 is an example of a head-mounted display device, and includes an imaging unit 301 and a display unit 309.

  The imaging unit 301 is a video camera that captures a moving image of the real space, and the captured image of each frame (real space image) is input to the subsequent image processing apparatus 300 as an image signal. The imaging unit 301 is attached to the HMD 390 so as to be located near the user's eyes when the user wears the HMD 390 on his / her head. Furthermore, the posture (of the imaging unit 301) to be attached is a posture that substantially matches the front direction (line-of-sight direction) of the user wearing the HMD 390 on the head. As a result, the imaging unit 301 can capture a moving image of the real space that is visible according to the position and orientation of the user's head. Therefore, in the following description, the imaging unit 301 may be referred to as a “user viewpoint”.

  The display unit 309 is, for example, a liquid crystal screen, and is attached to the HMD 390 so as to be positioned in front of the eyes of the user wearing the HMD 390 on the head. An image based on the video signal transmitted from the image processing apparatus 300 to the HMD 390 is displayed on the display unit 309. Accordingly, an image based on the video signal transmitted from the image processing apparatus 300 is presented in front of the user wearing the HMD 390 on the head.

  In the present embodiment, the imaging unit 301 and the display unit 309 are built in the HMD 390, and the imaging unit 301 and the display unit 309 are configured such that the optical system of the display unit 309 and the imaging system of the imaging unit 301 match. Built in the HMD 390.

  Next, the position / orientation measurement unit 306 will be described.

  The position / orientation measurement unit 306 is for measuring the position / orientation of the imaging unit 301. For example, a sensor system such as a magnetic sensor or an optical sensor can be applied to the position / orientation measurement unit 306. For example, when a magnetic sensor is applied to the position / orientation measurement unit 306, the position / orientation measurement unit 306 performs the following operation.

  First, when a magnetic sensor is applied to the position / orientation measurement unit 306, the position / orientation measurement unit 306 includes the following units.

A transmitter that generates a magnetic field in the surroundings.A receiver that detects changes in the magnetic field according to its position and orientation in the magnetic field generated by the transmitter. A sensor controller that generates position and orientation information in the sensor coordinate system The transmitter is placed at a predetermined position in the real space. The receiver is attached to the imaging unit 301. When the transmitter generates a magnetic field, the receiver detects a change in the magnetic field according to its own position and orientation (position and orientation of the imaging unit 301), and sends a signal indicating the detected result to the sensor controller. The sensor controller generates position and orientation information indicating the position and orientation of the receiver in the sensor coordinate system based on the signal. Here, the sensor coordinate system is a coordinate system in which the position of the transmitter is the origin and the three axes orthogonal to each other at the origin are the x-axis, y-axis, and z-axis, respectively. Then, the sensor controller sends the obtained position and orientation information to the subsequent image processing apparatus 300.

  However, any sensor system may be applied to the position / orientation measurement unit 306, and any sensor system may be applied to the position / orientation measurement unit 306, and the operation thereof is well known. Is omitted. Further, instead of the sensor system, a method for obtaining the position and orientation of the imaging unit 301 using an image captured by the imaging unit 301 may be used. In this case, the position and orientation measurement unit 306 is omitted, and the method is used. An arithmetic unit to be executed may be provided in the image processing apparatus 300 at the subsequent stage.

  Next, the image processing apparatus 300 will be described. As illustrated in FIG. 3, the image processing apparatus 300 includes a captured image capturing unit 302, a key region extracting unit 303, a motion vector detecting unit 304, a subject region extracting unit 305, an image composition unit 308, an image generation unit 307, and a storage device 310. , Is configured. Hereinafter, each part which comprises the image processing apparatus 300 is demonstrated.

  When the captured image capturing unit 302 receives the image signal of each frame image sent from the image capturing unit 301, the captured image capturing unit 302 sequentially converts the image signal into digital data. Send to 308.

  The key area extracting unit 303 extracts a key area (first area) from the real space image indicated by the digital data received from the captured image capturing unit 302. Here, the key area is an area composed of pixels having a predetermined pixel value. In the present embodiment, an area composed of pixels having pixel values indicating the color of the user's hand is defined as a key area. Then, the key area extraction unit 303 generates key area data that is data for specifying the key area in the real space image, and sends the generated key area data to the subject area extraction unit 305.

  The motion vector detection unit 304 uses the real space image (current frame) received from the captured image capture unit 302 and the real space image of the frame immediately before this real space image to use the real space in the current frame. A motion vector between frames is obtained for each pixel constituting the image. Then, the motion vector data obtained for each pixel is sent to the subject region extraction unit 305.

  The motion vector detection performed by the motion vector detection unit 304 can be executed by calculating an optical flow using an existing block matching method. In the present embodiment, motion vector detection (calculation) is performed by the block matching method. However, the following description is not limited to using such a method, and a motion vector between frames can be detected. Any method may be used as long as it is a method. For example, the motion vector may be detected by using an optical flow based on a gradient method.

  The subject region extraction unit 305 uses the key region data input from the key region extraction unit 303 and the motion vector data input from the motion vector detection unit 304 to use the subject region (subject) in the real space image. Region). As described above, the subject area is an area obtained by merging the user's hand area and the arm area. Then, the subject area data, which is data for specifying the extracted subject area, is sent to the image composition unit 308.

  The image generation unit 307 first constructs a virtual space using the virtual space data held by the storage device 310. The virtual space data includes data on a virtual object placed in the virtual space, light source data placed in the virtual space, and the like. The virtual object data includes, for example, polygon normal vector data, polygon color data, and coordinate position data of each vertex constituting the polygon when the virtual object is composed of polygons. . When texture mapping is performed on a virtual object, texture map data is also included in the virtual object data. The light source data includes, for example, data indicating the type of light source, data indicating the arrangement position and orientation of the light source, and the like.

  Then, after constructing the virtual space, the image generation unit 307 sets a viewpoint in the virtual space based on the position and orientation indicated by the position and orientation information received from the position and orientation measurement unit 306. Then, the image generation unit 307 generates a virtual space image (virtual space image) that can be seen from the viewpoint. Since a technique for generating an image of a virtual space that can be seen from a viewpoint having a predetermined position and orientation is well known, description thereof will be omitted. Then, the generated virtual space image data is sent to the image composition unit 308.

  The image composition unit 308 performs processing for superimposing the virtual space image indicated by the data received from the image generation unit 307 on the real space image indicated by the digital data received from the captured image capturing unit 302. In the superimposition processing, the virtual space image is not superimposed on the subject area indicated by the subject area data received from the subject area extraction unit 305. Then, the image composition unit 308 converts the mixed reality space image generated by such superposition processing into a video signal, and then sends the video signal to the display unit 309 included in the HMD 390. As a result, an image of the mixed reality space corresponding to the position and orientation of its own viewpoint is presented in front of the user wearing the HMD 390 on the head. Furthermore, in the mixed reality space image, the virtual space image is not superimposed on the subject area (the area of the hand and the arm).

  In the present embodiment, the mixed reality space image is sent to the display unit 309 of the HMD 390, but the destination of the mixed reality space image is not particularly limited. For example, a display device configured with a CRT, a liquid crystal screen, or the like may be connected to the image processing device 300, and an image of the mixed reality space may be output to the display device.

  As described above, the storage device 310 holds virtual space data, and the image generation unit 307 appropriately reads and uses the data. Further, the storage device 310 also stores and holds pixel value data (key color data) indicating the color of the user's hand.

  Here, the key color data will be described. The key color data can be described as coordinate values in a multidimensional color space. There are various well-known types of color systems such as RGB, YIQ, YCbCr, YUV, HSV, Lu * v *, La * b * (Japanese Standards Association JIS Color Handbook).

  The key color data may be arbitrarily selected according to the color characteristics of the subject, but is separated into luminance information and hue information in order to offset changes in the color characteristics of the subject due to differences in illumination conditions. It is desirable to use a format color system and only use hue information. Typical examples of such a color system are YIQ and YCbCr. In this embodiment, the YCbCr color system is used. Accordingly, the key color data stored and held in the storage device 310 is obtained by acquiring the color of the user's hand in advance and converting the acquired color as YcbCr color system data.

  Next, FIG. 5 showing a flowchart of the processing for a series of processes for the image processing apparatus 300 to generate an image of the mixed reality space and send the generated mixed reality space image to the display unit 309 of the HMD 390. This will be described below.

  First, in step S <b> 501, the captured image capturing unit 302 acquires the physical space image sent from the imaging unit 301 as digital data.

  Next, in step S502, the key area extraction unit 303 extracts a key area (first area) from the real space image acquired by the captured image capturing unit 302 in step S501. Then, the key area extraction unit 303 generates key area data that is data for specifying the key area in the real space image acquired by the captured image capturing unit 302 in step S501, and the generated key area data is used as the subject. The data is sent to the area extraction unit 305.

  In addition, the motion vector detection unit 304 uses the current space real space image acquired by the captured image capturing unit 302 in step S501 and the real space image one frame before the current frame to obtain the current frame real space image. A motion vector between frames is obtained for each pixel constituting the frame. Then, the motion vector detection unit 304 sends the motion vector data obtained for each pixel to the subject region extraction unit 305.

  The subject area extraction unit 305 uses the key area data generated by the key area extraction unit 303 and the motion vector data generated by the motion vector detection unit 304 to acquire the captured image capture unit 302 in step S501. The subject area in the real space image is extracted. Then, the subject area extraction unit 305 generates mask image data that masks the subject area in the real space image acquired by the captured image capturing unit 302 in step S501 as the subject area data. In the present embodiment, as described above, since the subject is the user's hand and arm, a region where the hand and arm are present is extracted from the real space image, and a mask image is generated from the region. Details of the processing in step S502 will be described later.

  In step S <b> 503, the image generation unit 307 acquires position and orientation information from the position and orientation measurement unit 306. As described above, the position and orientation information is the position and orientation of the viewpoint of the user who wears the HMD 390 on the head, and is the position and orientation of the imaging unit 301.

  In step S504, the image generation unit 307 reads virtual space data from the storage device 310, and constructs a virtual space based on the read data. Then, after constructing the virtual space, the image generation unit 307 sets a viewpoint in the virtual space based on the position and orientation indicated by the position and orientation information acquired from the position and orientation measurement unit 306 in step S503. Then, the image generation unit 307 generates a virtual space image (virtual space image) that can be seen from the viewpoint.

  In step S505, the image composition unit 308 superimposes the virtual space image generated by the image generation unit 307 in step S504 on the real space image indicated by the digital data acquired by the captured image capturing unit 302 in step S501. Process. In the superimposition processing, the virtual space image is not superimposed on the subject area indicated by the subject area data generated by the subject area extraction unit 305 in step S502. Details of the processing in step S505 will be described later.

  In step S506, the image composition unit 308 converts the mixed reality space image generated by the superimposition processing in step S505 into a video signal, and then transmits the video signal to the display unit 309 included in the HMD 390.

  Next, when an instruction to end the process is input from the user via an operation unit (not shown) included in the image processing apparatus 300, or when an end condition for the process is satisfied, the process is performed via step S507. finish. On the other hand, if the user has not input an instruction to end the present process via the operation unit (not shown) included in the image processing apparatus 300 and the end condition for the present process is not satisfied, the process proceeds to step S507. It returns to step S501. Then, in order to send the mixed reality space image of the next frame to the display unit 309, the processing from step S501 is performed.

  Next, details of the processing in step S502 will be described. FIG. 6 is a flowchart showing details of the processing in step S502.

  First, in step S <b> 601, the key area extraction unit 303 reads key color data from the storage device 310. Then, the key area extraction unit 303 collects pixels having pixel values indicated by the key color data read from the storage device 310 among the pixels constituting the real space image acquired by the captured image capturing unit 302 in step S501. Is extracted as a key area.

  Specifically, the key area extracting unit 303 has a pixel value indicated by the key color data read from the storage device 310 among the pixels constituting the real space image acquired by the captured image capturing unit 302 in step S501. “1” is assigned to the pixel. On the other hand, the key area extraction unit 303 does not have the pixel value indicated by the key color data read from the storage device 310 among the pixels constituting the real space image acquired by the captured image capturing unit 302 in step S501. “0” is assigned to the pixel. That is, in the real space image, “1” is assigned to each pixel constituting the hand region, and “0” is assigned to each pixel constituting the other region.

  Here, the process in step S601 will be described in more detail. FIG. 7 is a flowchart showing details of the processing in step S601. Note that the flowchart in FIG. 7 is a flowchart of processing performed on a pixel at image coordinates (i, j) in the real space image. Therefore, in step S601, the process according to the flowchart of FIG. 7 is actually performed for each pixel constituting the real space image.

  First, in step S701, the key area extracting unit 303 represents the pixel value of the pixel at the image coordinates (i, j) in the real space image acquired by the captured image capturing unit 302 in step S501 (in this embodiment, represented by an RGB value). Are converted into YCrCb values. The R value of the pixel at the image coordinates (i, j) is R (i, j), the G value is G (i, j), and the B value is B (i, j). In this case, in step S701, R (i, j), G (i, j), B (i, j) are converted using the function color_conversion () for converting RGB values into YCrCb values, and Y Value, Cr value, Cb value are obtained.

  Next, in step S702, it is determined whether or not the colors represented by the respective values Y, Cr, and Cb obtained in step S701 are substantially the same as the colors indicated by the key color data read from the storage device 310. For example, the function Key_area_func () is used to determine whether or not the colors represented by the values Y, Cr, and Cb obtained in step S701 are substantially the same as the colors indicated by the key color data read from the storage device 310. Judgment. The function Key_area_func () is a function that returns 1 if substantially the same, and returns 0 if not substantially the same.

  Here, as a determination method using the function Key_area_func (), for example, it is determined whether or not the coordinate values (Cr, Cb) on the CbCr plane defined by Cb, Cr belong to the color distribution area of the key color data. To do. The determination result may be expressed as a binary value, for example, 1 if it belongs to the color distribution of the key color data, or 0 if it does not belong, but the degree of belonging is expressed as a continuous value from 0 to 1. Anyway.

  The value returned by the function Key_area_func () is assigned to the array Key_area (i, j). This array key_area (i, j) is for storing a value indicating whether or not the pixel at the image coordinate (i, j) is a pixel constituting the key area.

  Then, by performing the processing according to the flowchart of FIG. 7 for all i and j, the array Key_area holds “1” or “0” for each pixel constituting the real space image. The array Key_area is the key area data.

  In the present embodiment, the pixel value of each pixel constituting the real space image acquired by the captured image capturing unit 302 is represented by RGB, but may be represented by YIQ or YUV. In that case, the processing in step S701 may be omitted, and coordinate values in IQ space and UV space may be used instead of (Cb, Cr) in step S702.

  As described above, the key area extraction unit 303 indicates whether or not each pixel constituting the real space image acquired by the captured image capturing unit 302 constitutes a key area (hand). Generate data.

  Returning to FIG. 6, in step S602, the motion vector detecting unit 304 obtains a motion vector between frames for each pixel constituting the real space image of the current frame acquired by the captured image capturing unit 302 in step S501. . In the present embodiment, the motion vector is obtained for each pixel constituting the real space image. However, the present invention is not limited to this, and the motion vector may be obtained at a plurality of locations on the real space image. For example, a motion vector may be obtained for pixels only in the vicinity of the key area. Thereby, the time cost required for obtaining the motion vector can be reduced.

  In step S603, the subject area extraction unit 305 configures an area (non-key area, other area) other than the key area (other than the first area) among the motion vectors for each pixel obtained in step S602. The average of the magnitudes of motion vectors obtained for each pixel is obtained. Then, it is determined whether or not the obtained average (representative motion vector magnitude) is greater than or equal to a predetermined threshold. As a result of such determination, if the obtained average is equal to or greater than the threshold, the process proceeds to step S604, and if it is less than the threshold, the process proceeds to step S606.

  The “motion vector magnitude” described here indicates a motion vector distance component. Of course, the “size” may be obtained from the angle component of the motion vector. Thus, the method for obtaining the magnitude of the motion vector is not particularly limited. Here, the significance of the determination process performed in step S603 will be described.

  A region having a high similarity to the motion vector of the hand region is extracted as an arm region. This is achieved because the hand and arm are moving together in most cases, and the motion vectors are similar. However, there are cases where only the hand moves and the arm hardly moves. For example, this is the case when only the wrist is turned. In this case, the arm is not extracted even if the subject area is extracted as it is. Further, when neither the hand nor the arm moves at all, and further, the imaging unit 301 does not move, the motion vector is not calculated, so that the subject area cannot be normally extracted.

  Therefore, in the present embodiment, in step S603, processing is performed to determine whether the hand is not moving and whether the hand, the arm, and all of the imaging unit 301 are not moving. That is, when the magnitude of the motion vector of the non-key area is almost 0, it is determined that the arm area is not moving or that all of the hand, the arm, and the imaging unit 301 are not moving. In that case, in step S606 The problem is avoided by processing. The process in step S606 will be described later.

  In step S604, the subject area extraction unit 305 identifies a second area to be merged with the key area based on the key area extracted in step S601 and the motion vector obtained in step S602. Then, an area obtained by merging the specified second area and the key area is obtained as a subject area. In the present embodiment, the arm region is specified as the second region. Then, the subject area is obtained by merging the specified arm area with the hand area as the key area.

  Here, details of the processing in step S604 will be described. FIG. 8 is a flowchart showing details of the processing in step S604.

  First, in step S801, the subject region extraction unit 305 normalizes each motion vector with each feature axis using a distance component and an angle component as features. This avoids weighting of values due to differences in units of features (general normalization). For example, normalization is performed by minimizing the distance between feature patterns.

  Next, in step S802, the subject region extraction unit 305 clusters only the features of the key region on the feature space among the motion vectors normalized in step S801. That is, as shown in FIG. 9, the features of the key area are clustered on the feature space defined by the vector distance component axis (vertical axis) and the angle component axis (horizontal axis) (feature pattern learning). FIG. 9 is a diagram illustrating an example of a result of clustering only the features of the key area on the feature space.

  Here, noise components may be excluded from the features of the clustered key regions as necessary. Specifically, a class having a small number of features or a class having a small distance component is excluded as noise.

  Further, although the feature of the key region is clustered, noise may be excluded by making the classing feature only the feature of the edge region of the key region. The edge region extraction can be realized by an existing labeling algorithm.

  Next, in step S803, the subject region extraction unit 305 selects the features determined to be included in the class of the key region clustered in step S802 out of all normalized motion vector features in this class. Process to include. That is, as shown in FIG. 10, in the feature vector motion space, a feature belonging to the key region class is regarded as a subject region class, so that the subject region class and the others are identified. Thus, the subject area includes an area having a motion vector component similar to the key area in addition to the area having the key color data. FIG. 10 is a diagram showing a class of the key area and features belonging to other classes.

  Such processing is performed based on, for example, the following equation.

target_area (i, j) = discriminant_func (whole_vec (i, j))
Here, (i, j) is a pixel coordinate in the real space image, and target_area is a subject area. Further, discriminant_func () is a subject area identification function learned by the motion vector characteristics of the key area class, and whole_vec is the entire motion vector of the real space image. Note that, as described above, the motion vector is not limited to being obtained for all the pixels constituting the real space image, and therefore, in step S803, not all the pixels are obtained, but all the motion vectors obtained are obtained. This is done for pixels.

  Next, in step S804, the subject region extraction unit 305 performs a labeling process for each pixel identified in step S803 as a pixel having a motion vector component similar to the key region. Hereinafter, an area composed of labeled pixels is referred to as an additional area. For example, only the pixels constituting the arm region are labeled. In addition, the pixels labeled here may include pixels constituting an area other than the user's hand and arm. Originally, these pixels are pixels that constitute a background region. For example, it may be a human hand or arm other than the user.

  In step S805, the subject area extraction unit 305 determines whether the additional area labeled in step S804 is appropriate as the subject area. There are two criteria used in the determination in step S805.

  The first criterion is whether or not the additional area is linked to the key area (linkage relationship). That is, since the arm to be the subject is connected to the hand as a matter of course, it is determined whether the additional area is connected to the key area, and if it is connected, the additional area is assumed to be the subject area. Those that are not are excluded from the subject area.

  The second criterion is whether or not the additional region belongs to the end region of the real space image. In other words, since the arm that should be the subject area is connected to the user as a matter of course, when the user looks at his / her hand, the arm connected to the hand must be connected to the end of the user's field of view. Therefore, there should be an arm region in the end region of the image viewed by the user. For these reasons, it is determined whether or not the additional region belongs to the end region of the real space image, and if it belongs, it is assumed that this additional region is the subject region. Those that are not are excluded from the subject area.

  By these processes, erroneous recognition of the background area as the subject area is reduced.

  Here, in order to make the second criterion more strict, it may be determined whether the additional region belongs to the left or right end region of the real space image.

  In step S806, the subject area is formed by merging the additional area recognized as the subject area in step S805 with the key area.

  That is, the subject area is as follows.

Subject area = Key area + Additional area However, the additional area is an area determined to be suitable as the subject area in step S805.

  Then, the process returns to step S605 in FIG.

  Here, in the above description, the distance component and the angle component of the motion vector are used as features, but only the distance component may be used as a feature. That is, what is used as the feature of the motion vector is not particularly limited, and any feature may be used as long as the similarity between the motion vectors can be obtained.

  Returning to FIG. 6, in step S <b> 605, the subject region extraction unit 305 stores mask image data obtained by masking the subject region extracted in step S <b> 604 from the real space image acquired by the captured image capturing unit 302 in step S <b> 501. It is generated as the subject area data.

  Here, details of the processing in step S605 will be described. FIG. 11 is a flowchart showing details of the processing in step S605. In addition, the flowchart of FIG. 11 is a flowchart of the process performed about the pixel in the image coordinate (i, j) in a real space image. Therefore, in step S605, the process according to the flowchart of FIG. 11 is actually performed for each pixel constituting the real space image.

  In step S1101, the subject area extraction unit 305 writes “1” in the array Key_area (i, j) if the pixel at the image coordinates (i, j) is recognized as the subject area in step S805. Such an operation is performed by executing the function mask_func (). Thereby, the array Key_area (i, j) becomes a two-dimensional array for storing values indicating whether or not the pixel at the image coordinate (i, j) is a pixel constituting the subject area.

  Then, by performing processing according to the flowchart of FIG. 11 for all i and j, the array Key_area holds “1” or “0” for each pixel constituting the real space image. The array Key_area is the subject area data.

  In the present embodiment, the mask image indicated by the generated subject area data may include noise in the mask area. In this case, the existing convex closing process is performed.

  On the other hand, returning to FIG. 6, in step S606, the subject area extraction unit 305 sets the subject area data generated one frame before as being used in the current frame. As described above, this is a process for substituting the previously generated mask image when the subject region cannot be normally extracted from the motion vector. However, in step S606, the key area portion may be constantly updated, and subject area data indicating a mask image in which only the additional area is not updated may be output. To always update the key area portion means to extract the area based on the key color data. Therefore, if such processing is performed, it is ensured that the shape of the key area (hand shape) is always accurately extracted by the key color data.

  Then, after any processing in steps S605 and S606, the process returns to step S503 in FIG.

  Next, details of the mixed reality space image generation processing in step S505 of FIG. 5 will be described. FIG. 12 is a flowchart showing details of the mixed reality space image generation processing in step S505. Note that the flowchart of FIG. 12 is a flowchart of processing performed on a pixel at image coordinates (i, j) in an image of the mixed reality space. Therefore, in step S505, the process according to the flowchart of FIG. 12 is actually performed for each pixel constituting the mixed reality space image.

  First, in step S1201, the image composition unit 308 performs the following processing. That is, the pixel real (i, j) of the image coordinate (i, j) in the real space image indicated by the digital data acquired by the captured image capturing unit 302 in step S501 is converted into the frame memory buffer (i in the image processing apparatus 300. , j).

  In step S1202, data Key_area (i, j) corresponding to the image coordinates (i, j) in the mask image indicated by the subject area data generated in step S502 is converted into a stencil buffer stencil (i , j).

  Next, in step S1203, when stencil (i, j) = 0, the image composition unit 308 determines the pixel CGI (i, j) of the image coordinates (i, j) in the virtual space image generated in step S504. ) Is overwritten in the frame memory buffer (i, j). On the other hand, when stencil (i, j) = 1, the image composition unit 308 does not process the frame memory buffer (i, j). That is, the subject area is not subject to superimposition of the virtual space image.

  Then, by performing processing according to the flowchart of FIG. 12 for all i and j, an image of the mixed reality space is generated in the frame memory buffer. In step S506, the mixed reality space image is sent to the display unit 309 included in the HMD 390 as a video signal.

  As described above, according to the present embodiment, when a virtual space image is superimposed on a real space image, if “hand” and “arm” are included as subjects in the real space image, the subject is The superimposition process can be controlled so that it is always displayed in front of the virtual space image.

  FIG. 13 is a diagram illustrating an example of an image of the mixed reality space generated by the present embodiment. The mixed reality space image 1301 shown in FIG. 13 is generated when the real space image and the virtual space image are the real space image 201 and the virtual space image 202 shown in FIG. 2, respectively. As shown in FIG. 13, not only the hand region 150 a but also the arm region 150 b having a pixel value different from that of the hand region 150 a can be displayed in front of the virtual space image 202.

[Second Embodiment]
As described at the beginning of the first embodiment, the area of the subject is not limited to the merged area of the “hand” area and the “arm” area. These regions may be formed. That is, in the following description, any subject area may be used as long as the subject area is displayed with a plurality of different pixel values.

  For example, the subject area may be determined by determining a real object held by the user's hand as an additional area. Thereby, in addition to the user's hand and arm, a real object grasped by the hand can be displayed in front of the virtual space image.

  In this case, what is necessary is just to change the process in step S805 as follows in 1st Embodiment.

  In the first embodiment, two determination criteria are provided in step S805. In the present embodiment, one of the judgment criteria is changed to the following judgment criteria.

  Specifically, it is not determined whether the additional area belongs to the edge area of the real space image. This is because the real object grasped by the hand does not necessarily hang on the edge of the real space image.

  In this embodiment, as an alternative determination process, a determination is made based on whether the subject area obtained by merging the key area and the additional area (the area determined in step S805) is on the edge of the real space image. I do.

  This avoids erroneous recognition of areas that are not gripped by the hand.

[Third Embodiment]
In the first and second embodiments, the key area and the additional area are calculated for each frame, and the subject area is determined based on them. That is, the process for obtaining the subject area based on the key area and the additional area is performed for each frame. In the present embodiment, the key area and the additional area are calculated only for designating the initial area, and the update of the subject area thereafter is performed by automatic contour extraction processing.

  In the present embodiment, the initially registered subject area can be stably updated every time. Here, “stable” means that, for example, it is possible to extract a region that does not change even if a new real object included in the key region appears. Here, this embodiment is different from the first and second embodiments only in the processing in step S502.

  FIG. 14 is a flowchart of the processing in step S502 performed in the present embodiment. In the flowchart shown in FIG. 14, the process in step S603 and the process in step S606 are deleted from the flowchart shown in FIG. Instead, between the steps S604 and S605, the dynamic contour object registration processing at step S1401 and the dynamic contour extraction processing at step S1402 are added.

  In step S1401, the subject region extraction unit 305 registers the subject region extracted in step S604 as a target for dynamic contour extraction.

  In step S1402, the subject region extraction unit 305 performs dynamic contour extraction of the subject region registered in step S1401. For the active contour extraction, an existing algorithm such as a snake may be used. Since the active contour extraction is an existing technique, description thereof is omitted.

  In step S605, the subject area extraction unit 305 generates and outputs a mask image (subject area data) based on the subject area extracted in step S1401.

[Fourth Embodiment]
In the above embodiment, the subject region is specified only from the motion vector calculated from the real space image captured by the imaging unit 301. However, identification of the subject area is not limited to this method. For example, the subject region may be specified from the motion vector obtained by correcting the motion vector calculated from the real space image using the motion vector generated from the change in the position and orientation of the imaging unit 301.

  When the imaging unit 301 moves or rotates, an error is likely to occur when extracting a subject area with a motion vector calculated only from a real space image. This is because the motion vector calculated only from the real space image includes not only the motion vector of the subject but also the motion vector of the imaging unit 301. For example, when the imaging unit 301 moves in the direction opposite to the movement of the subject, some of the motion vectors of the subject may be canceled.

  Therefore, in the present embodiment, the motion vector of the subject is calculated by subtracting the influence of the motion vector resulting from the change in the position and orientation of the imaging unit 301 from the motion vector calculated from the real space image. Then, the subject area is specified from the motion vector as the calculation result. In this case, what is necessary is just to change the process in step S502 as follows in 1st Embodiment.

  In the first embodiment, the motion vector is calculated only from the real space image in step S502 and further in step S602. In this embodiment, the motion vector calculation method is changed as follows.

  FIG. 16 is a flowchart of the processing performed in this embodiment in step S602. In this embodiment, in step S602, the motion vector is corrected only from the real space image based on the change in the position and orientation of the imaging unit 301 (imaging device).

  In step S1501, the motion vector detection unit 304 calculates a motion vector from the real space image. The process in step S1501 is the same as the process in step S602 described in the first embodiment.

  Next, in step S1502, the motion vector detection unit 304 corrects the motion vector calculated in step S1501 using information on a motion vector (posture change motion vector) due to a posture change of the imaging unit 301.

  More specifically, first, the motion vector detection unit 304 obtains posture information of the imaging unit 301 from the position / orientation measurement unit 306. Here, it is assumed that the motion vector detection unit 304 holds the position and orientation information of the imaging unit 301 in the previous frame in advance. The motion vector detection unit 304 calculates the amount of change in posture from the posture information of the imaging unit 301 in the previous frame and the posture information of the imaging unit 301 in the current frame. From this posture change amount, a motion vector (posture change motion vector) generated by the posture change is obtained. Since the motion vector calculation technique is a well-known technique, a detailed description thereof will be omitted. Here, the posture change means that the optical axis rotates about the lens center of the imaging unit 301.

  Next, the calculated posture change motion vector is projected onto the image plane of the imaging unit 301 to be converted as a motion vector on the image.

  Then, the motion vector detection unit 304 corrects the motion vector calculated in step S1501 using the posture change motion vector projected onto the image plane. Such correction is performed based on the following equation.

M '= M−Rv · I (Formula 1)
Here, M ′ is a matrix representing a motion vector obtained by subtracting a motion vector for posture change from a motion vector in the real space image, and M is a matrix representing a motion vector calculated from the real space image. Rv is a posture change vector projected onto the image plane, and I is a unit matrix (a matrix having the same size as the matrix M).

  Thus, the motion vector due to the posture change is subtracted from the motion vector calculated from the real space image.

  Returning to FIG. 16, next, in step S1503, the motion vector detection unit 304 uses the information on the motion vector (position change motion vector) due to the position change of the imaging unit 301, and the motion vector calculated in step S1501. Correct.

  More specifically, first, the motion vector detection unit 304 obtains position information of the imaging unit 301 from the position / orientation measurement unit 306. The motion vector detection unit 304 calculates the amount of change in position from the position information of the imaging unit 301 in the previous frame and the position information of the imaging unit 301 in the current frame. A motion vector generated by the position change is obtained from the position change amount. The change in position refers to a change in position when the lens unit of the imaging unit 301 is translated about the axis.

  Next, the position change motion vector is projected onto the image plane of the imaging unit 301 to be converted as a motion vector on the image. Here, when projecting the position change motion vector onto the image plane, it is necessary to consider the depth information to the subject, unlike the image plane projection of the position change motion vector. This is because the position change motion vector projected onto the image plane differs depending on the depth distance to the subject. Specifically, as the distance to the subject increases, the magnitude of the position change motion vector increases.

  Therefore, the motion vector detection unit 304 measures the depth distance to the subject in order to calculate the position change motion vector projected onto the image plane.

  In this embodiment, since the subject is imaged by the HMD 390 composed of a stereo video camera, the depth distance is measured by the stereo matching method. Since the stereo matching method is a well-known technique, a description thereof will be omitted.

  In the present embodiment, the depth distance is measured by the stereo matching method, but is not limited to this method. For example, the depth distance may be measured using an infrared distance measuring camera. In other words, any method may be used as long as it can measure the distance. Further, the depth distance may be set by the user in order to calculate the position change motion vector.

  When the depth distance is measured, the motion vector detection unit 304 calculates a position change motion vector projected on the image plane based on the following equation.

Tv = f · t / z (Formula 2)
Here, Tv is a position change motion vector projected onto the image plane, and f is the distance from the lens of the imaging unit 301 to the imaging plane. In addition, t is a motion vector generated by a change in the position of the imaging unit 301, and z is a depth distance to the subject.

  FIG. 17 is a diagram illustrating the principle of calculating the position change motion vector Tv projected onto the image plane. FIG. 17 shows an example in which the imaging unit 301 is translated (translated by t) when the current frame changes from the previous frame to the current frame (the subject is fixed). FIG. 17 shows a case where the imaging unit 301 translates in the X-axis direction among the XY coordinate axes that define the image plane. Here, in order to simplify the explanation, it is assumed that the movement is only in the X-axis direction, but it goes without saying that the principle of the method described here is also applicable when the movement direction has a Y-axis component. .

In FIG. 17, O ₁ indicates the lens center of the imaging unit 301 in the previous frame (previous frame) from the present. O ₂ indicates the lens center of the imaging unit 301 in the current frame (current frame). P (x, z) indicates one point (measurement point) of the subject imaged by the imaging unit 301. x represents the value of the x coordinate, and z represents the value of the z coordinate. The coordinate system expressed here is a coordinate system in the real space with the origin of the lens center of the imaging unit 301 in the previous frame. That is, z is a depth value from the imaging unit 301.

X ₁ is an x coordinate when the measurement point is projected on the image plane in the previous frame. X2 is the x coordinate when the measurement point is projected on the image plane in the current frame. That is, X2-X1 can be said to be a motion vector of the imaging unit 301 on the image plane. Others are the same as (Formula 2).

  As can be seen from FIG. 17, when a position change motion vector t is given, the position change on the image plane is calculated from the similarity between the distance f from the lens of the imaging unit 301 to the imaging plane and the distance z to the subject. It is possible to calculate the motion vector Tv.

  The motion vector detection unit 304 corrects the motion vector calculated in step S1501 using the position change motion vector projected onto the image plane. Such correction is performed based on the following equation.

M ”= M′−Tv · I
M ″ is a matrix representing a motion vector obtained by subtracting the position change motion vector from the motion vector M ′, and M ′ is a matrix representing the motion vector corrected in the processing of step S1502. Further, Tv is an image plane. Is a unit matrix (a matrix having the same size as the matrix M.) In this way, a motion vector due to a position change of the imaging unit 301 is calculated from a motion vector calculated from a real space image. Decrease.

  Finally, the motion vector generated by the position and orientation of the imaging unit 301 is subtracted from the motion vector calculated from the real space image captured by the imaging unit 301. Therefore, as a result, the motion vector of the subject from which the motion vector by the imaging unit 301 is excluded is calculated.

  In the present embodiment, the subject region is specified based on the motion vector in which the influence of the motion of the imaging unit 301 is corrected in this way. Here, in the present embodiment, it is assumed that the motion vector component caused by changes in both the position and orientation is corrected. However, the correction of the motion vector may be performed considering only the posture change of the imaging unit 301 or may be performed considering only the movement change.

  In the present embodiment, the motion vector generated by the movement of the imaging unit 301 is obtained using the position and orientation information obtained from the position and orientation measurement unit 306. However, the motion vector may be obtained by other methods. That is, it is not always necessary to obtain a motion vector based on position and orientation information obtained from a sensor system such as a magnetic sensor or an optical sensor. For example, a motion vector generated by the movement of the imaging unit 301 may be obtained using an image captured by the imaging unit 301.

  For example, the average of the motion vectors of the entire screen imaged by the imaging unit 301 may be assumed to be a motion vector generated by the movement of the imaging unit 301. Further, when the background area is known by dividing the captured image into areas, the motion vector generated in the background area may be assumed to be a motion vector generated by the movement of the imaging unit 301.

[Fifth Embodiment]
In each of the above embodiments, each unit configuring the image processing apparatus 300 illustrated in FIG. 3 has been described as being configured by hardware. However, each unit other than the storage device 310 and the captured image capturing unit 302 is software. It may be realized in the form of a program. In that case, the software program is installed in a computer having the storage device 310 and the captured image capturing unit 302, and the CPU of the computer executes the software program, thereby realizing the operation of each unit. . That is, a computer such as a general PC (personal computer) can be applied to the image processing apparatus 300.

  FIG. 15 is a diagram illustrating a hardware configuration example of a computer applicable to the image processing apparatus 300.

  The CPU 1501 controls the entire computer using programs and data stored in the RAM 1502 and the ROM 1503, and executes the processes described above as performed by the image processing apparatus 300.

  The RAM 1502 has an area for temporarily storing programs and data loaded from the external storage device 1506, various data received from the outside via an I / F (interface) 1507, and the like. The RAM 1502 also has a work area used when the CPU 1501 executes various processes. Further, the RAM 1502 also functions as the frame memory and stencil buffer. That is, the RAM 1502 can provide various areas as appropriate.

  The ROM 1503 stores setting data of the computer, a boot program, and the like.

  The operation unit 1504 is configured by a keyboard, a mouse, and the like, and can input various instructions to the CPU 1501 by being operated by an operator of the computer. For example, a process end instruction or the like can be input using the operation unit 1504.

  The display unit 1505 is configured by a CRT, a liquid crystal screen, or the like, and can display a processing result by the CPU 1501 as an image or text. For example, it is possible to display an image of the mixed reality space generated by the computer (CPU 1501) executing the above-described processes performed by the image processing apparatus 300.

  The external storage device 1506 is a mass information storage device represented by a hard disk drive device. The external storage device 1506 stores an OS (operating system) and programs and data for causing the CPU 1501 to execute the above-described processes that the image processing apparatus 300 performs. The program includes a program for causing the CPU 1501 to execute the functions of the motion vector detection unit 304, the key region extraction unit 303, the subject region extraction unit 305, the image composition unit 308, and the image generation unit 307. The external storage device 1506 also serves as the storage device 310. Programs and data stored in the external storage device 1506 are appropriately loaded into the RAM 1502 under the control of the CPU 1501. Since the CPU 1501 executes processing using the loaded program and data, the computer executes the above-described processing (processing according to the above-described flowcharts) that the image processing apparatus 300 performs. Become.

  An I / F 1507 is used to connect the above-described HMD 390 and position / orientation measurement unit 306 to this computer. The I / F 1507 transmits / receives signals to / from the HMD 390 and position / orientation measurement unit 306 via the I / F 1507. The I / F 1507 also serves as the captured image capturing unit 302.

  A bus 1508 connects the above-described units.

  Note that the hardware configuration of the computer applicable to the image processing apparatus 300 is not limited to the configuration shown in FIG. For example, a graphics card (board) may be attached to the computer, and the graphics card may generate a virtual space image or a mixed reality space image.

[Other Embodiments]
Needless to say, the object of the present invention can be achieved as follows. That is, a recording medium (or storage medium) in which a program code of software that realizes the functions of the above-described embodiments is recorded is supplied to the system or apparatus. Needless to say, such a storage medium is a computer-readable storage medium. Then, the computer (or CPU or MPU) of the system or apparatus reads and executes the program code stored in the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiment, and the recording medium on which the program code is recorded constitutes the present invention.

  Further, by executing the program code read by the computer, an operating system (OS) or the like running on the computer performs part or all of the actual processing based on the instruction of the program code. Needless to say, the process includes the case where the functions of the above-described embodiments are realized.

  Furthermore, it is assumed that the program code read from the recording medium is written in a memory provided in a function expansion card inserted into the computer or a function expansion unit connected to the computer. After that, based on the instruction of the program code, the CPU included in the function expansion card or function expansion unit performs part or all of the actual processing, and the function of the above-described embodiment is realized by the processing. Needless to say.

  When the present invention is applied to the recording medium, program code corresponding to the flowchart described above is stored in the recording medium.

Claims (13)

Means for acquiring a real space image comprising a plurality of frames;
In the frame of interest of the real space image, a first setting means for setting a region composed of a predetermined pixel satisfying a first region,
Within the frame of interest, and acquisition means you get a motion vector of another region other than the first region of the target frame the motion vector of the first region,
First determination means for determining whether or not the magnitude of a motion vector of the acquired other region is greater than or equal to a threshold ;
When the first determination unit determines that the magnitude of the motion vector of the other area is equal to or greater than a threshold, the motion vector of the other area is similar to the motion vector of the first area. A second judging means for judging whether or not
A second region that sets the other region as a second region when the second determination unit determines that the motion vector of the other region is similar to the motion vector of the first region; Setting means,
To the first region and the region other than the second region in the frame of interest is pre-Symbol set, the image processing apparatus characterized by obtaining Bei a synthesizing means for synthesizing the virtual space image. The second setting means includes
If the magnitude of the motion vector of said another region is determined to not more than the threshold value, the second area set in a previous frame from the frame of interest is set as a second area of the frame of interest The image processing apparatus according to claim 1. The second setting means includes
If the magnitude of the motion vector of said another region is determined to not more than the threshold value, a first region and a second region set in a previous frame from the frame of interest, the said frame of interest The image processing apparatus according to claim 1, wherein the image processing apparatus is set as one area and a second area . The second setting means includes
The first determination means determines that the magnitude of the motion vector of the other area is greater than or equal to a threshold value, and the second determination means determines that the motion vector of the other area is a motion of the first area. The second region is set as the second region when the second region is determined to be similar to a vector and the other region is connected to the first region. The image processing apparatus according to any one of 1 to 3. Said first determination means, and wherein the magnitude of the representative motion vector that is determined based on the respective motion vectors are determined Me with a plurality of other regions, it is determined whether or not the threshold value or more The image processing apparatus according to any one of claims 1 to 4. The acquisition means includes
Means for calculating an attitude change amount of the imaging device;
Means for calculating a motion vector generated by a posture change of the imaging device as a posture change motion vector based on the posture change amount;
6. The image processing apparatus according to claim 1 , further comprising: a unit that corrects a motion vector of the first region based on the posture change amount motion vector. The acquisition means includes
Means for calculating real space depth information;
Means for calculating a position change amount of the imaging device;
Means for calculating a motion vector caused by a position change of the imaging device as a position change motion vector based on the depth information and the position change amount;
The image processing apparatus according to claim 1 , further comprising: a unit that corrects a motion vector of the first area based on the position change motion vector. The virtual space image, the image processing according to any one of claims 1 to 7, characterized in Rukoto is generate based on the position and orientation information indicating the position and orientation of an imaging device imaging the real space image apparatus. It said first setting means, to any one of claims 1 to 8, characterized in that sets a region formed in the physical space image in pixels having a predetermined color information as the first region The image processing apparatus described. The image processing apparatus according to claim 1, wherein the obtaining unit obtains a motion vector for each pixel constituting the real space image. The image processing apparatus according to any one of claims 1 to 10, further comprising means for outputting a composite image obtained by the composition processing by the composition means. Acquiring a physical space image composed of a plurality of frames;
In the frame of interest of the real space image, a first setting step of setting an area consisting of a predetermined pixel satisfying a first region,
Within the frame of interest, an acquisition step get a motion vector of the first area regions other than the first region and the motion vector the target frame of,
A first determination step of determining whether or not the magnitude of a motion vector of the acquired other region is equal to or greater than a threshold ;
Whether the motion vector of the other region is similar to the motion vector of the first region when it is determined in the first determination step that the size of the motion vector of the other region is greater than or equal to a threshold value A second determination step for determining
When it is determined in the second determination step that the motion vector of the other region is similar to the motion vector of the first region, the second region is set as the second region. A setting process;
Image processing method characterized in that the first region and the region other than the second region of the target frame that have been pre-Symbol set, obtain Preparations and synthesis step of synthesizing the virtual space image.   A program for causing a computer to execute the image processing method according to claim 12.

Images