JP2003288611A - Image processing device and image transmission system - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing device which can display images which is suitable for the circumstance without providing sense of discomfort, or images having highly entertaining. <P>SOLUTION: Three-dimensional information of a player's head during playing a video game which is obtained by calculating using reflected-light images is input to an input part 57 of the image processing device. The image of the player's head is built into a position of a character's head of the video game by the image processing device 5. During the development of the scenario of the game, when the character turns its head toward a slanting direction, even if the image of the imaged player's head faces the front, the image of the head is processed to correspond with the direction of the character's head. <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and an image transmission system applicable to, for example, a computer game.

[0002]

2. Description of the Related Art Conventionally, many games using a computer have been provided, and computer games (hereinafter referred to as games) have become one of people's entertainment. With the development of computers, visually and aurally realistic games are realized and entertain the players.

[0003]

When a real object such as a head including a user's face is captured and used as an image in a game, the object is imaged and an attribute (for example, shape, It is necessary to input information on color and movement) into the game device. Examples of a device that can acquire the attribute of an object include a normal color camera, a stereo image processing device, and a laser range finder.

According to a normal color camera, a background color is formed by a color that does not exist in an object, the object is imaged, and the background is distinguished from the object, thereby obtaining information such as the shape and color of the object. be able to. Moreover, since a moving image can be taken, it is possible to obtain motion information when an object moves. However, in the case of an object having the same color as the background color, there are many restrictions such that the attribute information cannot be obtained, and the information on the three-dimensional shape and the movement information thereof cannot be obtained.

According to the stereo image processing apparatus, it is possible to obtain not only the color but also the three-dimensional shape information by performing processing such as image recognition on an image obtained by using an ordinary color camera. Also, the motion information can be obtained by consecutive frames. However, this device requires a large amount of calculation to obtain shape information, and is not suitable for obtaining motion information. Moreover, since the processing is performed by referring to the apparent contents of the image, the processing becomes unstable, or the processing cannot be performed at all, and there are many restrictions.

The laser range finder applies a surveying technique using a laser beam, and according to this, information on the three-dimensional shape of an object can be acquired with high accuracy. Since the laser range finder cannot obtain the color information, it is necessary to obtain the color information by using it together with an ordinary camera. But,
Since the three-dimensional shape information is acquired by scanning the surface of the object, it takes time to measure. Since the game device is required to have high speed (preferably real-time) image processing, it is not realistic to apply the laser range finder to the game device.

Further, according to these devices, since the picked-up image is simply displayed on the screen, it is not possible to provide a display that does not cause a sense of discomfort or a display that is entertaining according to the situation.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an image processing apparatus and an image transmission system capable of displaying a display suitable for the situation without a sense of incongruity and having entertainment. .

[0009]

An image processing apparatus according to the present invention includes input means for inputting three-dimensional shape information of an object calculated from a reflected light image of the object, and three-dimensional shape information input to the input means. In order to make the image different from the image of the object displayed on the screen by the calculation means, calculation means for calculating the adjustment information for adjusting the three-dimensional shape information input to the input means, the adjustment information calculated by the calculation means, and And a generation unit that generates an image of the object based on the three-dimensional shape information input to the input unit.

According to the image processing apparatus of the present invention, the adjustment information for adjusting the three-dimensional shape information calculated from the reflected light image of the object is calculated and displayed on the screen based on this and the three-dimensional shape information. An image of an object is created. Therefore,
An image different from the image of the object displayed on the screen can be displayed on the screen by the three-dimensional shape information. As a result, instead of simply displaying the captured image of the object on the screen, it is possible to perform a display that does not cause a sense of incongruity or a display that is entertaining according to the situation.

The generation means of the image processing apparatus according to the present invention is
It is possible to adjust the three-dimensional shape information based on the adjustment information and change the direction of the image of the object on the screen. According to this, the direction of the image of the object displayed on the screen can be made different from the direction in which the object is imaged.

The generating means of the image processing apparatus according to the present invention is
The adjusting means can adjust the three-dimensional shape information so as to dramatize the image of the object on the screen. According to this, the image of the object can be adapted so that the image can be made entertaining.

The generation means of the image processing apparatus according to the present invention is
It is possible to generate a moving image of the color stereoscopic image in which the object is extracted by combining the color planar image of the object including the background and the stereoscopic shape information. According to this, a realistic image can be obtained.

In the image transmission system according to the present invention, in order to make the image different from the image of the object displayed on the screen by the three-dimensional shape information of the object calculated from the reflected light image of the object, the image is input to the input means. Calculation means for calculating adjustment information for adjusting three-dimensional shape information, generation means for generating an image of an object based on the adjustment information and three-dimensional shape information calculated by the calculation means, and an object generated by the generation means A transmission side system including a transmission means for transmitting the image of
A receiving side system including a receiving means for receiving the image of the object transmitted from the transmitting means, and a receiving side system including a display means for displaying the image of the object received by the receiving means.

According to the image transmission system of the present invention,
Even in a distant place, it is possible to exchange images that are suitable for the situation and that have an amusement effect.

[0016]

DETAILED DESCRIPTION OF THE INVENTION An image processing apparatus according to an embodiment of the present invention will be described below with reference to the drawings. First, the outline of the first embodiment will be described with reference to FIG. 7. In the first embodiment, the image of the head of the player (CG: Computer Graphic
s) is applied to the game where the character's head is used. This image is a moving image of a color stereoscopic (three-dimensional) image. By capturing an image of the player's head in the game (the head is included in the head, and the following heads also have this meaning), the facial expression of the player in the game is expressed in real time. Reflected in.

Since the player usually plays the game in the same direction with respect to the image pickup unit, the head image of the character in the game can be obtained by simply displaying the image of the head imaged by the image pickup unit on the screen as it is. The image is always in the same direction (for example, facing forward). Therefore, while the story of the game is unfolding, even if the character is in an oblique direction, only the image of the head is in the frontal direction, which is uncomfortable.

In the first embodiment, the three-dimensional shape information of the head calculated from the reflected light image of the player's head H1 (a silhouette image, which will be described in detail later) is used as the adjustment information. (For example, the head direction parameter) is adjusted. As a result, even if the player is playing the game in the same direction (frontward in this case), if the character is in a different direction, the image G1 of the head is also oriented in this direction (diagonal in this case). ) Can be Therefore, according to the first embodiment, it is possible to provide a display (natural display) that does not cause a sense of discomfort that suits the situation. In addition, since this image is a color three-dimensional moving image, it can be a realistic image.

The above is the outline of the first embodiment. Next, details of the first embodiment will be described. FIG. 1 is a functional block diagram of the game system 1 according to the first embodiment. The game system 1 is a target object (for example, a player's head).
Object information acquisition device 3 that acquires information (three-dimensional shape, color, movement, and other external characteristics of the object) necessary to display the object on the game screen, and the object information acquisition device 3 that matches the story of the game. An image processing device 5 that performs a reconstruction process that changes the direction of an image, and an image display device 7 that displays the processed image on a game screen are provided.

FIG. 2 shows an example of embodying the game system 1.
Is. The game system 1 includes an operation device 9 for the player to input operation information such as pressing a button, a home-use game processing device 11 for processing the entire game, and a CRT 13 on which a game screen is displayed.

The operation device 9 includes an object information acquisition device 3 of FIG. 1 and an image sensor 15 which is an image pickup unit included in the object information acquisition device 3.
Is built in. The image processing device 5 of FIG. 1 is incorporated in the home-use game processing device 11. The image processing device 5 may be incorporated in an arcade game processing device or an ordinary personal computer. The CRT 13 is an example of the image display device 7 in FIG. In addition to this, the image display device 7 includes an LCD, an HMD.
(Head-mounted display), projector and so on.

Next, the configuration of the object information acquisition device 3 will be described in detail with reference to FIG. FIG. 3 is a functional block diagram of the object information acquisition device 3. This device 3 is an image information acquisition unit 17 for acquiring image information including color information of an object in real time, and three-dimensional shape information of the object in real time. The three-dimensional shape information acquisition unit 19 for acquiring. The acquisition units 17 and 19 can also acquire the motion information of the object in real time as described later.

The image information acquisition unit 17 captures an object with natural light (including illumination light) to acquire a natural light image that is a color image of the object including the background, and the operations of the image capturing unit 21. The image pickup operation control unit 23 that generates a control signal for controlling the natural light image storage unit 25 in which the natural light image captured by the image pickup unit 21 is stored, and the natural light image stored in the storage unit 25 is read out, A natural light image adjusting unit 27 that adjusts the brightness and contrast as necessary, and information of the natural light image adjusted by the adjusting unit 27 (this is the image information acquired by the acquisition unit 17) as an output destination device ( In the first embodiment, an image information output unit 29 that outputs in conformity with the protocol and data format of the home game processing device).

The natural light image pickup unit 21 is included in the image sensor 15 of FIG. 2, and is, for example, a CCD or CMOS image pickup device, and acquires a color plane image of an object including a background in real time. As a result, the color of the attributes of the object can be acquired. Further, since the color images are continuously captured, it is possible to grasp the motion information of the object from the change of the continuous frames. Thereby, the image of the object can be made into a moving image.

On the other hand, the three-dimensional shape information acquisition unit 19 takes a reflected light image pickup unit 31 for picking up a reflected light image of an object and a subject around it according to a control signal from the control unit 23, and an image pickup by the image pickup unit 31. The reflected light image storage unit 33 in which the reflected light image is stored, the reflected light image correction unit 35 that performs various corrections by reading the reflected light image stored in the storage unit 33, and the reflected light image for correcting the reflected light image A parameter storage unit 37 that stores parameters, a three-dimensional shape information calculation unit 39 that analyzes the reflected light image corrected by the correction unit 35 to calculate three-dimensional shape information of an object, and a three-dimensional object calculated by the calculation unit 39. The stereoscopic shape information output unit 41 that outputs the shape information in conformity with the protocol or the data format of the output destination device (the home game processing device in the first embodiment).

The reflected light image pickup unit 31 is included in the image sensor 15 shown in FIG. Here, details of the reflected light image capturing unit 31 will be described with reference to FIG. FIG. 4 is a functional block diagram of the image pickup unit 31. The reflected light image pickup unit 31 includes the light emitting unit 4
3, a light receiving optical system 45, a reflected light extraction unit 47, and a timing control unit 49.

The light emitting section 43 generates light whose intensity varies with time according to the timing signal generated by the timing control section 49. This light is applied to the player's head (an example of an object) in front of the light emitting unit 43. The light generated here is near infrared light, but not limited to this, light in other wavelength regions such as visible light can also be used.

The reflected light from the player's head is condensed by the light receiving optical system 45 composed of a lens or the like and is focused on the light receiving surface of the reflected light extraction unit 47. The light receiving optical system 45 is provided with a filter that passes near infrared light. External light such as visible light and far infrared light other than near infrared light is cut out of the reflected light by this filter.

The reflected light extraction section 47 extracts the spatial intensity distribution of the reflected light forming the above-mentioned image formation. This intensity distribution can be captured as an image of reflected light. This is the reflected light image (silhouette image). In order to achieve this function, the reflected light extraction unit 47 includes a first light receiving unit 51, a second light receiving unit 53, and a difference calculation unit 55. The first light receiving unit 51 and the second light receiving unit 53 perform light reception at different timings. Then, the timing control section 49 controls these so that the light emitting section 43 emits light when the first light receiving section 51 is receiving light and the light emitting section 43 does not emit light when the second light receiving section 53 is receiving light. Control the operation timing.
As a result, the first light receiving unit 51 receives the reflected light of the light from the light emitting unit 43 by the object and the other natural light (that is, external light such as sunlight and illumination light). On the other hand, the second light receiving unit 53
Receives only natural light. Since the two receive light at different timings, they are close to each other, so that fluctuations in natural light during this period can be ignored.

Therefore, if the difference between the image received by the first light receiving section 51 and the image received by the second light receiving section 53 is calculated, only the component of the light reflected by the object is extracted from the light from the light emitting section 43. , A reflected light image can be generated. The difference calculation unit 55 calculates and outputs the difference between the images received by the first light receiving unit 51 and the second light receiving unit 53.

The reflected light from the object greatly decreases as the distance between the object and the reflected light image pickup unit 31 increases. When the surface of the object scatters light uniformly, the amount of light received per pixel of the reflected light image decreases in inverse proportion to the square of the distance to the object. Therefore, since the intensity value of the reflected light from the object can be converted into the distance value from the reflected light image pickup unit 31 to the object, the three-dimensional shape of the object can be grasped. Also, the reflected light from the background is almost negligible. Therefore, it is possible to obtain a reflected light image of the object and the subject around the object with the background cut.

The reflected light image can be obtained, for example, by an apparatus having the following configurations (1) to (3).
(1) Timing signal generating means for generating a pulse signal or a modulation signal that is constant in time or changes in time. (2) Light emitting means for emitting light whose intensity changes based on the signal generated by the timing signal generating means. (3) A means for separating and detecting the reflected light of the light emitted from the light emitting means by the object in synchronization with the signal from the timing signal generating means is arranged, and Reflected light extraction means for detecting a reflected light image from an object.

A more detailed description of the above-mentioned components of the reflected light image pickup section 31 is described in Japanese Patent Application Laid-Open No. 10-177449 filed by the same applicant as this case.

Since the reflected light image pickup unit 31 continuously picks up reflected light images of a subject including an object, the movement information of the object can be grasped from continuous frame changes. Thereby, the image of the object can be made into a moving image. The reflected light image capturing section 31 constitutes the image sensor 15 shown in FIG. 2 together with the natural light image capturing section 21. As described later, a color stereoscopic image in which an object is extracted by combining a natural light image (a color planar image of an object including a background) and stereoscopic shape information obtained by calculating a reflected light image of the object. (CG of object) is generated. Therefore, since it is necessary to make the pixels of the reflected light image correspond to the pixels of the natural light image, the reflected light image capturing unit 31 and the natural light image capturing unit 2
It is preferable that 1s are arranged close to each other. From this point of view, it is preferable that the imaging units 21 and 31 are integrated into one chip.

Next, the correction in the reflected light image correction section 35 shown in FIG. 3 will be described. First, correction is performed to extract the object.
As described above, the reflected light image captured by the image capturing unit 31 is an image of the object and the subject around the object with the background cut. Therefore, the object is detected by using the pattern of the object (for example, the head) stored in advance in the parameter storage unit 37 or the natural light image stored in the natural light image storage unit 25, and the other subjects are deleted. Make corrections.

Next, the correction is performed in consideration of the color and reflection characteristics of the object. More specifically, the intensity of the reflected light from the object depends on factors other than the distance between the object and the reflected light image pickup unit 31. Therefore, even if the distance is simply calculated from the reflected light image, the distance (that is, the three-dimensional shape) may not be accurate. For example, when the color of the object surface is black, the intensity of the reflected light is reduced. Further, when the surface of the object contains a lot of specular reflection components, strong reflected light is generated at a portion where the normal line of the surface of the object is close to the light source direction. Therefore, before obtaining the distance information from the reflected light intensity of the object, the correction unit 35 refers to the parameters, such as the surface color and the reflection characteristic of the surface of the object that are stored in advance in the parameter storage unit 37, or the storage unit 25 The reflected light image of the object is corrected by referring to the stored natural light image corresponding to the reflected light image. Thereby, the accuracy of the three-dimensional shape information of the object calculated next is improved.

The three-dimensional shape information calculation section 39 analyzes the reflected light image corrected by the correction section 35 to calculate three-dimensional shape information of the object. More specifically, since the intensity value of the reflected light from the object can be converted into the distance value from the reflected light image pickup unit 31 to the object, the three-dimensional shape information of the object is calculated from this distance value.

Next, details of the image processing apparatus 5 shown in FIG. 1 will be described. FIG. 5 is a functional block diagram of the device 5.
The image processing device 5 receives the image information (color plane image information of an object including a background) and three-dimensional shape information output from the object information acquisition device 3 shown in FIG.
And a storage unit 59 that stores these pieces of information input to the input unit 57, and an object direction parameter (viewpoint position) for changing the direction of the object on the screen of the image display device 7.
And an arithmetic unit 61 for performing arithmetic operations such as changing
The object image generation unit 63 that generates an image (CG) of the object using the information such as the direction parameter calculated in step S6, and the image of the object generated by the generation unit 63 are displayed on the screen of the image display device 7. And an output unit 65 for outputting the output.

The calculation unit 61 reads the three-dimensional shape information of the object stored in the storage unit 59 and calculates the reference coordinates (or the barycentric coordinates) for generating the CG of the object, and the reference calculation unit 67. And a direction parameter calculation unit 69 that performs a calculation for adjusting the direction parameter of the image of the object by adjusting the upper viewpoint. The direction parameter is an example of adjustment information.

The object image generator 63 stores information such as reference coordinates and direction parameters calculated by the calculator 61 and the storage unit 5.
The image information and the stereoscopic shape information stored in 9 are combined to generate a color stereoscopic image in which the object is extracted. That is, the storage unit 59 stores image information and three-dimensional shape information. The image information is a planar color image of the object including the background. On the other hand, the three-dimensional shape information is a silhouette image (reflected light image) of an object whose background is cut. The generation unit 63 is a color three-dimensional image in which the background is cut by superimposing these images, cutting out a color image portion corresponding to an object from the color image, and texture-mapping this to a silhouette image. Generate an object image (CG).

According to the object information acquisition device 3 and the image processing device 5 described above, the object image (CG) can be generated at high speed (or in real time). Note that the processing in the object image generation unit 63 can use the technology (especially, the fifth embodiment) of Japanese Patent Application Laid-Open No. 10-177449 filed by the same applicant as the present case.

Of the configurations of the object information acquisition device 3 and the image processing device 5, the function of the storage unit is that capable of storing information, for example, a device formed of a magnetic medium such as a hard disk, a semiconductor such as an EEPROM. It can be realized by a memory. Other configuration features are:
For example, it can be realized by various processors (CPU, MPU) or ASIC and a predetermined program.

Next, the operation of the game system 1 according to the first embodiment will be described mainly with reference to FIGS. 6 and 7. FIG. 6 is a flow chart for explaining this. FIG. 7 is a diagram showing the relationship between the player's head during the game and the player's head on the screen.

When the player operates the operation device 9 (FIG. 2), instruction polling corresponding to the operation occurs in the game system 1 (step S1). Then, it is determined whether or not to start the image processing of the first embodiment (step S3). If the command polling corresponds to an operation of selecting a game that incorporates the image of the player's head on the character's head, image processing begins.

The object information acquisition device 3 images the player to acquire image information and three-dimensional shape information (step S5). The object information acquisition device 3 tries to detect the head image of the player from the three-dimensional shape information (step S7). It is determined whether or not the player's head image is successfully detected (step S9). If it is not detected, the detection process is tried until the upper limit of the number of repetitions of the detection process is reached (step S11), and if the upper limit is reached, an error is displayed on the screen of the image display device 7 and the process ends (step S13). .

On the other hand, when the head image of the player is detected, the reference coordinates of this head image are calculated (step S1).
5). The barycentric coordinates may be used instead of the reference coordinates. When calculating these coordinates, the standard information of the head image of the player stored in advance in the storage unit 59 of the image processing apparatus 5 may be used. For example, when only a part of the player's head can be imaged, the standard information of the player's head image is used to complement the missing information in order to reliably calculate the reference coordinates and the barycentric coordinates. .

When the direction of the character's body changes diagonally from the front on the story of the game,
The direction parameter of the image of the head is adjusted so that the image of the player's head is in the same oblique direction as the direction of the character's body.

Next, an image (CG) of the player's head is generated using the reference coordinates, the direction parameter, the image information and the three-dimensional shape information (step S17). Then, the generated image of the head is processed into a format that can be used by the application software of this game, and is output from the image processing unit 5 (step S19). The output image of the head is displayed on the screen of the image display device 7.

As described above, even if the player plays the game with the head H1 facing the front with respect to the image sensor 15 (FIG. 2) as shown in FIG. 7, the image G1 of the player's head is the body of the character. The image is displayed on the screen of the image display device 7 in the same oblique direction as the direction. Then, when the facial expression of the player changes during the game, for example, the head H1 changes to the head H2, the image G1 of the head changes on the screen in real time. Therefore, according to the first embodiment,
In the case of a game in which the image of the player's head is incorporated in the head of the character of the game, it is possible to display without a sense of discomfort that matches the situation of the game story.

Next, the second embodiment will be described. Here, like the image G3 of the head of the player shown in FIG. 7, the dramatization processing of the eyes and the mouth is performed. The game system according to the second embodiment differs from that according to the first embodiment shown in FIG. 1 in part of the configuration of the image processing device 5. Of the constituent elements of the second embodiment, the same constituent elements as those of the first embodiment are designated by the same reference numerals, and a description thereof will be omitted. Differences will be mainly described.

FIG. 8 is a functional block diagram of the image processing apparatus 5 included in the game system according to the second embodiment. The image processing apparatus 5 shown in FIG. 8 further includes, as components of the calculation section 61 of the apparatus 5 shown in FIG. 5, a parts image detection section 70 for detecting a parts image from stereoscopic shape information of the player's head,
A parameter calculation unit 71 that calculates a deformation parameter and a movement parameter of a part image, and a draWing color calculation unit 73 that calculates a draWing color for these parameters are provided.

The parts image detection unit 70 includes a parts pattern storage unit 72 in which patterns of parts of the object (for example, eyes, nose, mouth, and other characteristic parts) are stored in advance.
The parts image detection unit 70 extracts the image information and the three-dimensional shape information of the player's head stored in the storage unit 59. Then, the part image is detected from the three-dimensional shape information using the image information and the part pattern of the part pattern storage unit 72 (pattern matching in image processing).
This parts image is stored in the storage unit 59. By detecting the parts image from the plurality of frames of the three-dimensional shape information of the player's head, the history of the parts image is stored in the storage unit 59. The part image includes three-dimensional shape information and position information of the part image. The three-dimensional shape information of the part image is included in the three-dimensional shape information of the player's head.

The transformation and movement parameter calculation unit 71 is
Deformation parameters of the part image based on the history of the three-dimensional shape information of the part image and movement parameters of the part image based on the history of the position information of the part image are calculated. These parameters are an example of adjustment information.

The draWing color calculation unit 73 calculates a draWing color such as emphasizing, attenuating, deforming or reversing these parameters. These dramatized parameters are used by the object image generation unit 6
3 and is used to generate an image of the object.

Next, the operation of the game system according to the second embodiment will be described mainly with reference to FIGS. 7, 9 and 10. 9 and 10 are flowcharts for explaining this. First, steps S1 to S1 shown in FIG.
5, the difference from steps S1 to S15 shown in FIG. 6 is the condition for starting the image processing in step S3. That is, in the second embodiment, the image processing is started in the case of the command polling corresponding to the game in which the head image of the player in which the color of the head of the character has been dramatized is incorporated.

In step S15, after the reference coordinates of the player's head image are calculated, an attempt is made to detect a part image such as an eye or a mouth constituting this head image from each frame of the player's head image. (Step S23). Then, it is determined whether or not the detection of the part image is successful (step S25). If not detected, the detection process is tried until the upper limit of the number of repetitions of the detection process is reached (step S27). When the upper limit is reached, an error is displayed on the screen of the image display device 7 and the process ends (step S2).
9).

On the other hand, when parts images (eyes, mouth, etc.) are detected from a frame with the head image of the player, the three-dimensional shape information and position information of these parts images are stored in the storage unit 59 (FIG. 8). (Step S31). Then, when the parts image is detected for the other frames of the head image of the player, the same processing as step S31 is performed. As a result, past history information of parts such as eyes and mouth can be obtained.

Next, it is judged whether or not the three-dimensional shape information and the position information of the parts image are accumulated in an amount necessary for calculating the deformation parameter and the movement parameter (step S33). If not stored, the process returns to step S23. On the other hand, in the case of being accumulated, as shown in FIG. 10, the deformation parameter of the part image of each part is calculated using the history of the three-dimensional shape information of the part image within a predetermined time (step S35). . As this calculation method, for example, there is calculation of the amount of deformation of the eyes and mouth using calculation processing such as difference comparison in image processing.

Next, it is judged whether or not an effective deformation parameter that can be used for the color change has been calculated among the deformation parameters calculated in step S35 (step S37).
When the effective deformation parameter is calculated, the color-ratio calculation unit 7
3 calculates the dramatization such as emphasis, attenuation, deformation, and inversion for this deformation parameter (step S39). The calculation of the draWing color is realized by applying an algorithm prepared in advance in the draWing color computing unit 73 to the effective deformation parameters.

Next, in the object image generating unit 63, the image information and the three-dimensional shape information of the part image stored in the storage unit 59 are taken out, and the part image is deformed using the dramatized deformation parameter. (Step S41).
For example, processing is performed such that the degree of opening of the eyes is further increased or the bulge of the cheek is enlarged.

Next, after the deformation processing of the part image, or when the effective deformation parameter is not calculated in step S37, the history of the position information of the part image within a predetermined time is used to obtain the part image of each part. The movement parameter is calculated (step S43). An example of this calculation method is a pattern tracking technique in image processing. With this technique, for example, movement parameters such as the degree of eyebrow fishing and the lower jaw falling can be calculated.

Next, it is judged whether or not an effective movement parameter that can be used for a dramat has been calculated from the movement parameters obtained in step S43 (step S45).
When the effective movement parameter is calculated, the leg color calculation unit 73 applies the prepared algorithm to the effective movement parameter, and emphasizes, attenuates, deforms, reverses, etc. the movement parameter. Is calculated (step S47).

Then, the object image generating unit 63 takes out the image information and the three-dimensional shape information of the part image stored in the storage unit 59. Then, the moving processing of the part image is performed by the adapted moving parameter (step S4).
9). For example, raising the fishing position of the eyebrows,
Processing to lower the lower jaw position further.

The following steps S51 and S53 correspond to steps S17 and S19 of FIG. However, in the case of the second embodiment, since the parts image is deformed and moved, as shown in FIG. 7, the image G3 of the player's head on the screen is a dramatized image. For more information, "Open your eyes",
Specific examples of the dramatization contents for the items "Open Mouth" and "Raise Eyebrows" include "Enlarge the size of the eyes. Make the eyeballs pop out.", "Open the mouth extremely wide. "Emphasize the position of the eyebrows. Emphasize the deformation of the eyebrows."

As described above, according to the second embodiment, as shown in FIG. 7, the head image of the player is dramatized in real time like G3 during the game, so that the entertainment can be enhanced. . Further, similarly to the first embodiment,
Since the image G3 of the player's head is displayed on the screen of the image display device 7 in the same direction as that of the character's torso, it is possible to perform a display that does not feel uncomfortable according to the situation of the game story.

Next, a third embodiment will be described. The third embodiment is an image transmission system using the dramatization of the second embodiment. FIG. 11 is a functional block diagram of the image transmission system 75. Of the constituent elements of the third embodiment, the same constituent elements as those of the first and second embodiments are designated by the same reference numerals, and a description thereof will be omitted. Differences will be mainly described.

The image transmission system 75 includes a transmission side system 77 for generating and transmitting images such as head images G1, G2, G3 shown in FIG. 7, and a reception side system for receiving and displaying the transmitted images. And a system 79.

The transmission side system 77 receives the image information processed by the image processing apparatus 5 via a communication line such as the Internet 81 in addition to the object information acquisition apparatus 3 and the image processing apparatus 5 shown in FIG. The transmitter 83 is provided for transmitting to the side system 79. FIG. 12 shows the transmitting side system 77.
It is a figure which shows the system which materialized the functional block of.
The notebook computer 89 includes the object information acquisition device 3, the image processing device 5, and the transmission unit 83. The image sensor 15 attached to the notebook computer 89 corresponds to the natural light image capturing unit 21 and the reflected light image capturing unit 31 that form the object information acquisition device 3.

On the other hand, the receiving system 79 stores the image received by the receiving unit 85 and the receiving unit 85 that receives the image transmitted from the transmitting unit 83 via the communication line such as the Internet 81. An image processing device 87 that performs processing necessary for displaying the image and an image display device 7 that displays the processed image are provided. FIG. 13 is a diagram showing a system in which the functional blocks of the receiving side system 79 are embodied.

According to the third embodiment, by using a communication line, a dramatized image such as the image G3 of the head shown in FIG. 7 is displayed in real time even in a remote place different from the place where the image is taken. be able to. Therefore, the third
When the embodiment is applied to, for example, a videophone, when a user's emotion appears in a facial expression during a conversation, an image adapted from the facial expression is displayed on the image display device 7 of the other user. As a result, the videophone can be provided with entertainment.

[0071]

According to the image processing apparatus of the present invention, by adjusting the stereoscopic shape information input to the input means,
An image different from the image of the object displayed on the screen can be displayed on the screen based on the stereoscopic shape information before adjustment.
As a result, instead of simply displaying the captured image of the object on the screen, it is possible to perform a display that does not cause a sense of incongruity or a display that is entertaining according to the situation.

According to the image transmission system of the present invention,
Even in a distant place, it is possible to exchange images that are suitable for the situation and that have an amusement effect.

[Brief description of drawings]

FIG. 1 is a functional block diagram of a game system according to a first embodiment.

FIG. 2 is a diagram showing a system in which the functional blocks shown in FIG. 1 are embodied.

FIG. 3 is a functional block diagram of the object information acquisition device according to the first embodiment.

FIG. 4 is a functional block diagram of a reflected light image capturing unit according to the first embodiment.

FIG. 5 is a functional block diagram of the image processing apparatus according to the first embodiment.

FIG. 6 is a flowchart for explaining the operation of the game system according to the first embodiment.

FIG. 7 is a diagram showing a relationship between a player's head during a game and an image of the player's head on the screen.

FIG. 8 is a functional block diagram of an image processing apparatus according to a second embodiment.

FIG. 9 is a flowchart for explaining the first half of the operation of the game system according to the second embodiment.

FIG. 10 is a flowchart for explaining the latter half of the operation of the game system according to the second embodiment.

FIG. 11 is a functional block diagram of an image transmission system according to a third embodiment.

FIG. 12 is a diagram showing a system in which the functional blocks of the transmitting side system shown in FIG. 11 are embodied.

13 is a diagram showing a system in which the functional blocks of the receiving side system shown in FIG. 11 are embodied.

[Explanation of symbols]

1 ... Game system 9: Operating device 11 ... Home game processing device 13 ... CRT 15 ... Image sensor 31 ... Reflected light image pickup unit 45 ... Receiving optical system 47 ... Reflected light extraction unit 75 ... Image transmission system 77 ... Sending side system 79 ... Receiving system 81 ... Internet 89-notebook personal computer

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Takahiro Harashima             1st Komukai Toshiba-cho, Sachi-ku, Kawasaki-shi, Kanagawa             Inside the Toshiba Research and Development Center (72) Inventor Isao Mihara             1st Komukai Toshiba-cho, Sachi-ku, Kawasaki-shi, Kanagawa             Inside the Toshiba Research and Development Center (72) Inventor Miwako Doi             1st Komukai Toshiba-cho, Sachi-ku, Kawasaki-shi, Kanagawa             Inside the Toshiba Research and Development Center F term (reference) 2C001 BB00 BB08 BC00 BC08 CA09                 5B050 AA10 BA06 BA07 BA08 BA09                       CA07 DA01 EA19 EA24 EA27                       EA28 FA02 FA05 FA06                 5C054 AA01 AA05 CA05 CC02 CD03                       DA06 EA05 FA00 FC11 FD02                       FE02 GC03 HA15

Claims (5)

[Claims] 1. An input unit for inputting three-dimensional shape information of the object calculated from a reflected light image of the object, and the three-dimensional shape information of the object displayed on the screen by the three-dimensional shape information input to the input unit. Calculating means for calculating adjustment information for adjusting the three-dimensional shape information inputted to the input means to make the image different from the image; and inputting the adjustment information calculated by the calculating means and the input means An image processing apparatus comprising: a generation unit configured to generate an image of the object based on the generated three-dimensional shape information. 2. The image processing apparatus according to claim 1, wherein the generation unit adjusts the three-dimensional shape information based on the adjustment information to change a direction of an image of the object on the screen. 3. The image processing apparatus according to claim 1, wherein the generation unit adjusts the three-dimensional shape information by the adjustment unit to dramatize the image of the object on the screen. 4. The generation means generates a moving image of a color stereoscopic image in which the object is extracted by combining the color planar image of the object including a background and the stereoscopic shape information. The image processing device according to claim 1. 5. The three-dimensional shape information input to the input means in order to make the image different from the image of the object displayed on the screen by the three-dimensional shape information of the object calculated from the reflected light image of the object. Calculating means for calculating adjustment information for adjusting the object, generating means for generating an image of the object based on the adjustment information and the three-dimensional shape information calculated by the calculating means, and generated by the generating means. A transmitting side system including a transmitting unit that transmits the image of the object, a receiving unit that receives the image of the object transmitted from the transmitting unit, and a display that displays the image of the object received by the receiving unit. An image transmission system comprising: a receiving side system including means.