JP2019139793A - Character image generation device, character image generation method, program and recording medium - Google Patents

Abstract

To perform suitable compound reality presentation considering a characteristic of a real object existing in a real space.SOLUTION: A character image generation device acquires space information including arrangement of a real object existing in a real space and a characteristic of the real object. And, the device generates a character image by drawing a virtual character in a three-dimensional space structured corresponding to the real space for a predetermined view point in a display mode according to arrangement and a characteristic of the real object. When predetermined information is acquired from a predetermined information source, the device draws the virtual character in the display mode indicating a behavior corresponding to the real object having a characteristic associated with the predetermined information.SELECTED DRAWING: Figure 6

Description

  The present invention relates to a character image generation device, a character image generation method, a program, and a recording medium, and particularly to a technique for recognizing a real space and presenting mixed reality.

  By capturing a specific marker placed in the real space, the position and orientation of the apparatus are estimated, and a 3DCG character or the like is superimposed on the captured image, so that the character appears on a plane that is determined based on the marker. There exist some that present mixed reality as if they existed (Patent Document 1).

  On the other hand, since the technique described in Patent Document 1 places a different marker in the real space, there is a possibility that a suitable reality cannot be presented. Therefore, in recent years, a method of recognizing a plane in a captured image without arranging a marker and presenting mixed reality on the plane is being adopted.

Japanese Patent No. 5469409

Tango, [online], [January 10, 2017 search], Internet, <URL: https: // get. Google. com / tango />

  However, depending on the captured image, a desired plane is not necessarily recognized as a plane on which the character is arranged, and there may be a presentation lacking in reality, for example, a character standing on a table instead of the floor.

  The present invention has been made in view of the above-described problems, and is a character image generation device, a character image generation method, a program, and a program that perform suitable mixed reality presentation in consideration of the characteristics of real objects existing in real space. An object is to provide a recording medium.

  In order to achieve the above-described object, the character image generating apparatus of the present invention includes a first acquisition unit that acquires spatial information including the arrangement of real objects existing in the real space, and characteristics of the real objects, and a predetermined amount. A second acquisition means for acquiring predetermined information output from a given information source, and a drawing means for drawing a three-dimensional space constructed corresponding to the real space for a predetermined viewpoint, A character image generation apparatus that generates a character image in which a virtual character in a three-dimensional space is rendered in a display mode according to the arrangement and characteristics, the drawing means being predetermined by the second acquisition means In response to the acquisition of this information, a real object having characteristics related to the predetermined information is identified, and a virtual cache is displayed in a display mode that shows the behavior corresponding to the real object. And generating a character image drawing the connectors.

  With such a configuration, according to the present invention, it is possible to perform a suitable mixed reality presentation in consideration of the characteristics of a real object existing in the real space.

The system figure which showed the structure of the character image generation system which concerns on embodiment and the modification of this invention The hardware block diagram of the terminal device 10 which concerns on embodiment and the modification of this invention The software block diagram of the terminal device 10 which concerns on embodiment and modification of this invention Software configuration diagram of the server 20 according to the embodiment and the modification of the present invention The figure for demonstrating the various images produced | generated in connection with embodiment and the modification of this invention The flowchart which illustrated the display process performed with the terminal device 10 which concerns on embodiment and the modification 1 of this invention. Another figure for demonstrating the various images produced | generated according to embodiment and the modification of this invention

[Embodiment]
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In addition, one Embodiment described below demonstrates the example which applied this invention to the terminal device which can perform the real space measurement based on a captured image as an example of a character image generation apparatus. However, the present invention can be applied to any device capable of generating a character image having a display mode according to the characteristics of a real object existing in the real space. In this specification, “location description information” refers to information recorded as screen coordinates of the distribution of feature points in an image captured at an arbitrary position, and “three-dimensional spatial information (or simply spatial information)”. "" Refers to a set of information indicating the feature points in the real space and their positions (spatial coordinates) constructed based on the movement and orientation change of the terminal device and the captured images taken along with this. And

"System configuration"
FIG. 1 is a system diagram showing a configuration of a character image generation system according to an embodiment of the present invention. As shown in FIG. 1, the character image generation system according to the present embodiment includes a terminal device 10 that presents a character image to a user, a server 20 that performs analysis related to grasping the characteristics of a real object, which will be described later, and a space between these devices. The network 30 may be a mobile communication network, a wireless LAN, or the like that realizes communication connection and information communication. In the example of FIG. 1, a configuration in which one terminal device 10 is connected to one server 20 is shown. However, the embodiment of the present invention is not limited to this, and a plurality of terminal devices 10 includes one terminal device 10. Needless to say, the server 20 may be configured to be connectable.

<Hardware Configuration of Terminal Device 10>
FIG. 2 is a schematic hardware configuration diagram of the terminal device 10 according to the present embodiment. The terminal device 10 is a portable communication terminal such as a smartphone, for example. The speaker 101 converts an audio signal output from the control unit 100 into audio. The microphone 102 converts the sound into a sound signal and outputs the sound signal to the control unit 100. The imaging unit 103 includes a lens and a CCD sensor, acquires image information, and outputs the image information to the control unit 100. In this embodiment, the display 105 is a touch-sensitive display, and functions as an input unit for user operation in addition to presenting information to the user. When the display is not a touch-sensitive display, an input unit is provided separately. The depth sensor 106 measures the distance (depth) to the object and outputs the measurement result to the control unit 100 as depth information. The communication unit 107 performs communication processing via the network 30 which may be a mobile communication network or a wireless LAN. The geomagnetic sensor 108 detects geomagnetism and outputs azimuth information indicating the current orientation of the reference direction of the terminal to the control unit 100. The acceleration sensor 109 detects acceleration and outputs acceleration information to the control unit 100. The gyro sensor 110 detects angular velocity and outputs angular velocity information to the control unit 100.

  The control unit 100 is a control unit for the entire terminal device, and includes one or more processors and a memory unit. The memory unit includes a RAM used by the processor for temporarily storing data, a flash memory in which an operating system and applications executed by the processor, and data to be retained even when the power is turned off are stored. .

<Software Configuration of Terminal Device 10>
FIG. 3 is a software configuration diagram showing functional configuration modules realized in the terminal device 10 according to the embodiment of the present invention. These functional configuration modules are realized by the corresponding program being expanded in the memory by the control unit 100 and executed.
The location description information generation module 111 generates location description information based on the technique described in Non-Patent Document 1, for example. The location description information is information for determining the three-dimensional space position of the terminal device and the imaging direction of the imaging unit 103 from the image captured by the imaging unit 103. First, generation of location description information described in Non-Patent Document 1 will be described. When the user of the terminal device 10 starts to acquire the location description information, the user changes the position and the imaging direction while capturing an image with the imaging unit 103. The location description information generation module 111 uses the starting position as the origin during acquisition of the location description information, and based on the acceleration information and the angular velocity information, the position of the terminal device and the orientation (posture) of the imaging direction Continue to judge. In addition, the location description information generation module 111 obtains feature points in the image captured by the imaging unit 103. At this time, the location description information generation module 111 determines the position and imaging direction of the terminal device in the three-dimensional space and the position of the same feature point in the image, that is, the screen coordinates of the feature point. As a result, the location description information generation module 111 displays location description information indicating where each feature point can be seen in the image captured by the imaging unit 103 at each location and each imaging direction in the three-dimensional space. Generate. Therefore, the location description information generation module 111 compares the already generated location description information with the feature points of the image captured by the imaging unit 103, and thereby the three-dimensional (final) terminal device at that time. The spatial position and the imaging direction of the imaging unit 103 can be determined. Note that the origin of the three-dimensional space position is the position when the location description information is started. The location description information generation module 111 also compares the location description information already stored with the feature points of the image captured by the imaging unit 103 and the 3D spatial position of the terminal device and the imaging direction of the imaging unit 103. At the same time, the already-stored location description information is updated based on the feature points of the image captured by the imaging unit 103.

  As described above, the location description information is information indicating the relationship between the position and imaging direction of the terminal device and the position of each feature point of each object in the image captured by the imaging unit 103 in the screen coordinates. Based on the location description information, three-dimensional space information indicating the coordinates of the three-dimensional space position of each feature point can be generated. That is, with respect to information corresponding to combinations of different positions and imaging directions in the location description information, information in the depth direction is obtained by considering the movement of the same feature points on the screen coordinates, and the three-dimensional space of the feature points is obtained. The position can be derived. In the present embodiment, the configuration module 112 configures three-dimensional space information indicating the three-dimensional positions of feature points existing in the real space based on the location description information. The three-dimensional spatial information may be point group data indicating the distribution of the spatial coordinates of the feature points, and may manage the three-dimensional coordinates or color information and the like for each point.

  Note that the origin of the spatial position related to the three-dimensional spatial information may be, for example, the same as the position of the terminal device 10 when the location description information generation module 111 starts generating the location description information. The coordinate system of the three-dimensional spatial information to be configured is preferably determined so that the vertical direction (y-axis) is parallel to the vertical direction determined by the geomagnetic sensor 108. Further, the configuration module 112 updates the information of the three-dimensional position of the corresponding feature point of the already stored three-dimensional space information in synchronization with the update of the place space information.

  Here, the three-dimensional space information is configured in an aspect showing a scale substantially the same as the real space based on the acceleration information and the angular velocity information which are actually measured values, and the size of the real object existing in the real space can be acquired. However, it will be easily understood that it may be configured at a predetermined scale. Further, since the terminal device 10 of the present embodiment includes the acceleration sensor 109 and the gyro sensor 110, the dimensions of the real object can be acquired from the three-dimensional space information. However, the embodiment of the present invention includes these sensors. Is not a requirement. That is, for each feature point in the image corresponding to the image information captured by the captured image 103, the distance in the depth direction to the feature point can be acquired based on the depth information acquired from the depth sensor 106. Therefore, the three-dimensional space information of the scale that is considered to be the same as the real space can be configured by considering the correlation between the images corresponding to the plurality of image information captured at different posture positions and the depth information. Is possible.

  The character control module 113 controls the movement of a virtual character displayed on the display 105 as an example of presentation of mixed reality. The virtual character has a three-dimensional model showing its appearance stored in advance, and can control the motion of the virtual character based on the three-dimensional model or based on motion data provided for general use. Further, the control of the virtual character is not limited to the visually presented motion, and may include, for example, control related to voice presentation, decision making, and the like. In the present embodiment, the virtual character can be visualized as a character image by being placed in a predetermined three-dimensional space (hereinafter referred to as a world) provided corresponding to the real space and drawn. The world corresponds to the real space grasped by the three-dimensional space information, but is constructed separately from the three-dimensional space information composed of point cloud data and provided for drawing. Of course, the information on the shape and arrangement of the obstacle grasped from the three-dimensional space information is also reflected in the world, and is considered in the action and display restriction of the virtual character. The origin of the world coordinate system may be provided at a position corresponding to the three-dimensional spatial information, and the relative relationship (or coordinate transformation matrix) between the coordinate system of the three-dimensional spatial information and the world coordinate system is defined. It may be different in some cases.

  The viewpoint determination module 114 determines a viewpoint for drawing the world based on the position and orientation information of the terminal device 10 determined by the place description information generation module 111. The viewpoint for drawing the world is determined to have a line-of-sight direction corresponding to the attitude of the terminal device 10 at a position in the world corresponding to the position of the terminal device 10 in the real space. Here, the information on the viewing angle related to the viewpoint may be determined so as to correspond to the angle of view of the imaging unit 103.

  The drawing module 115 draws the virtual character placed in the world and controlled by the place description information generation module 111 for the viewpoint determined by the viewpoint determination module 114, and generates a character image. Specifically, the drawing module 115 draws an image of an area related to the virtual character in the image finally displayed on the display 105 as shown in FIG.

  The synthesizing module 116 synthesizes the character image generated by the drawing module 115 corresponding to the captured image with the captured image corresponding to the image information acquired by the imaging unit 103, and presents it for presentation related to mixed reality presentation. Generate an image. As a result of the synthesis, the captured image showing the real space as shown in FIG. 5B becomes a presentation image (FIG. 5C) in a state where the virtual character exists in the real space, and is displayed to the user via the display 105. Presented.

  As described above, in the present embodiment, the image for presentation is generated by synthesizing the captured image corresponding to the image information acquired by the imaging unit 103 and the character image generated in the drawing module 115. Therefore, in the drawing of the character image, the area where the virtual character is shielded by the real object existing in the real space does not have to appear on the character image. Specifically, when it can be determined from the depth information or the three-dimensional space information that a real object exists in front of the virtual object, the virtual object corresponding to the real object is placed in the world, and the drawing module 115 The virtual object is not drawn, and the virtual character is not drawn for the portion of the virtual character shielded by the virtual object (for example, FIG. 7A). By doing so, it is possible to present a realistic image (FIG. 7B) in which the context of the real object and the virtual character is matched when combined with the captured image.

<Software Configuration of Server 20>
The hardware configuration of the server 20 of this embodiment is omitted as long as it is configured to be able to execute communication with the terminal device 10 and predetermined processing. Hereinafter, the software configuration showing the functional configuration module realized in the server 20 of the present embodiment will be described with reference to FIG. Similar to the terminal device 10, these functional configuration modules are realized by a control unit, a flash memory, a RAM, and the like included in the server 20 (not illustrated).

  The analysis module 201 performs an analysis related to a real object existing in the imaging range based on one or more captured images captured by the terminal device 10. In this embodiment, the analysis module 201 includes a specifying module 202 that specifies which real object is, and a characteristic grasping module 203 that grasps the characteristics of the real object and outputs characteristic information. Here, the characteristic of the real object includes information (so-called affordance) indicating a normal use grasped from the type and shape such as a chair or a table. In addition, in the case of a real object that can affect other real objects by performing its function, such as a lighting fixture or a blower, the characteristics are necessary to cause the virtual character to have the influence. Various parameters may be included.

  In addition, the identification of which real object is performed by the identification module 202 is performed by comparing with a database in which a three-dimensional shape, a texture pattern, or the like is registered in advance based on size / shape / color information or various learning. It may include analysis based on a neural network that is constructed via the above.

  In addition, the real object information and the real object characteristic information referred to by the analysis module 201 do not have to be stored in the server 20 in advance. For example, the user captures an image on the terminal device 10. The region of the desired real object in the selected image is specified, and which real object or the characteristic of the real object is input, which is transmitted to the server 20 and held available for analysis. It is good also as a thing.

  For example, when the real object is a chair, the specifying module 202 specifies the area estimated as the real object in the captured image and the type of the real object is a chair based on the image information of the area, or any chair (Product) is specified. Then, the characteristic grasping module 203 also considers the specific result, and as characteristic information, “the virtual character can be seated on the object (in addition, which surface is the seat)” or “the surface Information indicating that the virtual character does not climb on the object is output.

  Also, for example, when the real object is a lighting fixture, the specifying module 202 specifies a region estimated as a real object in the captured image, and the type of the real object is lighting, or any lighting fixture (product). To identify. Then, the characteristic grasping module 203 considers the specific result, and as characteristic information, “the object defines a light source in the world”, “the luminance expression or shadow expression of the virtual character is changed by the object”, and In addition, information on at least one of the luminance of the light source, the irradiation direction, and the type of light source (direct illumination and indirect illumination) necessary for deriving the influence of luminance change or the like is output. Here, various types of information necessary for derivation of the influence may be analyzed based on, for example, a luminance distribution in the captured image, surrounding color information, and the like.

  The real object region information in the input captured image and the real object characteristic information acquired by the analysis module 201 in this way are transmitted to the network 30 by a communication unit (not shown) of the server 20. And returned as an analysis result to the terminal device 10. The analysis result may be managed in association with, for example, three-dimensional space information, so that the feature point forms a part of a predetermined real object or the characteristic of the real object related to the feature point is determined. It becomes possible to grasp whether there is.

《Display processing》
A specific process of the display process of the terminal device 10 according to the present embodiment having such a configuration will be described with reference to the flowchart of FIG. The processing corresponding to the flowchart can be realized by causing the control unit 100 to operate each module by reading a corresponding processing program stored in, for example, a flash memory, developing the program in a RAM, and executing the program. The present display process will be described as being started when a predetermined application related to mixed reality presentation is started and executed repeatedly for each frame displayed on the display 105, for example.

  In step S <b> 601, the imaging unit 103 and the depth sensor 106 perform imaging and sensing related to a frame to be displayed under the control of the control unit 100, and acquire image information, acceleration information, angular velocity information, and depth information.

  Hereinafter, in the terminal device 10 processing for location description information generation, position and orientation determination, and 3D spatial information configuration (S602 to S604) of the terminal device 10 and analysis of an actual object in the imaging range to be performed by the server 20 Such processing (S605 to S606) is executed in parallel.

  In step S602, the location description information generation module 111 determines whether there is already stored location description information. If the location description information generation module 111 determines that there is already stored location description information, the processing proceeds to S603. When the location description information generation module 111 determines that the saved location description information does not exist, the location description information generation module 111 determines the position and orientation of the terminal device 10 as initial values (the origin position and no rotation), and performs processing. Move on to S604.

  In step S <b> 603, the location description information generation module 111 determines the position and orientation of the terminal device 10 based on the image information, acceleration information, angular velocity information, and already stored location description information acquired in relation to the display frame. In the present embodiment, for the sake of simplicity, description will be made assuming that the location description information that has already been stored includes information on at least some feature points of the current imaging range. If at least some of the feature points are not included in the stored location description information, the position and orientation are determined as initial values as in the case where the stored three location description information does not exist. Just do it.

  In step S <b> 604, the configuration module 112 configures or updates the three-dimensional space information corresponding to the location space information updated based on the image information acquired for the display frame. The configuration of the three-dimensional spatial information is performed from the generation when the three-dimensional spatial information is not yet configured, and in response to the update of the location space information at the configured phase.

  On the other hand, for the processing related to the analysis of the real object to be performed by the server 20, the control unit 100 transmits the image information acquired in relation to the display frame to the server 20 together with the analysis request via the communication unit 107 in S605. Thereafter, in step S606, the control unit 100 waits until the analysis result related to the request is returned. When the analysis result is received, the process proceeds to step S607. Note that the image information transmitted together with the analysis request does not need to indicate the entire imaging range. For example, in response to a user request, the actual information is preferentially generated in the virtual character so that an action related to a specific real object is generated. An instruction input of a desired small area related to the object may be received and image information of the small area may be transmitted. By doing so, the amount of information related to the analysis request can be reduced, and the amount of analysis processing can be reduced, so that the processing speed can be increased.

  In step S <b> 607, the configuration module 111 associates the received analysis result (real object region information and real object characteristic information) with the corresponding feature point of the three-dimensional space information. In the present embodiment, the description will be made assuming that the three-dimensional spatial information and the analysis result are integrated. However, the implementation of the present invention is not limited to this, and it is only necessary that the information is associated with each other. .

  In S608, the character control module 113 determines the state of the virtual character corresponding to the display frame, and performs various processes for controlling the virtual character to the state and arranging it in the world. At this time, the character control module 113 may control the behavior of the virtual character according to the characteristics of the real object existing around the position in the real space corresponding to the position where the virtual character is arranged. Note that the behavior control of the virtual character does not need to start the behavior based on the characteristics of all the real objects included in the imaging range of the display frame at the same time. May be performed sequentially in accordance with an order (for example, the order of distances close to each other) or a predetermined priority. Further, when a specific motion is performed on one real object, control may be performed so that the motion does not occur for a predetermined period even if it is included in the imaging range so that the motion does not frequently occur.

  In step S609, the drawing module 115 draws the world for the viewpoint determined by the viewpoint determination module 114 based on the position and orientation information of the terminal device 10 related to the display frame, and generates a character image. When a real object whose characteristics are grasped as a light source exists in the imaging environment, a light source based on the characteristic information related to the real object is defined in the world, and the drawing module 115 performs a luminance calculation due to the light source.

  In S610, the synthesis module 116 generates a presentation image by synthesizing the generated character image with the captured image corresponding to the image information acquired in relation to the display frame.

  In step S611, the control unit 100 displays the presentation image on the display 105, and completes the display process related to the display frame.

  As described above, in the display process of the present embodiment, the display mode of the virtual character can be controlled according to the characteristics of the real object existing in the real space by the processes of S608 and S609. The control of the virtual character according to the characteristics of the real object may be performed as follows, for example.

(1) Control due to affordance of a real object As described above, when the characteristic information related to the use of the real object is acquired, the real object existing in the imaging environment can be grasped instead of simply being an obstacle. The virtual character can be controlled to take an action according to the characteristic information. For example, when the real object is a chair, a cushion, or the like and it can be grasped that it is used for sitting, the behavior control may be performed so that the virtual character is seated on the real object. Alternatively, if the real object is a table or the like and it can be grasped that the object is not for sitting but for object placement, the virtual character is not placed on the real object, but the character's wearing items can be placed. The behavior may be controlled such that the virtual character walks avoiding the real object. Further, for example, when the real object is a painting, a modeled object, or the like, and it can be grasped that it is an ornamental use, the virtual character may perform behavior control so as to watch the real object.

(2) Control caused by characteristics that affect other real objects When characteristic information that affects other real objects is acquired as a result of the functions of the real objects, it is caused by real objects that exist in the imaging environment. The depiction of the virtual character can be controlled so that a change occurs. For example, if the real object is a lighting fixture and if it can grasp the characteristics of causing a change in brightness or shading of the surrounding real object by functioning, the virtual character is within a predetermined range affected by the real object. May be arranged to control the brightness of the virtual character and the state of the shadow to be generated in relation to the character based on the position of the virtual character, the position of the real object, and information on the light source characteristics. In addition, for example, when the real object is an air conditioner and it is possible to grasp the characteristic that the function (sway) of the surrounding real object such as a curtain is caused by functioning, the predetermined range affected by the real object In the case where a virtual character is arranged on the screen, the behavior of hair, clothes, etc. may be controlled. Of course, whether or not it affects other real objects in this way depends on the operating conditions of the lighting fixtures and air conditioners, so the analysis module 201 changes the characteristic information depending on the operating conditions, or The analysis result may be transmitted by adding information on whether to perform control according to the characteristic information, and behavior control in the terminal device 10 may be performed adaptively.

  Note that these controls are merely examples, and other behavior control may be performed according to the characteristics of the real object existing in or around the position of the real space corresponding to the placement position of the virtual character. In addition to the state in which the virtual character is arranged at a position corresponding to the real object, for example, when the feature points of the real object whose characteristics are grasped are included in the three-dimensional space information, the action of the virtual character is caused by this. The behavior control may be performed so as to generate. That is, when it is already known that a chair is present around the terminal device 10 when the application is executed, behavior control is performed so that the virtual character is moved in the chair direction and seated. It may be a thing.

  In addition, even if the type is specified, such as a window, the characteristics of the object itself do not affect other real objects. When it is in the state given to the real object, the characteristic grasping module 203 is configured to grasp this based on the luminance distribution of the image information and output the characteristic information as a real object that gives a lighting effect to other real objects. May be.

  In addition, not only real objects that have characteristics that cause other real objects to change depending on their driving conditions, such as lighting equipment and air conditioners, but for real objects whose operating state can be determined by analysis, depending on the state. The characteristic information may be varied to change the behavior of the virtual character. For example, in the case of a television receiver, it is possible to analyze the driving state and the program being viewed based on the image information of the area corresponding to the screen, so that the virtual character starts behavior such as program guidance and watching operation You may comprise as follows.

  Moreover, although the analysis module 201 analyzed based on image information, specified the real object, and the aspect which returns the characteristic information to the terminal device 10 were demonstrated in this embodiment, implementation of this invention is limited to this. It is not a thing. The real object is specified based on a user operation performed on the terminal device 10 such as selection from a product list, input of a product name, and the like, and characteristic information related to the product is configured to be acquired from the server 20. There may be.

  Further, the characteristic information about an arbitrary real object does not necessarily have to be constant. For example, the characteristic information may be updated at a time zone, a date and time, or at a predetermined timing, and may be configured to cause a different behavior in a virtual character. Good.

  Further, in the character image generation system of the present embodiment, the function is separated between the terminal device 10 and the server 20 in order to present mixed reality to the user, and the server 20 executes only the characteristic analysis of the real object. However, the function separation mode is not limited to this. That is, the present invention can be realized as long as the configuration of the three-dimensional spatial information, the characteristic analysis of the real object, and the generation of the character image are executed by one or more devices.

  As described above, according to the character image generation apparatus of the present embodiment, it is possible to perform a suitable mixed reality presentation in consideration of the characteristics of a real object existing in real space.

[Modification 1]
In the above-described embodiment, the character control module 113 controls the behavior of the virtual character according to the characteristics of the real object around the position in the real space corresponding to the position where the virtual character is arranged. Implementation of the invention is not limited to this.

  The behavior control of the virtual character has characteristics related to the information in accordance with, for example, the acquisition of predetermined information via the communication unit 107 or the output of predetermined information by executing an application such as a scheduler. It may be performed so as to perform an action related to a real object. Here, the predetermined information is information (additional information) of a predetermined type that is presented through a virtual character, and is, for example, predetermined information such as weather forecast, broadcast program information, or commentary information on exhibits. The information may be supplied from an external device, or may be acquired from another application started by the terminal device 10 such as schedule information or search result information. Further, the characteristics related to the additional information may be grasped based on information indicating the characteristics grasped for the real object attached to the additional information, and the data structure, the distribution source, the character string, etc. It may be grasped by analyzing. In addition, the additional information may include voice information issued by the user.

  In this modification, the character control module 113 moves the virtual character to a position corresponding to the real object when additional information related to the characteristics of the real object already managed in association with the three-dimensional space information is acquired. The behavior control is performed so as to cause the additional information to perform a predetermined action. For example, when characteristic information indicating a television receiver is managed as a real object, when broadcast program information is received as additional information, the character control module 113 moves the virtual character to a position in the world corresponding to the television receiver. Then, while pointing to the television receiver, the behavior control may be performed so as to present “a program currently being broadcast and its information”, “guidance of the start time of a program that matches the user preference”, etc. with sound. Also, for example, when characteristic information indicating a predetermined exhibit is managed as a real object, when the comment information is acquired as additional information, the character control module 113 stops the virtual character at a position in the world corresponding to the exhibit. The behavior control may be performed so that the explanation of the exhibit starts. Also, for example, when characteristic information indicating a clock or a front door is managed as a real object, the character control module 113 moves the virtual character to the corresponding position in the world when acquiring schedule information related to the going-out schedule as additional information. Then, the behavior control may be performed so that the notification of the schedule or the notification that prompts the movement is performed.

  By doing so, it is possible to present a mixed reality presentation with higher added value according to the usage environment of the user and the characteristics of the arranged real object, not just for the purpose of viewing.

[Modification 2]
In the embodiment and the modification described above, the character image generated by the drawing module 115 and the captured image are combined to generate a presentation image that presents mixed reality and is displayed on the display 105. Although explained, presentation of a character image is not restricted to this. In other words, the above-described embodiment gives recognition that the virtual character exists in the real space through the presentation image displayed on the display 105. For example, an HMP (Head-Mounted Projector) or a glass-type device is used. For example, an apparatus using an optical transmission method may be used instead of the display 105. In this case, only the pixels related to the character of the character image need only be configured to be visible to the user using the optical transmission method, and the user recognizes the virtual character as if it exists in real space while directly observing the real world. be able to. In other words, it is easy to understand that the combination of the character image and the captured image is not essential in the embodiment of the present invention, and is an element adopted according to the presentation method. When a glass-type device or the like is used, it is preferable that the imaging unit 103 is provided in the device itself or a position corresponding to the device for determining the position and orientation.

[Modification 3]
In the embodiment and the modification described above, it has been described that the characteristics of the real object are grasped based on the captured image acquired in relation to the display frame. However, the characteristics grasp of the real object that exists in the space observed by the user is described. This may be performed based on a captured image obtained by an imaging device provided at a position different from the user's viewpoint or the terminal device 10.

  Similarly, a character image may be generated by drawing the world for a viewpoint that is different from the position and orientation determined based on the image information, acceleration information, and angular velocity information acquired in relation to the display frame. That is, in the aspect that does not require the synthesis of the captured image and the character image as shown in the modified example 2, the eyeball position of the user to be viewed and the installation position of the imaging unit 103 do not necessarily match, and therefore, the relative position of these The world may be drawn with respect to a viewpoint determined based on the positional relationship and the estimated position and orientation information.

[Other Embodiments]
The present invention is not limited to the above-described embodiment, and various changes and modifications can be made without departing from the spirit and scope of the present invention. The character image generation apparatus according to the present invention can also be realized by a program that causes one or more computers to function as the apparatus. The program can be provided / distributed by being recorded on a computer-readable recording medium or through a telecommunication line.

  10: terminal device, 20: server, 30: network, 100: control unit, 101: speaker, 102: microphone, 103: imaging unit, 105: display, 106: depth sensor, 107: communication unit, 108: geomagnetic sensor , 109: acceleration sensor, 110: gyro sensor, 111: place description information generation module, 112: configuration module, 113: character control module, 114: viewpoint determination module, 115: drawing module, 116: composition module, 201: analysis module 202: specific module 203: characteristic grasping module

Claims (8)

First acquisition means for acquiring space information including an arrangement of a real object existing in the real space and characteristics of the real object;
Second acquisition means for acquiring predetermined information output from a predetermined information source;
A drawing means for drawing a three-dimensional space constructed corresponding to the real space for a predetermined viewpoint, wherein a virtual character in the three-dimensional space is displayed in a display mode according to the arrangement and characteristics of the real object. A character image generating device having a drawing means for generating a drawn character image,
The drawing means identifies the real object having characteristics related to the predetermined information in response to the acquisition of the predetermined information by the second acquisition means, and displays a behavior corresponding to the real object. A character image generating apparatus, wherein a character image in which the virtual character is drawn is generated in the display mode shown.   The character image generating apparatus according to claim 1, wherein the predetermined information is information of a predetermined type that is presented through the virtual character.   The character according to claim 1 or 2, wherein the characteristic related to the predetermined information is specified by analyzing at least one of a data structure, a distribution source, and a character string of the predetermined information. Image generation device.   The behavior corresponding to the real object having the characteristic related to the specified predetermined information is moved to a position corresponding to the real object in the three-dimensional space, and a predetermined action is performed on the predetermined information. The character image generation device according to claim 1, wherein the character image generation device is a character image generation device. The real object is a device whose operating state changes,
The character image generation device further includes a changing unit that acquires an operating state of the real object and changes characteristics of the real object according to the acquired operating state.
5. The drawing unit according to claim 1, wherein when the characteristic of the real object is changed by the changing unit, the display mode of the virtual character is changed according to the changed characteristic. The character image generation device according to any one of the preceding claims. A first acquisition step of acquiring spatial information including an arrangement of a real object existing in the real space and characteristics of the real object;
A second acquisition step of acquiring predetermined information output from a predetermined information source;
A drawing step of drawing a three-dimensional space constructed corresponding to the real space with respect to a predetermined viewpoint, wherein a virtual character in the three-dimensional space is displayed in a display mode according to the arrangement and characteristics of the real object. A character image generation method comprising: a drawing step of generating a drawn character image,
In the drawing step, in response to the acquisition of the predetermined information in the second acquisition step, the real object having characteristics related to the predetermined information is specified, and a behavior corresponding to the real object is determined. A character image generation method, wherein a character image in which the virtual character is drawn is generated in the display mode shown.   The program for functioning a computer as each means of the character image generation apparatus of any one of Claims 1 thru | or 5.   A computer-readable recording medium on which the program according to claim 7 is recorded.