JP6517255B2 - Character image generation apparatus, character image generation method, program, recording medium, and character image generation system - Google Patents

Description

  The present invention relates to a character image generation apparatus, a character image generation method, a program, a recording medium, and a character image generation system, and more particularly to a technology for recognizing real space and performing mixed reality presentation.

  The position and orientation of the device are estimated by imaging a specific marker placed in real space, and a 3DCG character or the like is superimposed on the captured image to display the character as if it were determined on the basis of the marker. There has been a method of presenting mixed reality as if it exists (Patent Document 1).

  On the other hand, in the technique described in Patent Document 1, there is a possibility that a suitable sense of reality can not be presented because a heterogeneous marker is placed in the real space. Therefore, in recent years, a method of recognizing a plane in a captured image without arranging a marker and presenting mixed reality on the plane is being adopted.

Patent No. 5469409 gazette

Tango, [online], [search January 10, 2017], Internet, <URL: https: // get. google. com / tango />

  However, depending on the captured image, a desired surface may not necessarily be recognized as a plane on which the character is arranged, and for example, a presentation without a sense of reality may be made, such as the character standing on a table instead of a floor surface.

  The present invention has been made in view of the above problems, and is a character image generation apparatus, character image generation method, program for performing preferable mixed reality presentation considering characteristics of a real object existing in a real space. An object of the present invention is to provide a recording medium and a character image generation system.

In order to achieve the above-mentioned object, according to the character image generation device of the present invention, based on a first acquisition unit for acquiring a captured image obtained by imaging a real space, and a captured image acquired by the first acquisition unit, A configuration means for constructing spatial information including arrangement of real objects existing in real space, a second acquisition means for acquiring characteristics of real objects, and a three-dimensional space constructed corresponding to real space Drawing means for drawing an image pickup device that has picked up an image and a viewpoint corresponding to the image, and generating a character image in which a virtual character in a three-dimensional space is drawn in a display mode corresponding to the arrangement and characteristics of a real object; a character image generating apparatus having the real object is a device which operation state is changed, the character image generating apparatus acquires an operation state of the real object The image processing apparatus further includes changing means for changing the property of the real object according to the acquired operation state, and the drawing means displays the property changed by the changing means for the real object and the display mode according to the arrangement of the real object. , Generating a character image in which a virtual character is drawn .

  With such a configuration, according to the present invention, it is possible to perform preferable mixed reality presentation in consideration of the characteristics of the real object existing in the real space.

A system diagram showing a configuration of a character image generation system according to an embodiment and a modification of the present invention Hardware configuration of the terminal device 10 according to the embodiment and the modification of the present invention Software configuration of the terminal device 10 according to the embodiment and the modification of the present invention Software configuration of the server 20 according to the embodiment and modification of the present invention The figure for demonstrating the various images produced | generated according to embodiment and modification of this invention The flowchart which illustrated the display processing which is performed with terminal unit 10 concerning an embodiment and modification 1 of the present invention Another figure for explaining various images generated according to the embodiment and modification of the present invention

[Embodiment]
Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the drawings. In the embodiment described below, an example in which the present invention is applied to a terminal device capable of real space measurement based on a captured image as an example of a character image generation device will be described. However, the present invention is applicable to any device capable of generating a character image of a display manner according to the characteristics of a real object present in real space. Further, in the present specification, “location description information” refers to information in which the distribution of feature points in an image captured at an arbitrary position is recorded as screen coordinates, and “three-dimensional space information (or simply space information) "" Refers to a set of information indicating feature points in real space and their positions (space coordinates), which are constructed based on movement or posture change of the terminal device and a captured image captured accordingly I assume.

"System configuration"
FIG. 1 is a system diagram showing the configuration of a character image generation system according to an embodiment of the present invention. As shown in FIG. 1, the character image generation system of the present embodiment includes a terminal device 10 that presents a character image to a user, a server 20 that performs analysis related to grasping the characteristics of real objects described later, and the devices And a network 30, which may be a mobile communication network or a wireless LAN, for realizing communication connection and information communication. Although the example of FIG. 1 shows a configuration in which one terminal device 10 is connected to one server 20, the embodiment of the present invention is not limited to this, and a plurality of terminal devices 10 is one. It goes without saying that the server 20 may be configured to be connectable.

<Hardware Configuration of Terminal Device 10>
FIG. 2 is a schematic hardware configuration diagram of the terminal device 10 according to the present embodiment. The terminal device 10 is, for example, a portable communication terminal such as a smartphone. The speaker 101 converts an audio signal output from the control unit 100 into an audio. The microphone 102 converts voice into a voice signal and outputs the voice signal to the control unit 100. The imaging unit 103 includes a lens and a CCD sensor, acquires image information, and outputs the image information to the control unit 100. In the present embodiment, the display 105 is a touch-sensitive display, and in addition to presenting information to the user, it also functions as an input unit for user operation. If the display is not a touch sensitive display, an input unit is provided separately. The depth sensor 106 measures the distance (depth) to the object and outputs the measurement result to the control unit 100 as depth information. The communication unit 107 performs communication processing via the network 30, which may be a mobile communication network or a wireless LAN. The geomagnetic sensor 108 detects geomagnetism and outputs, to the control unit 100, direction information indicating the current direction of the reference direction of the terminal. The acceleration sensor 109 detects an acceleration and outputs acceleration information to the control unit 100. The gyro sensor 110 detects angular velocity and outputs angular velocity information to the control unit 100.

  The control unit 100 is a control unit of the entire terminal device, and includes one or more processors and a memory unit. The memory unit includes a RAM that the processor uses for temporary data storage, and a flash memory that stores an operating system and applications that the processor executes and data that should be held even if the power is turned off. .

<Software Configuration of Terminal Device 10>
FIG. 3 is a software configuration diagram showing functional configuration modules implemented in the terminal device 10 according to the embodiment of the present invention. These functional configuration modules are realized by the control unit 100 developing a corresponding program and executing it.
The place description information generation module 111 generates place description information based on, for example, the technology described in Non-Patent Document 1. The place description information is information for determining the three-dimensional spatial position of the terminal device and the imaging direction of the imaging unit 103 from the image captured by the imaging unit 103. First, generation of place description information described in Non-Patent Document 1 will be described. When the user of the terminal device 10 starts acquisition of place description information, while capturing an image by the imaging unit 103, the user changes the position and the imaging direction. The place description information generation module 111 uses the position at the start time as the origin during acquisition of the place description information, and based on the acceleration information and the angular velocity information, the (temporary) position of the terminal device or the orientation (posture) of the imaging direction Continue to judge. In addition, the place description information generation module 111 obtains feature points in the image captured by the imaging unit 103. At this time, the place description information generation module 111 determines the position of the terminal device in the three-dimensional space and the imaging direction, and the position of the same feature point in the image, that is, the screen coordinates of the feature point. As a result, the place description information generation module 111 generates place description information indicating which position in the image captured by the imaging unit 103 each feature point can be viewed at each position in the three-dimensional space and each imaging direction. Generate Therefore, the place description information generation module 111 compares the already generated place description information with the feature points of the image captured by the imaging unit 103 to obtain the (final) three-dimensional terminal device at that time. The spatial position and the imaging direction of the imaging unit 103 can be determined. The origin of the three-dimensional space position is assumed to be the position when the place description information is started. In addition, the place description information generation module 111 compares the place description information already stored with the feature points of the image captured by the imaging unit 103, and the three-dimensional spatial position of the terminal device and the imaging direction of the imaging unit 103. At the same time, the location description information already stored is also updated based on the feature points of the image captured by the imaging unit 103.

  As described above, the location description information is information indicating the relationship between the position and imaging direction of the terminal device and the position of each feature point of each object in the image captured by the imaging unit 103 in the screen coordinates, Based on the place description information, three-dimensional space information indicating the coordinates of the three-dimensional space position of each feature point can be generated. That is, for information corresponding to a combination of different positions and imaging directions in the place description information, information in the depth direction is obtained by considering the movement of the same feature point on screen coordinates, and the three-dimensional space of the feature point The position can be derived. In the present embodiment, the configuration module 112 configures three-dimensional space information indicating the three-dimensional position of feature points present in the real space based on the place description information. The three-dimensional space information may be point cloud data indicating a distribution of space coordinates of the feature points, and may manage color information or the like for each point in addition to the three-dimensional coordinates or this.

  In addition, the origin of the space position which concerns on three-dimensional space information may be the same as the position of the terminal device 10, for example, when the place description information generation module 111 starts the generation of the place description information. It is preferable that the coordinate system of the three-dimensional space information to be configured is determined so that the vertical direction (y axis) is parallel to the vertical direction determined by the geomagnetic sensor 108. Further, the configuration module 112 updates the information of the three-dimensional position of the corresponding feature point of the three-dimensional space information already stored, in synchronization with the update of the place space information.

  Here, the three-dimensional space information is configured in a manner showing a scale substantially the same as the real space based on the acceleration information and the angular velocity information which are measured values, and the dimensions etc. of the real object existing in the real space can be obtained It will be readily understood that it may be configured at a predetermined scale, although it will In addition, since the terminal device 10 according to the present embodiment includes the acceleration sensor 109 and the gyro sensor 110, the dimensions of the real object can be obtained from three-dimensional space information, but the embodiment of the present invention includes these sensors. Does not have to be a requirement. That is, the distance in the depth direction to the feature point can be obtained based on the depth information obtained from the depth sensor 106 for each of the feature points in the image corresponding to the image information captured by the captured image 103 Therefore, by considering the correlation between images corresponding to a plurality of pieces of image information captured at different posture positions and depth information, it is necessary to configure three-dimensional spatial information of a scale that is considered identical to the real space. Is possible.

  The character control module 113 performs operation control on a virtual character displayed on the display 105 as an example of mixed reality presentation. A virtual character has a three-dimensional model indicating its appearance stored in advance, and its motion can be controlled based on motion data provided for the three-dimensional model or generally. The control of the virtual character is not limited to the operation presented visually, but may include, for example, control relating to voice presentation, decision making, and the like. In the present embodiment, the virtual character can be visualized as a character image by arranging in a predetermined three-dimensional space (hereinafter, identified as a world) provided corresponding to the real space and drawing. The world also corresponds to a real space grasped by three-dimensional space information, but is constructed separately from three-dimensional space information composed of point cloud data and provided for drawing. Of course, information on the shape and arrangement of obstacles grasped from three-dimensional space information is also reflected in the world, and is considered in the action and display restriction of the virtual character. The coordinate system origin of the world may be provided at a position corresponding to the three-dimensional space information, or the relative relationship (or coordinate conversion matrix) between the coordinate system of the three-dimensional space information and the world coordinate system is defined. The case may be different.

  The viewpoint determination module 114 determines a viewpoint to draw a world based on the information of the position and orientation of the terminal device 10 determined by the location description information generation module 111. The viewpoint from which the world is drawn is determined to have a gaze direction corresponding to the attitude of the terminal device 10 at a position in the world corresponding to the position of the terminal device 10 in the real space. Here, the information of the viewing angle related to the viewpoint may be determined to correspond to the angle of view of the imaging unit 103.

  The drawing module 115 draws a virtual character arranged in the world and subjected to operation control by the place description information generation module 111 with respect to the viewpoint determined by the viewpoint determination module 114 to generate a character image. Specifically, the drawing module 115 draws an image of a region related to a virtual character among images to be finally displayed on the display 105 as shown in FIG. 5A.

  The synthesizing module 116 synthesizes the character image generated by the drawing module 115 corresponding to the captured image with respect to the captured image corresponding to the image information acquired by the imaging unit 103, for presentation related to mixed reality presentation. Generate an image. As a result, the captured image showing the real space as shown in FIG. 5B becomes a presentation image (FIG. 5C) in a state in which the virtual character exists in the real space. Will be presented.

  As described above, in the present embodiment, the presentation image is generated by combining the captured image corresponding to the image information acquired by the imaging unit 103 and the character image generated by the drawing module 115. Therefore, in the drawing of the character image, the area where the virtual character is occluded by the real object existing in the real space may not appear on the character image. Specifically, when it is possible to determine that a real object exists in front of a virtual object based on depth information or three-dimensional space information, the virtual object corresponding to the real object is placed in the world, and the drawing module 115 Does not draw the virtual object, and performs control so as not to draw the virtual character for the portion of the virtual character that is covered by the virtual object (for example, FIG. 7A). By doing this, it is possible to present a realistic image presentation (FIG. 7B) in which the anteroposterior relationship between the real object and the virtual character is consistent when combined with the captured image.

<Software Configuration of Server 20>
The hardware configuration of the server 20 according to the present embodiment is omitted as long as communication with the terminal device 10 and predetermined processing can be performed. Hereinafter, the software configuration showing the functional configuration module realized in the server 20 of the present embodiment will be described using FIG. 4. Similar to the terminal device 10, these functional configuration modules are realized by a control unit, a flash memory, a RAM, and the like that the server 20 (not shown) has.

  The analysis module 201 analyzes the real object present in the imaging range based on the one or more captured images captured by the terminal device 10. In the present embodiment, the analysis module 201 includes a specific module 202 that specifies which real object is, and a property recognition module 203 that recognizes the property of the real object and outputs property information. Here, the characteristics of the real object include, for example, information (so-called affordance) indicating a typical application grasped from the type and shape, such as a chair or a table. In addition to this, for example, in the case of a real object such as a lighting fixture or a fan, which is capable of affecting other real objects by its function being implemented, it is necessary to cause the virtual character to have an influence on the characteristic. May be included.

  In addition, identification of which real object is performed by the identification module 202 is, based on size, shape, color information, etc., comparison with a database in which three-dimensional shapes, texture patterns, etc. are registered in advance, or various learnings. It may include analysis based on a neural network constructed through.

  Further, the real object information and the real object characteristic information to be referred to by the analysis module 201 do not have to be all held in advance in the server 20, and for example, the user is imaged in the terminal device 10 The area of the desired real object in the selected image is specified, and which real object or which property of the real object is input is transmitted to the server 20 and held available for analysis. It is good also as a thing.

  For example, when the real object is a chair, the identification module 202 identifies the area presumed to be a real object in the captured image and the image information of the area, and the type of the real object is a chair or which chair Identify if it is a product. Then, the characteristic grasping module 203 takes into consideration the specific result, and as characteristic information, “can sit on the virtual character with respect to the object (in addition, which surface is the seating surface)” or “this Output information indicating that the virtual character does not climb on the object.

  Also, for example, when the real object is a lighting fixture, the specification module 202 specifies a region estimated to be a real object in the captured image, and that the type of the real object is lighting, or any lighting fixture (product) Identify the Then, the characteristic grasping module 203 takes into consideration the specific result, and as characteristic information, “the object defines a light source in the world”, “the luminance expression or the shading expression of the virtual character is changed by the object”, And outputting at least one of information on the brightness of the light source, the direction of illumination, and the type of light source (direct illumination or indirect illumination), which are necessary for deriving the influence of a change in luminance, etc. Here, various pieces of information necessary for deriving the influence may be analyzed based on, for example, the luminance distribution in the captured image, the color information of the surroundings, and the like.

  Note that the information on the area of the real object in the input captured image thus acquired by the analysis module 201 and the characteristic information on the real object are transmitted to the network 30 by the communication unit (not shown) of the server 20. It is returned to the terminal device 10 as an analysis result through the communication. The analysis result may be managed, for example, in association with three-dimensional space information, so that the feature points form a part of a predetermined real object, or the characteristics of the real object concerning the feature points are It becomes possible to know if there is any.

Display processing
About the display processing of the terminal device 10 of this embodiment which has such a structure, a specific process is demonstrated using the flowchart of FIG. The processing corresponding to the flowchart can be realized by operating each module by the control unit 100 reading out a corresponding processing program stored in, for example, a flash memory, and expanding and executing the program in the RAM. The present display processing is described as being started when, for example, a predetermined application related to mixed reality presentation is started, and repeatedly executed for each frame displayed on the display 105.

  In step S601, under the control of the control unit 100, the imaging unit 103 and the depth sensor 106 perform imaging and sensing related to a frame to be displayed, and acquire image information, acceleration information, angular velocity information, and depth information.

  Hereinafter, for processing (S602 to S604) related to location description information generation of the terminal device 10, position and orientation determination, and configuration of three-dimensional space information performed in the terminal device 10, and analysis of a real object in an imaging range to be performed Such processing (S605 to S606) is assumed to be executed in parallel.

  In step S602, the place description information generation module 111 determines whether the place description information already stored is present. If the place description information generation module 111 determines that there is already stored place description information, the process proceeds to step S603. When the place description information generation module 111 determines that the stored place description information does not exist, it determines the position and orientation of the terminal device 10 as an initial value (origin position and no rotation), and performs processing. Move to S604.

  In step S603, the place description information generation module 111 determines the position and orientation of the terminal device 10 based on the image information acquired regarding the display frame, the acceleration information, the angular velocity information, and the place description information already stored. In the present embodiment, for the sake of simplicity, it is assumed that the place description information already stored includes information of at least a part of feature points of the current imaging range. If at least some of the feature points are not included in the stored place description information, the position and orientation are set as initial values as in the case where the stored three place description information does not exist. Just do it.

  In step S604, the configuration module 112 configures or updates three-dimensional space information in accordance with the location space information updated based on the acquired image information pertaining to the display frame. The configuration of the three-dimensional space information is performed from the generation of the three-dimensional space information at the stage where the three-dimensional space information is not yet configured, and the update of the location space information at the configured stage.

  On the other hand, with regard to the process related to the analysis of the real object to be performed by the server 20, the control unit 100 in S605 transmits the acquired image information to the display frame together with the analysis request to the server 20 via the communication unit 107. Thereafter, in S606, the control unit 100 waits until the analysis result related to the request is returned, and shifts the processing to S607 when the analysis result is received. Note that the image information transmitted along with the analysis request does not have to indicate the entire imaging range, and for example, the action related to a specific real object is preferentially caused to the virtual character according to the user's request. An instruction input of a desired small area related to an object may be received, and image information of the small area may be transmitted. In this way, the amount of information related to the analysis request can be reduced, and the amount of analysis processing can also be reduced, so that the processing speed can be increased.

  In S607, the configuration module 111 associates the received analysis results (information of the real object area and characteristic information of the real object) with the corresponding feature points of the three-dimensional space information. Although the present embodiment will be described as integrating three-dimensional space information and an analysis result, the implementation of the present invention is not limited to this, as long as mutual information is associated with each other. .

  In step S608, the character control module 113 determines the state of the virtual character corresponding to the display frame, and performs various processes for controlling the virtual character to the state and arranging the virtual character in the world. At this time, the character control module 113 may control the behavior of the virtual character in accordance with the characteristic of the real object existing around the position in the real space, which corresponds to the position where the virtual character is arranged. The behavior control of the virtual character does not need to simultaneously start the behavior based on the characteristics of all the real objects included in the imaging range of the display frame, for example, and the relative relationship with the virtual character (arrangement position of virtual character And may be sequentially performed according to an order (for example, an order of close distance) according to the distance of the position of the corresponding real space and a predetermined priority. When a specific operation is performed on one real object, control may be performed so that the operation does not occur for a predetermined period, even if it is included in the imaging range, so that the operation does not occur frequently.

  In step S609, the drawing module 115 draws the world for the viewpoint determined by the viewpoint determination module 114 based on the information on the position and orientation of the terminal device 10 related to the display frame, and generates a character image. When a real object whose characteristic is grasped as a light source exists in the imaging environment, a light source based on the characteristic information on the real object is defined in the world, and the drawing module 115 performs luminance calculation due to the light source.

  In S610, the combining module 116 combines the generated character image with the captured image corresponding to the acquired image information pertaining to the display frame to generate a presentation image.

  In step S611, the control unit 100 causes the display 105 to display the presentation image, and completes the display process related to the display frame.

  As described above, in the display process of the present embodiment, the display mode of the virtual character can be controlled according to the characteristics of the real object existing in the real space by the processes of S608 and S609. The control of the virtual character according to the characteristics of the real object may be performed, for example, as follows.

(1) Control due to affordance of real object As described above, when the characteristic information related to the application of the real object is acquired, the real object existing in the imaging environment can be grasped not simply as an obstacle. The virtual character can be controlled to take an action according to the characteristic information. For example, when the real object is a chair, a cushion, or the like, and it can be understood that it is a seating application, behavior control may be performed such that the virtual character is seated on the real object. Alternatively, when the real object is a table or the like, and it can be understood that it is not a seating application but an object mounting application, the virtual character is not arranged on the real object, but the wearing article of the character can be arranged. The virtual character may perform behavior control such as walking while avoiding the real object. Also, for example, when the real object is a picture, a shaped object, or the like, and it can be understood that it is for ornamental use, the virtual character may perform behavior control so as to focus on the real object.

(2) Control due to characteristics affecting other real objects When property information affecting other real objects is acquired by the real objects performing their functions, it is due to the real objects existing in the imaging environment The depiction of the virtual character can be controlled so that a change occurs. For example, in the case where the real object is a luminaire and it is possible to grasp the characteristics of causing a change in brightness or a change in shade of the surrounding real object by functioning, a virtual character within a predetermined range affected by the real object. When is arranged, the luminance of the virtual character or the state of the shadow to be generated may be controlled based on the position of the virtual character and the information on the position of the real object and the light source characteristic. Also, for example, when the real object is an air conditioner and it is possible to grasp the characteristics that it causes movement (swing) of the real object such as a curtain by functioning, a predetermined range affected by the real object In the case where a virtual character is placed on the object, the behavior of hair, clothes, etc. may be controlled. Of course, whether to affect other real objects in this way depends on the operating conditions of the lighting fixtures and air conditioners, so the analysis module 201 changes the characteristic information according to the operating conditions, or The analysis result may be sent out by adding information as to whether or not to perform control according to the characteristic information, and behavior control in the terminal device 10 may be performed adaptively.

  Note that these controls are merely examples, and other behavior control may be performed depending on the position of the virtual character and the position of the corresponding real space or the characteristics of the real object existing around the position. Further, the behavior of the virtual character is not limited to the state in which the virtual character is arranged at the position corresponding to the real object, for example, when the feature points of the real object whose characteristics are grasped are included in the three-dimensional space information. Behavioral control may be performed to generate That is, if it is already known that a chair is present around the terminal device 10 when the application is executed, behavior control is performed such that the virtual character is moved in the chair direction to be seated. It may be

  Also, for example, even if the type is specified, such as a window, the property of the object itself does not affect other real objects, but the same effect as the lighting fixture can be obtained by the incidence of external light, etc. When in the state given to the real object, the characteristic grasping module 203 grasps this based on the luminance distribution of the image information, and is configured to output the characteristic information as a real object giving the illumination effect to the other real objects. May be

  In addition, not only real objects that have characteristics that cause other real objects to change depending on their operating conditions, such as lighting fixtures and air conditioners, and for real objects whose operating state etc. can be determined by analysis, depending on their state The characteristic information may be different, and the behavior of the virtual character may be changed. For example, in the case of a television receiver, based on image information of a region corresponding to a screen, it is possible to analyze the driving state, the program being watched, etc. It may be configured as follows.

  In the present embodiment, the analysis module 201 analyzes based on the image information, identifies the real object, and returns the characteristic information to the terminal device 10. However, the embodiment of the present invention is limited to this. It is not a thing. Identification of a real object is performed based on user operation performed in the terminal device 10, such as selection from a product list, input of a product name, and characteristic information related to the product can be acquired from the server 20. It may be.

  Also, the characteristic information of any real object need not necessarily be constant, and may be updated, for example, at a time zone, a date or time, or at a predetermined timing, so as to cause the virtual character to behave differently. Good.

  Further, in the character image generation system of the present embodiment, it is assumed that the functions are separated between the terminal device 10 and the server 20 for mixed reality presentation to the user, and only the characteristic analysis of real objects is executed by the server 20. However, the separation of functions is not limited to this. That is, the present invention can be realized as long as the configuration of three-dimensional space information, the characteristic analysis of a real object, and the generation of a character image are executed by one or more devices.

  As described above, according to the character image generation device of the present embodiment, it is possible to perform preferable mixed reality presentation in consideration of the characteristics of the real object existing in the real space.

[Modification 1]
In the embodiment described above, the character control module 113 performs behavior control of the virtual character in accordance with the characteristics of the real object around the position in the real space corresponding to the position where the virtual character is arranged. The implementation of the invention is not limited to this.

  Behavior control of a virtual character has characteristics associated with the information in response to, for example, predetermined information acquisition being performed via the communication unit 107 or predetermined information output being performed by application execution such as a scheduler or the like. It may be performed to perform an operation related to a real object. Here, the predetermined information is information (additional information) of a type previously determined to be presented via a virtual character, and for example, predetermined information such as weather forecast, broadcast program information, or commentary information on an exhibit What is supplied from an external device, and what can be acquired from the other applications started with the terminal device 10, such as schedule information and search result information, may be used. Further, the characteristic associated with the additional information may be grasped based on the information indicating the characteristic to be grasped about the real object attached to the additional information, the data structure, the distribution source, the character string, etc. It may be grasped by analysis. Besides, the additional information may include voice information generated by the user.

  In this modification, the character control module 113 sets the virtual character to the position corresponding to the real object when the additional information related to the characteristic of the real object which is already managed in association with the three-dimensional space information is acquired. Behavior control is performed so as to move and perform a predetermined action on additional information. For example, when characteristic information indicating a television receiver is managed as a real object, when broadcast program information is received as additional information, the character control module 113 moves the virtual character to a position in the world corresponding to the television receiver. The behavior control may be performed so as to present, with voice, "a program currently being broadcast and its information", "a guide for a start time of a program matching a user preference", etc. while pointing to a television receiver. Also, for example, when characteristic information indicating a predetermined exhibit is managed as a real object, when the comment information is acquired as the additional information, the character control module 113 stops the virtual character at the position in the world corresponding to the exhibit. Behavior control should be performed to start explaining the exhibits. Also, for example, when characteristic information indicating a clock or an entrance door is managed as a real object, the character control module 113 moves the virtual character to the corresponding position in the world when schedule information related to the leaving schedule is acquired as additional information. The behavior control may be performed so as to give a notice of schedule and a notice of prompting movement.

  By doing this, mixed reality presentation with higher added value can be performed according to the user's usage environment and the characteristics of the real objects placed, not just for the purpose of viewing.

[Modification 2]
In the embodiment and the modification described above, the character image generated by the drawing module 115 and the captured image are combined to generate a presentation image that presents mixed reality and is displayed on the display 105. Although described, the presentation of the character image is not limited to this. That is, although the aspect mentioned above gives recognition that a virtual character exists in real space via an image for presentation displayed on display 105, for example, HMP (Head-Mounted Projector) or a glass type device Alternatively, a device adopting an optical transmission method may be used instead of the display 105. In this case, only the pixels relating to the character of the character image may be made visible to the user by the optical transmission method, and the user recognizes as if the virtual character exists in the real space while directly observing the real world. be able to. In other words, it is easy to understand that the combination of the character image and the captured image is not essential in the practice of the present invention, but an element adopted according to the presentation method. In the case of using a glass-type device or the like, it is preferable that the imaging unit 103 be provided at a position corresponding to the device itself or the device for the determination of the position and the posture.

[Modification 3]
In the embodiment and the modification described above, it has been described that the characteristics of the real object are grasped based on the captured image acquired by the display frame, but the characteristics grasp of the real object existing in the space observed by the user is The processing may be performed based on a captured image obtained by an imaging device provided at a position different from the viewpoint of the user or the terminal device 10.

  Similarly, the world may be drawn for a viewpoint different from the position and orientation determined based on the acquired image information, acceleration information and angular velocity information related to the display frame, and a character image may be generated. That is, as shown in the second modification, in the aspect that does not require the combination of the captured image and the character image, the eye position of the user who is watching and the installation position of the imaging unit 103 do not necessarily match. The drawing of the world may be performed with respect to the viewpoint determined based on the information on the positional relationship and the estimated position and orientation.

Other Embodiments
The present invention is not limited to the above embodiment, and various changes and modifications can be made without departing from the spirit and scope of the present invention. The character image generation device according to the present invention can also be realized by a program that causes one or more computers to function as the device. The program can be provided / distributed by being recorded on a computer readable recording medium or through a telecommunication line.

  10: terminal device 20: server 30: network 100: control unit 101: speaker 102: microphone 103: imaging unit 105: display 106: depth sensor 107: communication unit 108: geomagnetic sensor , 109: acceleration sensor, 110: gyro sensor, 111: location description information generation module, 112: configuration module, 113: character control module, 114: viewpoint determination module, 115: drawing module, 116: synthesis module, 201: analysis module , 202: specific module, 203: characteristic grasping module

Claims (14)

A first acquisition unit for acquiring a captured image obtained by capturing an image of a real space;
Configuration means for configuring spatial information including an arrangement of real objects existing in the real space based on the captured image acquired by the first acquisition means;
A second acquisition unit that acquires the characteristic of the real object;
A drawing unit that draws a three-dimensional space constructed corresponding to the real space with respect to a viewpoint corresponding to an imaging device that has captured the captured image, and in a display mode according to the arrangement and characteristics of the real object, A character image generating apparatus having: drawing means for generating a character image in which the virtual character in the three-dimensional space is drawn ;
The real object is a device whose operating state changes,
The character image generation device further includes change means for acquiring the operating state of the real object, and changing the characteristic of the real object according to the acquired operating state,
The drawing means generates a character image in which the virtual character is drawn in a display mode according to the characteristic changed by the changing means of the real object and the arrangement of the real object. Character image generation device to be. The image capturing apparatus further includes an identifying unit that analyzes the captured image and identifies characteristics of a real object in the captured image.
The character image generating device according to claim 1, wherein the second acquisition unit acquires a characteristic of the real object as a specification result by the specification unit. The apparatus further comprises communication means for transmitting the captured image to an external device and receiving, from the external device, information indicating characteristics of a real object in the captured image based on analysis of the transmitted captured image.
The character image generating device according to claim 1, wherein the second acquisition unit acquires the characteristic of the real object based on the information received by the communication unit. Based on the arrangement and characteristics of the real object, further have a control means for controlling the virtual character of the operation of the three-dimensional space,
The character image generating apparatus according to any one of claims 1 to 3, wherein the control means controls the operation of the virtual character to be different according to the characteristic changed by the changing means. . And receiving means for receiving additional information to be presented via the virtual character,
The control means adds the virtual character to the virtual character at a position corresponding to the real object in the three-dimensional space, when the additional information related to the property of the real object existing in the real space is received. 5. The character image generation device according to claim 4, wherein control is performed to cause a predetermined action to be performed on information.   When the real object existing in the real space has a characteristic that affects other real objects, the drawing unit generates the character image in a state in which the influence is reflected on the virtual character. The character image production | generation apparatus of any one of the Claims 1 thru | or 5 characterized by the above-mentioned. When the real object existing in the real space is a light source,
The second acquisition unit acquires, as a characteristic, at least one of the brightness of the light source, the irradiation direction, and the type of the light source.
The character image generating apparatus according to claim 6, wherein the drawing unit generates the character image in a state in which a change in luminance or a change in shade based on the light source is reflected on the virtual character. Combining means for generating a presentation image obtained by combining the character image generated by the drawing means and the captured image;
The character image generation apparatus according to any one of claims 1 to 7, further comprising a display for presenting the presentation image generated by the combining unit.   The character image generation apparatus according to any one of claims 1 to 7, further comprising a presentation unit configured to present the character image generated by the drawing unit in an optical transmission method. A first acquisition unit for acquiring a captured image obtained by capturing an image of a real space;
Configuration means for configuring spatial information including an arrangement of real objects existing in the real space based on the captured image acquired by the first acquisition means;
A second acquisition unit that acquires the characteristic of the real object;
It is a drawing means for drawing a three-dimensional space constructed corresponding to the real space with respect to a view point to be presented, and a virtual character in the three-dimensional space in a display manner according to the arrangement and characteristics of the real object. A character image generating apparatus having a drawing means for generating a character image drawing the
The real object is a device whose operating state changes,
The character image generation device further includes change means for acquiring the operating state of the real object, and changing the characteristic of the real object according to the acquired operating state,
The drawing means generates a character image in which the virtual character is drawn in a display mode according to the characteristic changed by the changing means of the real object and the arrangement of the real object. Character image generation device to be. A first acquisition step of acquiring a captured image obtained by capturing an image of a real space;
A configuration step of configuring spatial information including an arrangement of real objects existing in the real space based on the captured image acquired in the first acquisition step;
A second acquisition step of acquiring characteristics of the real object;
It is a drawing step of drawing a three-dimensional space constructed corresponding to the real space with respect to a viewpoint corresponding to an imaging device that has picked up the captured image, in a display mode according to the arrangement and characteristics of the real object, And a drawing step of generating a character image in which the virtual character in the three-dimensional space is drawn .
The real object is a device whose operating state changes,
The character image generation method further includes a changing step of acquiring an operation state of the real object, and changing a characteristic of the real object according to the acquired operation state,
In the drawing step, a character image in which the virtual character is drawn is generated in a display mode according to the characteristics changed in the changing step of the real object and the arrangement of the real object. Character image generation method to be characterized.   The program for functioning a computer as each means of the character image production | generation apparatus of any one of Claims 1-10.   A computer readable recording medium having the program according to claim 12 recorded thereon. A first acquisition unit for acquiring a captured image obtained by capturing an image of a real space;
Configuration means for configuring spatial information including an arrangement of real objects existing in the real space based on the captured image acquired by the first acquisition means;
An identifying unit that analyzes the captured image and identifies characteristics of a real object in the captured image;
A drawing unit that draws a three-dimensional space constructed corresponding to the real space with respect to a viewpoint corresponding to an imaging device that has captured the captured image, and in a display mode according to the arrangement and characteristics of the real object, A character image generation system including: drawing means for generating a character image in which the virtual character in the three-dimensional space is drawn ;
The real object is a device whose operating state changes,
The character image generation system further includes changing means for acquiring the operating state of the real object and changing the characteristic of the real object according to the acquired operating state,
The drawing means generates a character image in which the virtual character is drawn in a display mode according to the characteristic changed by the changing means of the real object and the arrangement of the real object. Character image generation system to be.