JP2019160332A - Computer program, terminal, and method - Google Patents

Abstract

To arrange an object in a location specified in a real space, and display it on a display.SOLUTION: A computer program causes a terminal having imaging means and a display to implement the functions of: specifying a position of the terminal in a three-dimensional model in a real space; acquiring correspondence between the three-dimensional model and actual three-dimensional space information acquired by measurement means, on the basis of the specified terminal position; determining a position of an object in an image, on the basis of the acquired correspondence and a position of the object in the three-dimensional model; and displaying an image with the object superimposed in the determined position, on the display.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a computer program, a terminal, and a method for displaying an object.

  Non-Patent Documents 1 and 2 disclose systems in which a sensor and a camera are combined. This system can identify the current position and orientation of the terminal device based on the information of the camera and sensor.

Tango, [online], [Search January 10, 2017], Internet, <URL: https://get.google.com/tango/> The first smartphone “Lenovo PHAB2 Pro” equipped with Google ’s spatial recognition technology “Tango” will have an AR experience like this: [online], [Search January 10, 2017], Internet, <URL: http: //gigazine.net/news/20160630-snapdragon-tango-ar/>

  When the techniques described in Non-Patent Documents 1 and 2 are applied to AR, various things can be realized. However, at present, no method has been established for arranging an object at a designated location in the real space and displaying it on a display.

  The present invention has been made in view of these problems, and an object of the present invention is to provide a technique for arranging an object at a designated place in a real space and displaying the object on a display.

  One embodiment of the present invention relates to a computer program. The computer program is acquired by the three-dimensional model and the measuring unit based on the function of specifying the position of the terminal in the three-dimensional model in real space and the specified position of the terminal in the terminal including the imaging unit and the display. A function for acquiring the corresponding relationship with the actual three-dimensional spatial information, a function for determining the position of the object in the image based on the acquired correspondence and the position of the object in the three-dimensional model, and the determined position And a function of displaying an image on which an object is superimposed on a display.

  Note that any combination of the above-described constituent elements, or those obtained by replacing the constituent elements and expressions of the present invention with each other between apparatuses, methods, systems, recording media storing computer programs, and the like are also included as aspects of the present invention. It is valid.

  According to the present invention, an object can be arranged at a designated place in the real space and displayed on the display.

It is a schematic block diagram of the terminal device which concerns on embodiment. It is a block diagram which shows the function and structure of a control part of FIG. It is a flowchart which shows the flow of a series of processes in the terminal device of FIG. It is a typical screen figure of a display. It is a typical screen figure of a display. It is a typical screen figure of a display. It is a typical screen figure of a display. It is a typical screen figure of a display. It is a typical screen figure of a display. It is a typical screen figure of a display.

  Hereinafter, the same or equivalent components, members, and processes shown in the drawings are denoted by the same reference numerals, and repeated description is appropriately omitted. In addition, in the drawings, some of the members that are not important for explanation are omitted.

  FIG. 1 is a schematic configuration diagram of a terminal device 10 according to an embodiment. The terminal device 10 is a portable communication terminal such as a smartphone, for example. The speaker 101 converts the sound signal output from the control unit 100 into sound and outputs the sound. The microphone 102 converts the sound into a sound signal and outputs the sound signal to the control unit 100. The imaging unit 103 has a lens and a CCD sensor, acquires image information, and outputs it to the control unit 100. The gyro sensor 104 measures the angle and angular velocity of the terminal device 10 and outputs the measured information to the control unit 100 as posture information. In this embodiment, the display 104 is a touch-sensitive display, and functions as an input unit for user operation in addition to presenting information to the user. When the display is not a touch-sensitive display, an input unit is provided separately. The depth sensor 106 measures the distance (depth) to the object and outputs the measurement result to the control unit 100 as depth information. The communication unit 107 performs communication processing via a mobile communication network or a wireless LAN. The temperature sensor 108 detects the temperature and outputs temperature information including the detected temperature to the control unit 100. The acceleration sensor 109 measures the acceleration of the terminal device 10 and outputs it to the control unit 100 as acceleration information.

  The control unit 100 is a control unit for the terminal device 10 as a whole, and includes one or more processors and a memory unit. The memory unit includes a RAM used by the processor for temporarily storing data, a flash memory in which an operating system and applications executed by the processor, and data to be retained even when the power is turned off are stored. .

  In the present embodiment, the three-dimensional model may be a correspondence table that holds a group of feature points of an image captured by the imaging unit 103 and an imaging position in association with each other. The point cloud is configured so that it can be recognized as the same object even if the imaging location or angle changes slightly. The terminal device 10 reads the feature point information and determines where the imaging position is when the imaging position is unknown. That is, the three-dimensional model is for storing the appearance of the landscape and determining where the imaging position is, and can be generated without using the depth sensor 106. On the other hand, the depth sensor 106 is used when the object stands on the ground or when the size of an actual object is measured.

  FIG. 2 is a block diagram showing the function and configuration of the control unit 100 of FIG. Each block shown here can be realized by hardware and other elements such as a computer CPU and a mechanical device, and software can be realized by a computer program or the like. Draw functional blocks. Therefore, it is understood by those skilled in the art who have touched this specification that these functional blocks can be realized in various forms by a combination of hardware and software.

  The control unit 100 includes a terminal position specifying unit 201, a correspondence acquisition unit 202, an object position determination unit 203, a display control unit 204, a notification unit 205, a 3D model holding unit 211, and a 3D model generation unit. 212. The terminal device 10 may download object information associated with a position (coordinates) based on the position of the terminal device 10 specified from a later-described three-dimensional model from the server. There are various downloading methods. The user operates the terminal device 10 to select a download icon, and also periodically uploads a captured image or an image of a feature point. The presence or absence may be determined, and the corresponding three-dimensional model may be downloaded. When the GPS information acquired by the terminal device 10 is uploaded to the server, a three-dimensional model corresponding to the location (along with the display of information specifying the corresponding application) may be selected and received. Alternatively, the terminal device 10 holds in advance object information associated with a 3D model stored and generated in advance by the terminal device 10 and a position (coordinates) based on the position of the terminal specified from the 3D model. Also good.

  The three-dimensional model generation unit 212 obtains feature points in the image corresponding to the image information based on the image information obtained by the imaging unit 103 imaging the real space. The three-dimensional model generation unit 212 stores a plurality of feature points and the positions and orientations of the terminals when the feature points are imaged, and generates a three-dimensional model that can identify the position of the terminal from the feature points. The 3D model generation unit 212 stores the generated 3D model in the 3D model holding unit 211. The position of the feature point is represented by coordinates (x, y, z) in a virtual three-dimensional space that imitates real space. Note that the origin of the position in the three-dimensional space is, for example, the position of the terminal device 10 when the three-dimensional model generation unit 212 starts generating the three-dimensional model. Alternatively, the 3D model generation unit 212 may acquire 3D model data from the outside and store the data in the 3D model holding unit 211.

  When a 3D model already exists in the 3D model holding unit 211, the 3D model generation unit 212 may compare the newly acquired 3D model with the already stored 3D model. For example, if the comparison result indicates that the feature points equal to or greater than a predetermined ratio match, the newly acquired 3D model is acquired at the same location as the already stored 3D model. Recognize In this case, the 3D model generation unit 212 updates the already saved 3D model with the newly acquired 3D model.

  The terminal position specifying unit 201 specifies the current position of the terminal device 10 in the real space three-dimensional model. The current position is specified by coordinates (x, y, z) in the three-dimensional space. The terminal position specifying unit 201 specifies the current position and inclination of the terminal device 10 by motion tracking using posture information output from the gyro sensor 104 and acceleration information output from the acceleration sensor 109. For example, the terminal position specifying unit 201 first performs matching based on feature points, and specifies where the terminal device 10 is in the three-dimensional model. After the identification, the terminal position identification unit 201 can identify the coordinates on the three-dimensional model even if the terminal device 10 is moved by the acceleration sensor 109 and the gyro sensor 104. The terminal position specifying unit 201 may correct the accumulation of shifts in motion tracking by area learning based on image information obtained by the imaging unit 103.

  The correspondence relationship acquisition unit 202 is included in the 3D model held in the 3D model holding unit 211 and the image acquired by the imaging unit 103 based on the current position of the terminal device 10 specified by the terminal position specifying unit 201. The correspondence relationship with the object information associated with the position (coordinates) to be obtained and the correspondence relationship with the real space information acquired by the depth sensor 106 are acquired. The real space information includes information such as the location of the ground and the location of the wall, and the correspondence between the position of the ground and the three-dimensional model and the correspondence between the three-dimensional model and the position of the wall can be obtained.

  The object position determination unit 203 determines the coordinates where the virtual object is to be arranged based on the coordinates of the object information specified by the correspondence acquisition unit 202 and the correspondence between the real space information acquired by the depth sensor 106. To do. In addition, instead of displaying the object as it is in the coordinates of the object acquired with the correspondence information, considering the position with the real space (floor or wall) acquired as the correspondence information, it can stand on the floor, It becomes possible to display so as not to overlap. The object may be, for example, an animal body such as a human being or an animal, or a so-called character that models them. The object has object attributes such as gender and age. If the size (height and width) of the object is registered in the terminal device 10, the object placement location can be corrected according to the real space so that it does not overlap with the real space. In addition, obstacles can be avoided if they are moving.

  The display control unit 204 causes the display 105 to display a superimposed image in which the object is superimposed at the position determined by the object position determination unit 203. The display control unit 204 acquires an image from the imaging unit 103. The display control unit 204 generates a superimposed image by superimposing the display of the object on the position determined by the object position determination unit 203 in the acquired image. The display control unit 204 causes the display 105 to display the composite image.

  The notification unit 205 notifies the user of the terminal device 10 of information via the object displayed on the display 105. The notification unit 205 includes an application cooperation unit 206, a network cooperation unit 207, a text generation unit 208, a voice generation unit 209, and an object change unit 210.

  The application cooperation unit 206 acquires information to be notified to the user from the application program installed in the terminal device 10. The network cooperation unit 207 acquires information received by the communication unit 107 from a predetermined server via the network. The text generation unit 208 converts the information acquired by the application cooperation unit 206 and the network cooperation unit 207 into text. The notification unit 205 causes the display 105 to display the text message generated by the text generation unit 208 in association with the object. The voice generation unit 209 converts the information acquired by the application cooperation unit 206 and the network cooperation unit 207 into voice. The notification unit 205 converts the voice message generated by the voice generation unit 209 into voiced voice that matches the attribute of the object, and causes the speaker 101 to output the voice message. The application cooperation unit 206 may determine notification based on the notification condition. This notification condition includes, for example, date and calendar information.

The text generation unit 208 may change or edit the text message so as to correspond to at least one of the object attribute and the user attribute. The voice generation unit 209 may change or edit the voice message so as to correspond to at least one of the attribute of the object and the attribute of the user.
The object changing unit 210 changes the display mode or operation of the object according to information notified to the user.

The operation of the terminal device 10 having the above configuration will be described.
FIG. 3 is a flowchart showing a flow of a series of processes in the terminal device 10. The terminal device 10 specifies the current position of the terminal device 10 in the three-dimensional model (S301). The terminal device 10 acquires a correspondence relationship between the three-dimensional model and actual three-dimensional spatial information acquired by measurement means such as the imaging unit 103 and the depth sensor 106 (S302). The terminal device 10 determines the position of the object in the three-dimensional model (S303). The terminal device 10 generates a superimposed image by superimposing the object image (S305). The terminal device 10 generates notification effects such as a text message displayed in association with the object and a voice message output in accordance with the movement of the mouth of the object (S306). The terminal device 10 displays the superimposed image on the display 105 (S307). The terminal device 10 outputs a notification effect via the object (S308).
In step S302, the terminal device 10 acquires a three-dimensional model and displays a captured image on the display 105 (for example, by selecting an icon indicating that the television is displayed while displaying the television). The location of the three-dimensional model and the item type may be registered. Thus, depending on the application, the movement of the object can be changed after recognizing the location of the article.

4 to 10 are representative screen views of the display 105, respectively. FIG. 4 shows a screen that displays an image in the room captured by the image capturing unit 103. There is no superimposed display of objects. Real objects such as the TV 401, the intercom 402, and the window 403 are shown in the screen. In this state, the terminal device 10 moves the virtual humanoid object 501 from the outside of the screen of the display 105 to the inside. For example, an animation in which the object 501 appears by slowly moving from the lower lower side to the upper side in FIG. 4 is displayed, and the state shown in FIG. 5 is obtained. This is realized by the following processing flow.
(1) The coordinates (xa, ya, za) and the coordinates (xb, yb, zb) before movement of the object 501 in the three-dimensional model are determined.
(2) Determine a movement path connecting the coordinates before movement (xa, ya, za) and the coordinates after movement (xb, yb, zb), and divide the movement path into N equal parts (N is a natural number of 2 or more). N + 1 coordinates to be superimposed (xn, yn, zn) are determined. n = 0, 1,..., N, and (x0, y0, z0) = (xa, ya, za), (xN, yN, zN) = (xb, yb, zb).
(3) A superimposed image when the object 501 is arranged at the coordinates (xa, ya, za) before movement is generated and displayed on the display 105.
(4) A superimposed image is generated when the object 501 is arranged at the superimposition target coordinates (x1, y1, z1) next to the pre-movement coordinates (xa, ya, za) and displayed on the display 105.
(5) Increment n and perform the same processing as (4). If n = N as a result of the increment, the process proceeds to the next.
(6) A superimposed image when the object 501 is arranged at the coordinates (xb, yb, zb) after movement is generated and displayed on the display 105.
Thereby, the position of the object 501 in the three-dimensional model can be changed, and the object 501 can be continuously moved so as to correspond on the screen displayed on the display 105. For example, when the position of the object 501 is changed by 1 m, the object 501 is not instantaneously moved from the position before the movement to the position after the movement, but is moved to 1 m ahead at the walking speed of the object 501.
If the coordinates of the terminal device 10 become a specific coordinate or enter a specific distance, and if the orientation (height / direction) of the display 105 matches a specific condition, an animation is displayed. Also good. Alternatively, the animation may be displayed only during a specific time period and a specific number of times (if the time is displayed once per day, the second time is not displayed).

  5 and 6 show examples of screens in which the TV program guide application is installed in the terminal device 10. Referring to FIG. 5, the application cooperation unit 206 acquires information on a program that broadcasts the game of the team A live from the TV program guide application. When the current time approaches the start time of the program, the text generation unit 208 generates a text message 502 for the program. The notification unit 205 displays the generated text message 502 on the screen as a balloon of the object 501.

Further, when the notification unit 205 detects that the television 401 is turned off, the notification unit 205 generates a text message 503 inquiring whether the television 401 is turned on, and displays the text message 503 on the screen as a balloon. If the notification unit 205 determines that the volume level obtained from the microphone 102 is lower than the predetermined volume level, the notification unit 205 may determine that the television 401 is off. The user inputs an instruction to turn on the television 401 by operating the terminal device 10 or speaking into the microphone 102. Referring to FIG. 6, the object changing unit 210 generates an animation that brings the position of the object 501 closer to the television 401, and performs an additional display 601 as if the object 501 switches on the television 401 (in this case, the object 501 Arm). The application cooperation unit 206 turns on the TV 401 via a TV program guide application installed in the terminal device 10 or an application for remotely operating the TV 401 (for example, a remote controller application of the TV 401). Many televisions basically employ infrared communication. Therefore, if the terminal device 10 implements infrared communication, the television 401 can be easily operated. Alternatively, the set-top box and the like also support other communication methods, and can be operated with a smartphone.
Note that the television 401 may acquire a moving image related to the program to be notified from the server, and superimpose it on the TV screen as an object to guide the program to be notified.

  7 and 8 show examples of screens in which the interphone remote control application is installed in the terminal device 10. Here, the intercom location is set in the three-dimensional image. Referring to FIG. 7, the application cooperation unit 206 receives a notification indicating that the intercom 402 is ringing from the intercom remote operation application. The text generation unit 208 generates a text message 701 indicating that the intercom 402 is ringing. The notification unit 205 displays the generated text message 701 on the screen as a balloon of the object 501.

  Referring to FIG. 8, the object changing unit 210 generates an animation that brings the position of the object 501 closer to the intercom 402 and also displays an additional display 702 as if the object 501 confirms the monitor of the intercom 402 (in this case, The face of the object 501 is directed to the intercom 402). The application cooperation unit 206 acquires the name of the visitor determined by analyzing the monitor image from the intercom remote operation application. The text generation unit 208 generates a text message 704 including the acquired name. The notification unit 205 displays the generated text message 704 on the screen as a balloon of the object 501. The application cooperation unit 206 acquires an interphone monitor image from the interphone remote operation application, and superimposes and displays the acquired image as a small screen 703 on the lower right of the screen.

  9 and 10 show examples of screens in which a schedule management application is installed in the terminal device 10. Referring to FIG. 9, the application cooperation unit 206 acquires event information related to the user from the schedule management application. When the current time approaches the start time of the event, the text generation unit 208 generates a text message 901 for the event. The notification unit 205 displays the generated text message 901 on the screen as a balloon of the object 501.

  Referring to FIG. 10, the object changing unit 210 generates an animation that brings the position of the object 501 closer to the window 403 and an additional display 902 as if the object 501 confirms the outside of the window 403 (in this case, The face of the object 501 is directed toward the window 403). The network cooperation unit 207 acquires a weather forecast at the start time of the event from the weather information server via the network. The text generation unit 208 generates a text message 903 corresponding to the acquired weather forecast. The notification unit 205 displays the generated text message 903 on the screen as a balloon of the object 501. In addition, the object changing unit 210 generates an additional display 904 (in this case, an umbrella silhouette) according to the acquired weather forecast.

  Alternatively, when the weather forecast indicates that the outside air temperature = 0 degrees, a text message “It is cold, 0 degrees” may be generated instead of the text message 903, and an animation in which the object 501 shakes may be generated. .

In another example, an air conditioner operation instruction may be received from the user while displaying a text message that is different from the temperature information obtained from the temperature sensor 108 as an object balloon in cooperation with the air conditioner management application.
In the above example, in addition to or instead of displaying the text message, the corresponding content voice may be output from the speaker 101.

  In the embodiment described above, examples of the holding unit are a hard disk and a semiconductor memory. Further, based on the description of the present specification, each unit is configured by a CPU (not shown), a module of an installed application program, a module of a system program, a semiconductor memory that temporarily stores the content of data read from the hard disk, or the like. It will be appreciated by those skilled in the art who have touched this specification that this can be achieved.

  According to terminal device 10 according to the present embodiment, the user can feel that information has been presented from an object superimposed on the captured image. As a result, the user's familiarity and attachment to the terminal device 10 increase, and the quality of the user experience improves.

  For example, when the object 501 is a character, it is possible to create a feeling that the character is close to the user in a world synchronized with the reality. Since the object 501 is “invisible” only through the terminal device 10, it cannot be seen by anyone other than the user of the terminal device 10. Thereby, the special feeling which a user feels increases.

  The configuration and operation of the terminal device 10 according to the embodiment have been described above. This embodiment is an exemplification, and it will be understood by those skilled in the art that various modifications can be made to each component and combination of processes, and such modifications are within the scope of the present invention.

  In the embodiment, specific examples of object display have been described with reference to FIGS. 5 to 10, but the present invention is not limited to this. For example, the object information may have a display condition based on a predetermined distance, and it may be determined whether to display the object based on a distance condition regarding the distance between the position of the terminal device 10 and the position coordinates of the object. In this case, it is possible to realize a display method of displaying from a nearby object.

  Alternatively, if the object information has a predetermined stay time condition and the position of the terminal device 10 and the position of the object stay within a predetermined range for a predetermined stay time or longer, the object may be displayed. Good. In this case, if it passes only near the object, it is not displayed and notification can be given when the object is close to the target object to some extent.

Alternatively, the object information may have predetermined priority information, and the object with the highest priority may be displayed based on the priority information. In this case, when there are a plurality of objects, they are not displayed at a time, but can be displayed from the one with the highest priority.
In addition, about the display conditions of said object, a display condition can be made editable by a user by selecting the displayed object.

  In the embodiment, GPS information may be stored in association with the three-dimensional model. Accordingly, for example, when one of the above notification messages is received and stored near the place, the user may be notified to the user by the display 105 of the terminal device 10 or vibration. Further, if the user is in the vicinity of the place where the 3D model is stored by GPS information and receives / stores any of the above notification messages, the application or camera is turned on by the display 105 of the terminal device 10 or by vibration. You may be encouraged to do so.

  In the embodiment, the animation may change the color to suit the surrounding environment according to the luminance of the captured image of the imaging unit 103. The expression and appearance (clothing) may be changed according to the season and time zone (morning, daytime, night). In addition, when the voice output is stopped as in the manner mode, the text mode may be set. When the voice output mode is set, the text may be stopped and the voice mode may be set.

  10 terminal device, 100 control unit, 103 imaging unit, 105 display.

Claims (11)

In a terminal equipped with an imaging means and a display,
A function of specifying the position of the terminal in a three-dimensional model of real space;
Based on the identified position of the terminal, a function of acquiring a correspondence relationship between the three-dimensional model and actual three-dimensional spatial information acquired by the measurement unit;
A function of determining the position of the object in the image based on the acquired correspondence and the position of the object in the three-dimensional model;
A computer program for realizing a function of displaying the image in which the object is superimposed on the determined position on the display.   The computer program according to claim 1, further causing the terminal to realize a function of notifying information to a user of the terminal via the object displayed on the display.   The computer program according to claim 2, wherein the information includes information to be notified to the user from an application program installed in the terminal.   The computer program according to claim 2 or 3, wherein the information includes information received from a server by the terminal via a network.   The notifying function is a function of displaying a text message corresponding to at least one of the attribute of the object and the attribute of the user as the information in association with the object or displaying the attribute of the object and the information as the information. The computer program according to any one of claims 2 to 4, including a function of outputting a sound corresponding to at least one of user attributes or both.   The computer program according to claim 2, further causing the terminal to realize a function of changing a display mode or an operation of the object according to the information notified to the user in the function of notifying.   7. The terminal according to claim 1, further causing the terminal to realize a function of moving the object in the image displayed on the display in accordance with a change in the position of the object in the three-dimensional model. Computer program.   The computer program according to claim 7, wherein the function of moving includes a function of moving the object from outside the display range of the display.   The computer program according to claim 1, further causing the terminal to realize a function of generating the three-dimensional model from an image acquired by the imaging unit. A terminal comprising an imaging means and a display,
Means for identifying the location of the terminal in a three-dimensional model of real space;
Means for acquiring a correspondence relationship between the three-dimensional model and actual three-dimensional spatial information acquired by the measuring means based on the identified position of the terminal;
Means for determining the position of the object in the image based on the acquired correspondence and the position of the object in the three-dimensional model;
Means for causing the display to display the image in which the object is superimposed at the determined position. A method executed by a terminal including an imaging unit and a display,
Identifying the location of the terminal in a three-dimensional model of real space;
Obtaining a correspondence relationship between the three-dimensional model and the actual three-dimensional spatial information acquired by the measurement unit based on the identified position of the terminal;
Determining the position of the object in the image based on the acquired correspondence and the position of the object in the three-dimensional model;
Displaying the image with the object superimposed on the determined position on the display.

Images