JP5244012B2 - Terminal device, augmented reality system, and terminal screen display method - Google Patents

Description

  The present invention relates to a terminal device, an augmented reality system, and a terminal screen display method.

  Conventionally, the mobile communication technology used in the mobile phone business has been changed from the first generation of the analog system to the second generation of the PDC system and the GSM system that have realized data communication by digital technology in addition to voice calls. It has been developed into a third generation such as W-CDMA which realizes higher-speed data communication and various communication services using multimedia. In addition, in mobile communication terminals using such communication technology, as a result of technological innovations in the fields of processors such as CPUs, memories, various peripheral devices, and mounting technologies, both multi-functionality and high functionality have been achieved. Has made great progress.

  As described above, with the advancement of technologies related to mobile communication and communication terminals, various communication services such as video calls, e-mails, web browsing, and chats are realized in mobile phone terminals in addition to voice calls. ing. Multifunctional communication terminals include liquid crystal / organic EL display modules, camera modules, non-contact IC card modules, infrared communication modules, Bluetooth (registered trademark) communication modules, WiFi (registered trademark) communication modules, and GPS modules. Various devices are installed.

  Furthermore, multi-functionalization of communication terminals has also advanced in terms of installed software, and various application software for controlling various modules, and various viewers, multimedia players, browsers, mailers, games, PIM, and other applications Software is installed on the operating system of the terminal, and it is possible to realize an extremely advanced function by enabling a plurality of applications to be executed concurrently by the multitask function.

  On the other hand, in the field of information processing, virtual reality (virtual reality) that forms a virtual reality space by computer graphics, or augmented reality (AR;) that processes a video of the real space with a computer and adds further information. Research on Augmented Reality) is underway, and recently, augmented reality technology (AR technology) has attracted attention. There are various modes for realizing the augmented reality using the AR technology, and one of them is a method using a camera-equipped mobile phone terminal.

  For example, US MOBILIZY, Inc. has a travel guide function that overlays and displays tourist information on landscape images captured by a camera for Android (registered trademark), one of the open platforms for mobile phones. Application products with This application causes the information providing server to transmit the position information of the communication terminal (for example, position information obtained by GPS positioning or user input) to the communication terminal, and the communication terminal uses the information registered in advance in the information providing server. Sightseeing information of the surrounding area is received by the communication terminal from this information providing server. And this application implement | achieves augmented reality by superimposing and displaying the tourist information received by the communication terminal with respect to the landscape image of real space.

  Further, for example, in Patent Document 1 below, an optical image is formed with the camera facing in a landscape direction, and the position determined by the measured position and orientation is measured by measuring the position of the camera and the orientation of the optical axis. A technique for displaying a combined electronic image and an optical image obtained by a camera in combination is disclosed.

Japanese Patent No. 3700021

  In the augmented reality technique in which the additional information is superimposed on the optical image obtained by the camera, it is desired to superimpose the additional information at an appropriate position in the optical image. For this reason, when the camera moves, for example, the orientation and angle of the camera are accurately monitored by an azimuth sensor, and the superimposed position of the additional information is appropriately re-determined according to the detected amount of displacement of the orientation and angle. Drawing is done.

  However, the system load imposed by the process of re-determining and redrawing the additional information appropriately is not negligible, and trying to superimpose additional information while continuously acquiring real-space video with a moving camera. As a result, processing may not be performed smoothly due to system load, and continuous screen drawing may not be possible. Such a system load is a problem that can be particularly noticeable in a mobile phone terminal that has limitations in terms of built-in memory and processing capability. In addition, even if the processing itself is performed smoothly, there is a problem that if the additional information to be superimposed is excessive, the display screen flickers and becomes difficult to see.

  Therefore, the present invention has been made to solve the above-described problems, and superimposing additional information on the continuously displayed video while continuously acquiring and displaying the real space video with the camera. A terminal device, an augmented reality system, and a terminal capable of reducing the system load when redrawing the additional information while superimposing the additional information on an appropriate position even when the camera moves in the processing An object is to provide a screen display method.

  In order to solve the above problems, a terminal device according to the present invention includes a camera unit that continuously acquires real-space video, an orientation sensor that detects the orientation and angle of the camera unit, a display unit that displays a screen, A storage unit that stores augmented reality information associated with position information, and a control unit that controls the operation of each unit and implements an information processing function. The control unit expands based on the attitude and angle of the camera unit. Determine the display position of the reality information, superimpose it on the real space image and display it on the display unit, detect the displacement amount by which the attitude and angle of the camera unit are displaced after determining the display position, When the amount of displacement exceeds a threshold value, the terminal device performs processing for re-determining and redrawing the display position of the augmented reality information, and the control unit is configured such that the distance from the terminal device is equal to or greater than the first predetermined distance. Augmented reality information Therefore, when the displacement amount exceeds the first predetermined amount, the display position is re-determined and redrawn, and the augmented reality information whose distance from the terminal device is less than the first predetermined distance is When the amount exceeds a second predetermined amount larger than the first predetermined amount, the display position is redetermined and redrawing is performed.

  In this configuration, for the augmented reality information whose distance from the terminal device is equal to or greater than the first predetermined distance, the display position is re-determined and redrawn when the displacement exceeds the first predetermined amount. For augmented reality information whose distance from the terminal device is less than the first predetermined distance, the display position is re-determined and redrawn when the displacement exceeds a second predetermined amount that is greater than the first predetermined amount. For this reason, for augmented reality information whose distance from the terminal device is equal to or greater than the first predetermined distance, a process of re-determining and redrawing the display position when the displacement amount exceeds the first predetermined amount is performed. Here, for augmented reality information in which the distance from the terminal device is less than the first predetermined distance, the display position is re-determined and redrawn when the displacement exceeds a second predetermined amount greater than the first predetermined amount. Therefore, it is possible to reduce the system load as compared with the case where the display position is re-determined and redrawn regardless of the comparison with the second predetermined amount simply by exceeding the first predetermined amount. As a result, the camera moves in the process of superimposing augmented reality information as additional information on the continuously displayed video while continuously acquiring and displaying the video in the real space with the camera. However, it is possible to reduce the system load when redrawing the additional information while superimposing the additional information on an appropriate position.

  In the above configuration, the control unit divides the augmented reality information into a group whose distance from the terminal device is less than the first predetermined distance and a group whose distance is the first predetermined distance or more, and performs positioning and drawing for each group. It is also preferable to carry out. Thereby, since positioning and drawing are performed for each group, it is possible to further reduce the system load during redrawing, compared to the case of performing processing of positioning and drawing individual augmented reality information.

  In addition, for the augmented reality information in which the distance from the terminal device is greater than or equal to the second predetermined distance, the displacement amount exceeds a third predetermined amount that is less than the first predetermined amount. It is also preferable to re-determine and redraw the display position. As a result, for augmented reality information whose distance from the terminal device is greater than or equal to the second predetermined distance that is greater than the first predetermined distance, the display position is reset when the displacement exceeds a third predetermined amount that is less than the first predetermined amount. Determine and redraw. As a result, when the distance from the terminal device becomes longer, the influence of the displacement amount on the display position of the augmented reality information becomes larger. In comparison, it is possible to superimpose additional information at a more appropriate position.

  In addition, the control unit includes the augmented reality information, a group whose distance from the terminal device is less than a first predetermined distance, a group that is greater than or equal to a first predetermined distance and less than a second predetermined distance, It is also preferable to perform processing for positioning and drawing for each group. Thereby, since positioning and drawing are performed for each group, it is possible to further reduce the system load during redrawing, compared to the case of performing processing of positioning and drawing individual augmented reality information.

  Further, the process of re-determining the display position of the augmented reality information by the control unit is performed by shifting the current display position in accordance with the displacement of the camera unit detected by the orientation sensor and the angle. Is also preferable. As a result, when redrawing augmented reality information, it is only necessary to shift the display position rather than deleting augmented reality information, determining the redisplay position, and redrawing. It is possible to further reduce the system load.

  An augmented reality information system according to the present invention includes a terminal device that displays a screen by a display unit, and a server that is configured to be communicable with the terminal device and has augmented reality information associated with position information. The augmented reality information whose distance from the terminal device is greater than or equal to the first predetermined distance, the server re-determines and redraws the display position when the displacement exceeds the first predetermined amount. For augmented reality information whose distance from the terminal device is less than the first predetermined distance, the display position is redetermined when the displacement exceeds a second predetermined amount that is greater than the first predetermined amount. And a server control unit for performing the redrawing process.

  In this configuration, for the augmented reality information whose distance from the terminal device is equal to or greater than the first predetermined distance, the display position is re-determined and redrawn when the displacement exceeds the first predetermined amount. For augmented reality information whose distance from the terminal device is less than the first predetermined distance, the display position is re-determined and redrawn when the displacement exceeds a second predetermined amount that is greater than the first predetermined amount. For this reason, for augmented reality information whose distance from the terminal device is equal to or greater than the first predetermined distance, a process of re-determining and redrawing the display position when the displacement amount exceeds the first predetermined amount is performed. Here, for augmented reality information in which the distance from the terminal device is less than the first predetermined distance, the display position is re-determined and redrawn when the displacement exceeds a second predetermined amount greater than the first predetermined amount. Therefore, it is possible to reduce the system load as compared with the case where the display position is re-determined and redrawn regardless of the comparison with the second predetermined amount simply by exceeding the first predetermined amount. As a result, the camera moves in the process of superimposing augmented reality information as additional information on the continuously displayed video while continuously acquiring and displaying the video in the real space with the camera. However, it is possible to reduce the system load when redrawing the additional information while superimposing the additional information on an appropriate position.

  A terminal screen display method according to the present invention comprises an augmented reality comprising: a terminal device that displays a screen by a display unit; and a server configured to be communicable with the terminal device and having augmented reality information associated with position information. A terminal screen display method performed in an information system, wherein the server sets a display position when the displacement exceeds a first predetermined amount for augmented reality information whose distance from the terminal device is a first predetermined distance or more. For the augmented reality information in which the distance from the terminal device is less than the first predetermined distance and the displacement amount exceeds the second predetermined amount that is greater than the first predetermined amount, the re-determined and redrawing process is performed. It has a step of performing a process of re-determining the display position and redrawing.

  In this configuration, for the augmented reality information whose distance from the terminal device is equal to or greater than the first predetermined distance, the display position is re-determined and redrawn when the displacement exceeds the first predetermined amount. For augmented reality information whose distance from the terminal device is less than the first predetermined distance, the display position is re-determined and redrawn when the displacement exceeds a second predetermined amount that is greater than the first predetermined amount. For this reason, for augmented reality information whose distance from the terminal device is equal to or greater than the first predetermined distance, a process of re-determining and redrawing the display position when the displacement amount exceeds the first predetermined amount is performed. Here, for augmented reality information in which the distance from the terminal device is less than the first predetermined distance, the display position is re-determined and redrawn when the displacement exceeds a second predetermined amount greater than the first predetermined amount. Therefore, it is possible to reduce the system load as compared with the case where the display position is re-determined and redrawn regardless of the comparison with the second predetermined amount simply by exceeding the first predetermined amount. As a result, the camera moves in the process of superimposing augmented reality information as additional information on the continuously displayed video while continuously acquiring and displaying the video in the real space with the camera. However, it is possible to reduce the system load when redrawing the additional information while superimposing the additional information on an appropriate position.

  According to the present invention, even when the camera moves in the process of superimposing additional information on the continuously displayed image while continuously acquiring and displaying the image of the real space with the camera, It is possible to provide a terminal device, an augmented reality system, and a terminal screen display method capable of reducing the system load when redrawing the additional information while superimposing the additional information on an appropriate position.

It is the block diagram which showed schematically the structure of the augmented reality system containing a terminal device and an augmented reality server. It is the functional block diagram which showed the main function structures of the terminal device. It is an example of a list of position information contents among augmented reality information stored in the storage unit. It is an example of a screen display. It is an example of a screen display. It is a model figure which shows the aspect which a position shift produces in a far point by the angle shift of a camera part, and the chart which shows the result of having calculated the position shift produced in a far point according to an angle shift. It is a table | surface of the investigation result regarding the distance from a terminal device, the allowable width | variety of icon arrangement | positioning, and the shift | offset | difference angle which satisfy | fills an allowable width | variety. It is the functional block diagram which showed the main function structures of the augmented reality server. It is the flowchart which showed operation | movement of the terminal device. It is the physical block diagram which showed the main physical structures of the terminal device. It is the physical block diagram which showed the main physical structures of the augmented reality server.

  The best mode for carrying out the present invention will be described below with reference to the accompanying drawings. In the description of the drawings, the same elements are denoted by the same reference numerals, and redundant description is omitted.

(1) Configuration of Augmented Reality Information System Including Terminal Device First, the configuration of an augmented reality information system including a terminal device according to an embodiment of the present invention will be described with reference to FIG. FIG. 1 is a configuration diagram schematically showing a configuration of an augmented reality system 1 including a terminal device 100 according to an embodiment of the present invention and an augmented reality server 300 that provides augmented reality information to the terminal device 100. It is. Note that the augmented reality information is additional information for explanation by being superimposed on an optical image obtained by a camera.

  The terminal device 100 is a mobile terminal such as a mobile phone terminal that performs communication by connecting to the communication network 150 via the base station 140 or the like, and a camera unit and an image display having an image acquisition function as described later A display unit or the like having a function is provided.

  The augmented reality server 300 is a server computer system that is communicatively connected to the network 200 that is communicatively connected to the communication network 150. The augmented reality server 300 is configured to be able to communicate with the map information server 351, the route information server 361, the location information content server 371, and the terminal device 100, which are external servers similarly connected to the network 200.

  The augmented reality system 1 having such a configuration superimposes and displays additional information provided by the augmented reality server 300 on an image acquired by the user of the terminal device 100 with the camera unit directed to the real space, A service that provides augmented reality to this user is provided. Note that augmented reality is a sense that allows virtual space to be captured more realistically by adding additional information to an optical image obtained by a camera. Hereinafter, the terminal device 100 and the augmented reality server 300 will be described in detail.

(2) Configuration of Terminal Device Next, the configuration of the terminal device 100 will be described using FIG. 2 and FIG. FIG. 2 is a functional block diagram illustrating a main functional configuration of the terminal device 100 according to the embodiment of the present invention, and FIG. 10 is a physical block diagram illustrating a main physical configuration of the terminal device 100. As illustrated in FIG. 10, the terminal device 100 includes hardware such as a CPU 11, a RAM 12, a ROM 13, an operation unit 14, a wireless communication unit 15, a display 16, and an antenna 17. The functions described below of the terminal device 100 are exhibited by the operation of these components.

  As illustrated in FIG. 2, the terminal device 100 includes a position information acquisition unit 101 (acquisition unit) that acquires position information of the terminal device 100, a display unit 102 (display unit) that displays a screen, and user input. An input unit 103 (input unit) that accepts, an orientation sensor 104 (detection unit) that detects the orientation in which the terminal is facing, a communication unit 105 (communication unit) that communicates by connecting to the communication network 150, and a real space A camera unit 106 (imaging unit) that continuously acquires video, a storage unit 107 (storage unit) that stores predetermined programs and data, and a control unit that controls the operation of each unit and realizes a predetermined information processing function 108 (control means).

  The position information acquisition unit 101 can use, for example, a GPS module that receives radio waves from a plurality of GPS (Global Positioning System) satellites and acquires current position information from their intensities. In addition to this GPS system, by communicating with the communication network 150 via the communication unit 105, information on satellites existing near the terminal is acquired as assist data, and AGPS (Assisted Global Positioning System) that performs position measurement is performed. It is possible to use a module corresponding to the method, and it is preferable to use this because position information can be acquired more quickly and accurately.

  However, the position information acquisition unit 101 is not limited to the one using the above GPS or AGPS, and may be configured to acquire the position information without depending on the GPS. For example, a method may be used in which a plurality of wireless communication base stations such as WiFi and Bluetooth (registered trademark) are arranged in the space, and position information is acquired based on the radio field intensity therefrom. It is also possible to use a method used in combination with AGPS.

  The display unit 102 is a liquid crystal display device or an organic liquid crystal display device, and displays the operating state of the terminal device 100, a user interface for operating the terminal device 100, and the like under the control of the control unit 108. The terminal device 100 provides the user with augmented reality by superimposing and displaying the additional information acquired from the augmented reality server 300 on the video of the real space acquired by the camera unit 106 according to the operation described below.

  The input unit 103 is an input device such as a keypad provided in the casing of the terminal device 100 and other buttons. A key event that is a signal corresponding to each key or button is operated by the user, and the control unit 108 Is used for various operations or control.

  The direction sensor 104 is a module that detects the direction in which the user is facing and the posture and angle of the camera unit 106 and outputs a signal corresponding to the detected direction to the control unit 108. The control unit 108 can recognize the posture / angle at which the user holds the terminal device 100 from the detection value of the direction sensor 104. A combination of a plurality of geomagnetic sensors can be used as such a direction sensor 104, and a motion sensor combining a plurality of acceleration sensors in addition to the geomagnetic sensor may be used. A 6-axis sensor combining a triaxial geomagnetic sensor and a triaxial acceleration sensor, a 5-axis sensor combining a triaxial geomagnetic sensor and a biaxial acceleration sensor, or instead of an acceleration sensor Any device using a gyro can be used for the direction sensor 104 as a motion sensor. When the motion sensor is used as described above, it is possible to detect not only information on the attitude and angle of the terminal device 100 but also the direction and speed of the operation in which the user moves or tilts the terminal device 100.

  The communication unit 105 is a module that communicates with the communication network 150 to communicate with other terminal devices and server systems. Examples of the communication here include circuit-switched communication performed by communication connection with other terminal apparatuses, packet-switched communication performed by communication connection with a server system connected to the network 200 via the communication network 150, and the like. Can be mentioned. When acquiring the position information of the terminal device 100 by the AGPS method, the communication unit 105 cooperates with the position information acquisition unit 101 to assist the acquisition of the position information.

  The camera unit 106 is a module that has a predetermined optical system and an image receiving element and provides a function of acquiring a digital image. When the terminal device 100 captures a digital image, the activated camera unit 106 normally repeats the process of acquiring data by the image receiving element at a predetermined exposure time from the subject image acquired by the optical system at a predetermined time interval. The live view function is realized by displaying the data on the display unit 102. Then, when the user further performs a predetermined operation with the input unit 103 in this state, the camera unit 106 generates image data under shooting conditions set from the subject image acquired by the optical system, and the generated image data is It is stored and saved in the storage unit 107. The augmented reality system 1 superimposes and displays information that provides augmented reality on the real space video displayed on the display unit 102 by the above-described live view function.

  The storage unit 107 is a memory for storing predetermined information under the control of the control unit 108 and providing the stored information to the control unit 108. Further, the storage unit 107 stores various programs executed by the control unit 108, and the control unit 108 reads and executes them as appropriate. The storage unit 107 does not need to have a uniform configuration, and can be configured by appropriately combining various memories such as ROM, RAM, Flash ROM, and a secondary storage device such as an HDD. It doesn't matter. The unique terminal ID assigned to the terminal device 100 is also stored in the storage unit 107 together with other data.

  The storage unit 107 stores various augmented reality information provided from the augmented reality server 300 described later. FIG. 3 shows, as an example, a list of position information contents in the augmented reality information stored in the storage unit 107. The location information content is associated with the location information, is acquired from the location information content server 371 by the augmented reality server 300 in response to a request from the terminal device 100, and is returned to the terminal device 100 as a response. The list of the storage unit 107 includes, for each position information content, the content ID, the content type, the content name, and the position information of the terminal device 100. Further, the distance from the current terminal position of the terminal device 100 is within the viewing angle. No item is provided. Among these items, information obtained from the augmented reality server 300 is recorded as it is for items other than the distance and the viewing angle, and the control unit 108 of the terminal device 100 calculates and determines the items of the distance and the viewing angle. Results are recorded.

  The control unit 108 is a functional block that is virtually configured by executing a program stored in the storage unit 107 on a CPU (not shown). The control unit 108 includes a position information acquisition unit 101, a display unit 102, and the like of the terminal device 100. Various functions of the terminal device 100 are realized by exchanging data and control signals with the functional blocks such as the input unit 103, the orientation sensor 104, the communication unit 105, the camera unit 106, and the storage unit 107.

  Further, the control unit 108 transmits the request for the location information of the terminal device 100 acquired by the location information acquisition unit 101 to request augmented reality information according to the location information of the terminal device 100 from the augmented reality server 300. In response to the response, the augmented reality is provided to the user. As a characteristic function in this case, the display position of the augmented reality information is determined based on the attitude and angle of the camera unit 106, and is displayed on the display unit so as to be superimposed on the real space image.

  Further, after determining the display position, the control unit 108 detects the displacement amount in which the posture and angle of the camera unit are displaced by the azimuth sensor 104, and the display position of the augmented reality information when the displacement amount exceeds the threshold value. It is possible to re-determine and redraw. Hereinafter, these processes will be described in detail.

(3) Details of Processing by Control Unit Subsequently, processing for determining the display position of augmented reality information based on the attitude and angle of the camera unit 106 and superimposing the information on the real space image and displaying it on the display unit 102 will be described. . In this process, first, the orientation / direction in which the camera unit 106 is facing is derived from the detection value of the azimuth sensor 104, and the storage unit 107 stores it based on the horizontal viewing angle and the vertical viewing angle of the camera unit 106. It is determined whether or not the augmented reality information is within the field of view of the camera unit 106, and information regarding the determination result is recorded in the augmented reality information list (see FIG. 3).

  Subsequently, it is assumed that the augmented reality information determined to enter the field of view is arranged in the coordinate space with the current position of the terminal device 100 as the origin, and the horizontal field of view of the camera with the predetermined height of the origin as the center point. Based on the angle, the viewing angle in the vertical direction, and the image size, the augmented reality information is transmitted and projected onto a rectangular plane determined at a predetermined distance from the origin, and the display position is determined based on the position of the transparent projection. Here, it is preferable to set the height of the center point to be transparently projected to a height of about 1.5 m in order to reproduce the situation in which the user uses the terminal device 100 while standing. When the terminal device 100 can detect altitude information at the current position by the position information acquisition unit 101 or the like, the center point may be set at a height corresponding to the detected altitude.

  After the above process, a process is performed in which each augmented reality information is drawn at the determined display position so as to be superimposed on the real space image. Although there is no particular limitation on such processing, each augmented reality information is drawn at a predetermined position and a transparent screen layer is created, and this screen layer is superimposed on the real space video and displayed on the display unit 102. Thus, augmented reality information is displayed at an appropriate position in the real space image. In this case, the augmented reality information may be displayed in the form of an icon, an information balloon, a sub window, or the like. Here, the augmented reality information is assumed to be an icon (or information balloon).

  Although the display of augmented reality information as an icon avoids complication of the screen, the amount of information displayed on the screen is reduced. For example, icons having different appearances according to the type of augmented reality information are displayed. You may make it use properly. FIGS. 4A, 4B, and 4C show examples of a screen that displays augmented reality information with a circular icon. 4A shows an image of the real space acquired by the camera unit 106, and FIG. 4B shows an image in which icons are drawn by obtaining the display position of each augmented reality information by the method described above. FIG. 4C shows a screen in which these are superimposed. FIGS. 5A, 5B, and 5C similarly show the case where the augmented reality information is displayed with an information balloon.

(4) Details of Processing When Displacement Exceeds Threshold Value Subsequently, after the display position is determined, the displacement amount in which the posture and angle of the camera unit 106 are displaced is detected by the direction sensor 104, and the displacement amount exceeds the threshold value. A process for re-determining the display position of the augmented reality information and redrawing in the case of the above will be described. As described above, when providing augmented reality to the user using the process of superimposing and displaying augmented reality information on the real space video, the camera unit 106 continuously acquires the real space video. A live view function displayed on the display unit 102 is also used. Therefore, it is necessary to appropriately control the position where the augmented reality information is superimposed in accordance with the displacement of the posture and angle of the camera unit 106 that changes every moment. For this reason, the orientation sensor 104 detects the amount of change in posture and angle in the camera unit 106, and performs redrawing processing when the threshold value is exceeded.

  However, it is not preferable in terms of system load and user experience to excessively repeat the process of repositioning and redrawing all the augmented reality information to be superimposed. FIG. 6A is a model diagram showing a mode in which a positional shift d occurs at the far point due to the angle shift α of the camera unit 106, and FIG. 6B shows a positional shift generated at the far point in accordance with the angular shift α. It is a graph which shows the result of having calculated d. As shown in FIGS. 6A and 6B, even when the angle deviation α is the same, the position deviation d decreases as the distance from the current location of the terminal device 100 to the object decreases.

  For example, when an angle shift α of 10 ° occurs, the position shift d at the point of 500 m from the current location of the terminal device 100 reaches about 88 m, but the position shift d at the point of 10 m from the current location of the terminal device 100 is about 1.7 m. Only. Therefore, when such an angle shift α occurs in a state where information is superimposed on a 10 m square object, if the distance from the current location of the terminal device 100 to the object is 500 m, the position where the information is superimposed However, if the distance from the current location of the terminal device 100 to the object is 10 m, the necessity is not recognized. That is, it is not necessary to frequently rewrite the augmented reality information about the object at a short distance even if the posture angle of the camera unit 106 changes.

  Therefore, in the terminal device 100, for augmented reality information in which the distance from the terminal device 100 is equal to or greater than the predetermined value L1 (first predetermined distance), the above displacement amount exceeds θ1 (first predetermined amount). While performing the process of re-determining the display position and redrawing, for augmented reality information whose distance from the terminal device 100 is less than the predetermined value L1, θ2 (second predetermined amount) in which the displacement amount is larger than θ1 is set. When it exceeds, the display position is determined again and redrawing is performed. That is, the threshold value θ2 for determining whether or not to redraw the augmented reality information associated with a relatively short distance point is redrawn, and the augmented reality information associated with the relatively far point is redrawn. It is set larger than the threshold value θ1 for determining whether or not.

  By doing so, the frequency of redrawing augmented reality information associated with a short distance point is reduced, the system load can be reduced, and the frequent redrawing prevents the screen from becoming difficult to see. be able to. Further, for augmented reality information in which the distance from the terminal device 100 is greater than or equal to the predetermined value L2 (second predetermined distance) greater than the predetermined value L1, the displacement amount exceeds θ3 (third predetermined amount) smaller than θ1. In this case, the display position may be re-determined and redrawing may be performed. Further, the threshold value may be set more finely and individually set within the scope of the gist of the present invention.

  In order to realize such a function, the control unit 108 divides the augmented reality information into a short-distance group whose distance from the terminal device 100 is less than L1 and a long-distance group whose distance is L1 or more, and performs positioning for each group. In addition, a drawing process may be performed. In this case, a display screen in which a layer in which a short distance group is drawn and a layer in which a long distance group is drawn can be superimposed on a real space video. Further, the control unit 108 classifies the augmented reality information into a short distance group whose distance from the terminal device 100 is less than L1, a medium distance group that is greater than or equal to L1 and less than L2, and a long distance group that is greater than or equal to L2. It is also possible to subdivide it further.

(5) Example of Specific Setting of Threshold Value Next, an example of specific setting of the threshold value will be described with reference to FIG. FIG. 7 shows the allowance of icon arrangement when there is an object 10 m wide at each point of 50 m, 100 m, and 400 m from the terminal device 100 and the control unit 108 superimposes an icon on the image of the object photographed by the camera unit 106. It is a chart of the result of having investigated the camera part 106 and the display part 102 of a specific standard about the deviation | shift angle (theta) which satisfy | fills the width | variety A and the tolerance | permissible-width A. The model camera unit 106 has a horizontal field of view angle of 52 ° and a vertical field of view angle of 36 °, and the display unit 102 is an HVGA 3-inch liquid crystal. In this case, the object size on the display unit 102 is calculated as described in FIG. 7 by the transmission projection method.

  Here, when the augmented reality information is divided into a short distance group, a medium distance group, and a long distance group, and further, as an effect on display, the icon of information associated with a long distance object is reduced, the horizontal The width A allowed for the icon display position in the direction is 21.12 mm, 12.56 mm, and 5.64 mm, respectively. Here, when the respective positions of 50 m, 100 m, and 400 m are photographed, the misalignment angle θ that causes such a misalignment on the display unit 102 is obtained, and is found to be 18.45 °, 10.97 °, and 4.929. °. Assuming that an icon is superimposed on the center of the object in the initial stage, the allowable deviation angle θ is ½ of the above.

  In consideration of this result, in the terminal device 100 having the above-described conditions, for example, augmented reality information is divided into a short distance group with a distance to the terminal of less than 50 m, a middle distance group with a distance of 50 m to 400 m, and a long distance group with a distance of 400 m or more. It is possible to set the threshold value of the camera angular displacement for re-drawing the short-distance group and 10 °, the threshold value of the medium-distance group 6 °, and the threshold value of the long-distance group 2.5 °. However, the drawing position is calculated according to the angle of view and resolution of the lens and imaging device serving as the imaging unit of the terminal apparatus 100 and the size and resolution of the screen serving as the display unit.

(6) Configuration of Augmented Reality Server Next, the configuration of the augmented reality server 300 will be described with reference to FIGS. 8 and 11. FIG. 8 is a functional block diagram showing the main functional configuration of the augmented reality server 300 according to the embodiment of the present invention, and FIG. 11 is a physical block diagram showing the main physical configuration of the augmented reality server 300. It is. As shown in FIG. 11, the augmented reality server 300 is configured by hardware such as a CPU 31, a RAM 32, a ROM 33, a communication module 34, and an auxiliary storage device 35 such as a hard disk. By operating these components, the later-described functions of the augmented reality server 300 are exhibited.

  The augmented reality server 300 is configured to be communicable with the terminal device 100, is a resident server system having augmented reality information, and is a server on which a web server resides. The augmented reality server 300 includes a communication unit 301 (device communication unit), a control unit 310 (device control unit), and a user DB 316 (device storage unit).

  More specifically, the augmented reality server 300 receives a request from the terminal device 100 via the communication network 150 and the network 200, and if necessary, the map information server 351, the route information server 361, or the route information server 361 connected to the network 200. Data is acquired from the position information content server 371 and a response process is performed by transmitting augmented reality information to the terminal device 100.

  The communication unit 301 is a network module that communicates with an external server such as the terminal device 100, the map information server 351, the route information server 361, and the location information content server 371 by connecting to a network. it can. The user DB 316 is a database machine in the system area of the augmented reality server 300, and can identify a service user and a use terminal such as the terminal ID, position information, and handle name of the terminal device 100 received from the terminal device 100. Record and manage information in association.

  The control unit 310 is a functional block that is virtually configured by executing a predetermined program by a CPU (not shown). More specifically, the control unit 310 includes a map data acquisition unit 311 that acquires map data from the map information server 351, a route data acquisition unit 312 that acquires route data from the route information server 361, and position information from the position information content server 371. A location information content acquisition unit 313 that acquires content and a user data unit 315 that accesses the user DB 316 are configured.

  These map data acquisition unit 311, route data acquisition unit 312, location information content acquisition unit 313, and user data unit 315 are all augmented reality servers that are corresponding CGI programs based on the request transmitted from the terminal device 100. A predetermined process is performed by calling an API. Hereinafter, the map data acquisition unit 311, the route data acquisition unit 312, and the location information content acquisition unit 313 will be described in detail.

  The map data acquisition unit 311 is an API that is called when the terminal device 100 requests map data. The map data about the predetermined range centered on the position information transmitted from the terminal device 100 is disclosed using the public API. Requests to the map information server 351. In response to this, the map information server 351 extracts corresponding map data from the map information DB 352 and returns it to the augmented reality server 300, and the augmented reality server 300 can display the acquired map data on the terminal device 100. To the terminal device 100. At this time, when the terminal device 100 requests map data, if the range and scale of the map data are specified as parameters, a process of calling these APIs specified by the map information server 351 is performed. .

  The route data acquisition unit 312 is an API that is called when the terminal device 100 requests route data, and the location information that the terminal device 100 transmits together with the location information acquired by the location information acquisition unit 101 as a starting point. Is requested to the route information server 361. In response to this, the route information server 361 generates corresponding route data from the route information DB 362 and sends it back to the augmented reality server 300, and the augmented reality server 300 is data that the terminal device 100 can handle the acquired route data. A reply is sent to the terminal device 100 as a format. At this time, the route information server 361 may calculate the total required time and the total fee, add them to the route data, and send a reply.

  The location information content acquisition unit 313 is an API that is called when the terminal device 100 requests location information content. Here, the location information content refers to the location information of stations, airports, ports, bus stops, interchanges, service areas, gas stations, restaurants, taverns, convenience stores, department stores, hospitals, schools, signals, corners, intersections, etc. It refers to information on the associated facility. For example, for restaurants, location information contents including addresses, telephone numbers, fax numbers, e-mail addresses, homepage URLs, introductions, business hours information, store images, etc. are published by services provided by information service providers. .

  The location information content acquisition unit 313 has released location information content existing within a predetermined distance centered on the location information of the terminal device 100 acquired by the location information acquisition unit of the terminal device 100 by an information service provider or the like. A request signal for requesting the location information content server 371 is transmitted. In response to this, the location information content server 371 returns a list in which the location information content satisfying the conditions from the location information content DB 372 is extracted and transmitted to the augmented reality server 300, and the augmented reality server 300 transmits the acquired location information content to the terminal device. The data is edited into a data format that can be used at 100 and sent to the terminal device 100 as a reply.

  At this time, the location information content acquisition unit 313 can transmit a request signal for requesting location information to the plurality of location information content servers 371, and in this case, the augmented reality server 300 has a plurality of location information content servers. The list from 371 is integrated into a common format and sent to the terminal device 100 as a reply. If a location information content server 371 used for each field of location information content is registered, a request signal for requesting location information content can be transmitted only for the field requested by the terminal device 100. .

  In addition, the terminal ID and position information of another terminal device are recorded in the user DB 316 together with the handle name of the user, but such contents are also contents associated with the position information and are handled equivalently to other position information contents. be able to. That is, the position information content acquisition unit 313 transmits a request signal requesting the user's handle name recorded in association with the position information within a predetermined distance from the current position of the terminal device 100 to the user data unit 315, If the user data unit 315 returns a handle name list extracted from the user DB 316 in response to the augmented reality server 300 and transmits the reply from the augmented reality server 300 to the terminal device 100 in a common format, another position can be obtained. It can be used as content together with information content.

(7) Operation of Terminal Device in Augmented Reality System Next, the operation of the terminal device 100 in the augmented reality system 1 will be described with reference to FIG. FIG. 9 is a flowchart showing the operation of the terminal device 100. Here, a threshold value θ3 for determining whether or not to redraw a long distance group of augmented reality information, a threshold value θ1 for determining whether or not to redraw a medium distance group, and a short distance group are set. The threshold value θ2 for determining whether or not to redraw satisfies the relationship θ3 <θ1 <θ2.

  This operation is started by the user operating the input unit 103 of the terminal device 100 to start the augmented reality program. First, the camera unit 106 and various sensors (particularly, the position information acquisition unit 101 and the orientation sensor 104) are operated. The live view display is started on the initial screen (step ST101). Live view display is a process in which video data is generated by an image receiving element with a predetermined exposure time from a subject image acquired by the optical system of the activated camera unit 106 and displayed on the display unit 102 according to a conventional method. It is performed by repeating at a predetermined time interval. Subsequently, the position information related to the current position of the terminal device 100 is acquired from the detection value of the position information acquisition unit 101 (step ST102).

  Next, the terminal device 100 transmits the position information acquired in step ST102 to the augmented reality server 300, transmits a request signal for requesting augmented reality information (step ST103), and the augmented reality server 300 responds. The list of augmented reality information around the terminal device 100 is received and recorded in the storage unit 107 (step ST104).

  Next, for the augmented reality information of the list received in step ST104, the terminal device 100 is based on the orientation and angle information of the camera unit 106 detected by the orientation sensor 104 and the position information acquired in step ST102. Then, it is individually determined whether the point associated with the information described in the list falls within the field of view of the camera unit 106, and the result is recorded in the list of augmented reality information stored in the storage unit 107 (step ST105). ).

  Next, the terminal device 100 calculates the distance between the point associated with each piece of information and the terminal device 100 and records it in the list stored in the storage unit 107 (step ST106), and according to the distance from the terminal device 100 The information is classified into a short distance group, a medium distance group, and a long distance group and recorded in a list (step ST107).

  Next, assuming that the terminal device 100 has placed the extracted augmented reality information in a coordinate space with the current position of the terminal device 100 as the origin, the horizontal direction of the camera with the predetermined height of the origin as the center point Based on the viewing angle, the vertical viewing angle, and the video size, each augmented reality information is translucently projected onto a rectangular plane determined at a predetermined distance from the origin, and the augmented reality information is transmitted and projected on the rectangular plane. The display position is determined (step ST108). Then, the terminal device 100 draws the augmented reality information at the drawing position of the display unit 102, and displays it superimposed on the live view screen (step ST109).

  With the above operation, an operation of superimposing and displaying the augmented reality information on the video in the real space is performed, and a service for providing augmented reality to the user can be realized. In the subsequent operation, while the user scans the real space while tilting or moving the terminal device 100, a process for superimposing the augmented reality information on an appropriate position is performed.

  Here, first, the timer t of the terminal device 100 starts counting (step ST110), and the presence / absence of an end instruction is confirmed (step ST111). If it is determined here that an end instruction has been issued, the series of operations ends. On the other hand, if it is not determined that there has been an end instruction, it is determined whether or not the timer t has reached a predetermined value n (step ST112). Here, when it is determined that the timer t has not reached the predetermined value n, the operation after step ST111 is repeated (step ST112).

  On the other hand, when it is determined in step ST112 that the timer t has reached the predetermined value n, the orientation sensor 104 detects the attitude and angle of the camera unit 106 (step ST113). Subsequently, with respect to the posture and angle of the camera unit 106 detected in step ST113, the displacement amount from the posture and angle when the augmented reality information of the long distance group was previously drawn is calculated, and the displacement amount is θ3 or more. Is determined (step ST114). If it is determined that the displacement is greater than θ3, the display position of the long-distance group augmented reality information is re-determined by the same processing as in step ST108, and redrawing is performed (step ST115). At this time, the values of the posture and angle of the camera unit 106 used to determine the display position of the long-distance group last time are stored and stored in the storage unit 107. It is updated to the values of the posture and angle of the camera unit 106 used for redetermination. If it is not determined in step ST114 that the displacement is greater than θ3, the process returns to step ST110, the timer is reset and restarted, and the above processing is repeated.

  In step ST114, it is determined that the displacement is greater than θ3, and after redrawing the long distance group in step ST115, the displacement amount from the posture and angle when the augmented reality information of the middle distance group was drawn last time is calculated. Then, it is determined whether the displacement amount is equal to or greater than θ1 (step ST116). If it is determined that the displacement is greater than θ1, the display position of the medium-distance group augmented reality information is re-determined by the same processing as in step ST108, and redrawing is performed (step ST117). At this time, the values of the posture and angle of the camera unit 106 used to determine the display position of the previous intermediate distance group are stored and stored in the storage unit 107, and the values are stored in the current step ST117. It is updated to the values of the posture and angle of the camera unit 106 used for redetermination. If it is not determined in step ST116 that the displacement is greater than θ1, the process returns to step ST110, the timer is reset and restarted, and the above processing is repeated.

  On the other hand, it is determined in step ST116 that the displacement is equal to or more than θ1, and after redrawing the medium distance group in step ST117, the displacement amount from the posture and angle when the augmented reality information of the short distance group was previously drawn. Is calculated, and it is determined whether the displacement amount is equal to or larger than θ2 (step ST118). If it is determined that the displacement is greater than θ2, the display position of the short-distance group augmented reality information is re-determined by the same processing as in step ST108, and redrawing is performed (step ST119). At this time, the values of the posture and angle of the camera unit 106 used to determine the display position of the previous middle distance group are stored and stored in the storage unit 107, and the values are displayed at the current step ST118. The values are updated to the values of the attitude and angle of the camera unit 106 used for redetermination, and the process returns to step ST110 to reset and restart the timer, and repeat the above processing.

  Even if it is not determined in step ST116 that the displacement is greater than θ1, the process returns to step ST110 and the above processing is repeated. In the flowchart of FIG. 9, the processing after step ST113 is sequentially performed. However, step ST114, step ST116, and step ST118 are performed in parallel according to the attitude and angle of the camera unit 106 detected at step ST113. Only a group that has been subjected to execution processing and requires re-determination and re-drawing of the display position may be re-determined and re-drawn.

(8) Operation and effect of the present invention Through the processing as described above, the terminal device 100 according to the embodiment of the present invention continuously acquires and displays the real-space video by the camera unit 106, and continuously. In the augmented reality technology that superimposes information on the displayed video, augmented reality information as additional information is grouped according to the distance from the terminal device 100, and the posture and angle of the camera unit 106 is equal to or greater than the threshold for each group. In this case, the display position is redetermined and redrawn.

  Then, for a group at a short distance from the terminal device 100, the threshold value is set larger than that for the group at a long distance, thereby suppressing unnecessary display position redetermination and redrawing processing. Therefore, it is possible to reduce the system load of redrawing according to the posture / angle displacement while always superimposing augmented reality information at an appropriate position, and it is difficult to see the screen of the display unit 102 due to excessive redrawing. It is possible to prevent this.

  In steps ST115, ST117, and ST119 of the above operation, the display position of the augmented reality information is determined by the same process as in step ST106. However, the method for determining the display position repeats the above-described transmission projection process. It is not limited to things. For example, the position of each currently displayed augmented reality information can be determined by uniformly shifting the position of the camera unit 106 by a predetermined amount in accordance with the posture and angle change amount and direction of the camera unit 106. In this way, it is possible to further reduce the system load in step ST115, step ST117, and step ST119.

(9) Modified Examples The augmented reality system 1 according to the embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made. For example, in the above-described embodiment, the configuration is shown in which the augmented reality information is divided into three groups and the redrawing control is performed. However, the augmented reality information is divided into two groups, a short-distance group and a long-distance group. Of course, redrawing control may be performed. Further, the augmented reality information may be divided into four or more groups as long as they do not depart from the gist of the present invention.

  In the above-described embodiment, the terminal device 100 connected to the communication network 150 has been described. However, if the augmented reality information is appropriately stored in the storage unit 107, the terminal device 100 is connected to the communication network 150. Communication connection is not an essential configuration requirement. For example, the terminal device may be a navigation terminal mounted on a vehicle such as an automobile.

  According to the present invention, even when the camera moves in the process of superimposing additional information on the continuously displayed image while continuously acquiring and displaying the image of the real space with the camera, It is possible to provide a terminal device, an augmented reality system, and a terminal screen display method capable of reducing the system load when redrawing the additional information while superimposing the additional information on an appropriate position.

DESCRIPTION OF SYMBOLS 1 ... Augmented reality system 11, 31 ... CPU, 12, 32 ... RAM, 13, 33 ... ROM, 14 ... Operation part, 15 ... Wireless communication part, 16 ... Display, 17 ... Antenna, 34 ... Communication module, 35 ... Auxiliary storage device 100 ... Terminal device 101 ... Position information acquisition unit 102 ... Display unit 103 ... Input unit 104 ... Direction sensor 105 ... Communication unit 106 ... Camera unit 107 ... Storage unit 108 ... Control , 140 ... base station, 150 ... communication network, 200 ... network, 300 ... augmented reality server, 301 ... communication unit, 310 ... control unit, 311 ... map data acquisition unit, 312 ... route data acquisition unit, 313 ... position Information content acquisition unit, 315 ... user data unit, 316 ... user DB, 351 ... map information server, 361 ... route information server, 371 ... location information container Tsusaba, d ... positional displacement, the map information DB ... 352, route information DB ... 362, position information content DB ... 372, alpha ... angular deviation, theta ... deviation angle.

Claims (7)

A camera unit that continuously acquires real-space video, an orientation sensor that detects the posture and angle of the camera unit, a display unit that displays a screen, and a memory that stores augmented reality information associated with position information A control unit that controls an operation of each unit and realizes an information processing function, and the control unit determines a display position of the augmented reality information based on an attitude and an angle of the camera unit, The amount of displacement of the posture and angle of the camera unit after the display position is determined is detected by the azimuth sensor after being superimposed on the real space image and displayed on the display unit. A terminal device for performing a process of re-determining and redrawing the display position of the augmented reality information when exceeding,
The controller re-determines and redraws the display position when the displacement exceeds a first predetermined amount for the augmented reality information whose distance from the terminal device is a first predetermined distance or more. As well as processing
For the augmented reality information whose distance from the terminal device is less than the first predetermined distance, the display position is re-determined when the displacement amount exceeds a second predetermined amount that is greater than the first predetermined amount. A terminal device that performs redrawing processing. The control unit divides the augmented reality information into a group having a distance from the terminal device less than the first predetermined distance and a group having the first predetermined distance or more, and performs a process of positioning and drawing for each group. Do,
The terminal device according to claim 1. The control unit, for the augmented reality information whose distance from the terminal device is a second predetermined distance greater than the first predetermined distance, the displacement amount exceeds a third predetermined amount smaller than the first predetermined amount. The display position is re-determined and redrawn when
The terminal device according to claim 1, wherein: The control unit includes the group having a distance from the terminal device that is less than the first predetermined distance, the group that is not less than the first predetermined distance and less than the second predetermined distance, and the second predetermined distance. Divide into the above groups, and perform the process of positioning and drawing for each group.
The terminal device according to claim 3. The process of re-determining the display position of the augmented reality information by the control unit is performed by shifting the current display position in accordance with the displacement amount of the attitude and angle of the camera unit detected by the direction sensor. Do,
The terminal device according to any one of claims 1 to 4, wherein: A terminal device that continuously acquires real-space video by a camera unit and displays a screen by a display unit; a server configured to be able to communicate with the terminal device and having augmented reality information associated with position information; An augmented reality information system comprising:
The server
For the augmented reality information whose distance from the terminal device is a first predetermined distance or more, the display position of the augmented reality information is determined based on the orientation and angle of the camera unit, and then the orientation of the camera unit and When the amount of displacement in which the angle is displaced exceeds a first predetermined amount, the display position is re-determined and redrawn, and
For the augmented reality information whose distance from the terminal device is less than the first predetermined distance, the display position is re-determined when the displacement amount exceeds a second predetermined amount that is greater than the first predetermined amount. Do the redraw process,
An augmented reality information system comprising a server control unit. A terminal device that continuously acquires real-space video by a camera unit and displays a screen by a display unit; a server configured to be able to communicate with the terminal device and having augmented reality information associated with position information; A terminal screen display method performed in an augmented reality information system comprising:
For the augmented reality information whose distance from the terminal device is a first predetermined distance or more, the server determines the display position of the augmented reality information based on the attitude and angle of the camera unit, and then the camera. Performing a process of re-determining and redrawing the display position when the displacement amount of the position and angle of the part exceeds a first predetermined amount;
For the augmented reality information whose distance from the terminal device is less than the first predetermined distance, the display position is re-determined when the displacement amount exceeds a second predetermined amount that is greater than the first predetermined amount. Do the redraw process,
A terminal screen display method comprising steps.