JP2016081225A - Information presenting system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information presenting system that can recognize the object of image pickup with limited calculation resources at a terminal and give high quality presentation of information without sacrificing real-time performance.SOLUTION: At a terminal 10, a picked-up image is acquired by an image pickup unit 1; an object of image pickup in the picked-up image is recognized by a recognizing unit 2 to figure out its positional attitude relative to the image pickup unit 1; the result is transmitted as object information to a server 20; information to be presented is generated as a control unit 4 superposes drawn information, drawn by the server 20, over the picked-up image; and causes the generated information to be presented by a presenting unit 3. In the server 20, an image drawing unit 6 generates drawn information drawing related information as viewed in the positional attitude in the object information by reading in from a memory unit 5 the related information prescribing expanded reality display according to the object of image pickup in the object information, and transmits the drawn information to the terminal 10.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an information presentation system that presents information, and more particularly to an information presentation system that can render high-quality information and control presentation information in a presentation unit.

  In the augmented reality in which the relative position and orientation between the imaging target and the imaging unit are estimated and related information is presented, if the information can be presented in real time and with high quality, the convenience for the user can be improved.

  Examples of the prior art for realizing the above are disclosed in Patent Documents 1 and 2, and the following methods are disclosed here.

  Non-Patent Document 1 discloses a technique of rendering using a plurality of environment maps approximated by a spherical harmonic function and a bidirectional reflectance distribution function when superimposing information in augmented reality.

  Patent Document 1 discloses a technique for transmitting video from a terminal to a server and transmitting a result of executing recognition processing and drawing processing by the server to the terminal as video.

JP 2014-44655 A

Photorealistic rendering for Augmented Reality: A global illumination and BRDF solution

  In Non-Patent Document 1, photorealistic expression is possible by simulating an environmental light source, but there is a problem that it cannot be executed on a terminal having a high calculation cost and limited calculation resources.

  On the other hand, in Patent Document 1, since the video is transmitted from the terminal to the server, a part of the problem of Non-Patent Document 1 can be solved by using abundant calculation resources of the server. However, since bidirectional video transmission is necessary, there is a problem that it cannot be realized if the communication band is narrow. In particular, when there are a plurality of terminals, there is a problem that the amount of transmission becomes enormous and cannot be realized. Further, since the transmission time is proportional to the transmission amount, there is a problem that the real-time property is lacking. Further, when video is transmitted and encoded with a lossy compression method for the purpose of data reduction, there is a problem that high-quality information cannot be obtained due to image quality degradation.

  An object of the present invention is to provide an information presentation system capable of recognizing an imaging target and presenting presentation information with high quality without lacking real-time characteristics.

  To achieve the above object, the present invention provides an information presentation system including a terminal that presents presentation information and a server that communicates with the terminal, and the terminal includes an imaging unit that captures an image and acquires a captured image. A recognition unit that recognizes a predetermined imaging target from the captured image, calculates a position and orientation with respect to the imaging unit, and uses the recognized type of the imaging target and the calculated position and orientation as target information; A terminal-side transmitter that transmits the target information to the server, a terminal-side receiver that receives the drawing information drawn by the server from the server, and the presentation information by superimposing the received drawing information on the captured image A control unit for generating and a presentation unit for presenting the generated presentation information, wherein the server is related information prepared in advance for each imaging target and is used for a predetermined augmented reality display. A storage unit that stores the related information configured; a server-side reception unit that receives target information transmitted from the terminal-side transmission unit; and a storage unit that stores related information corresponding to an imaging target in the received target information. A drawing unit that generates drawing information by drawing the related information read out according to the position and orientation in the received target information, and a server-side transmission unit that transmits the drawn drawing information to the terminal It is characterized by including these.

  ADVANTAGE OF THE INVENTION According to this invention, the information relevant to an imaging target can be shown only by holding up an imaging part with respect to an imaging target in a terminal.

  At this time, generally the calculation load is high, and when high quality is ensured, the calculation load is further increased. Since the server is responsible for drawing processing according to the position and orientation of the imaging target, a high calculation load is imposed on the terminal. Therefore, it is possible to present high-quality information that cannot be calculated with the calculation resources of the terminal.

  Furthermore, since the information to be transmitted is limited to the target information and the presentation information during communication between the server and the terminal, it can be realized even when the data amount is small and the communication band is narrow, and the transmission time is shortened and the real time operation is performed. realizable.

It is a figure which shows the structure outline | summary of the information presentation system which concerns on one Embodiment as a functional block diagram. It is a sequence diagram of operation | movement of the information presentation system which concerns on one Embodiment. It is a figure which shows the typical example of the presentation information in each time of the loop process of a terminal, buffering data, a captured image, and drawing information in a table format. It is a figure which shows related information, drawing information, and presentation information by a typical example. FIG. 5 is a diagram schematically showing drawing information with reduced resolution by an example corresponding to the example of FIG. 4.

  FIG. 1 is a functional block diagram showing an outline of the configuration of an information presentation system according to an embodiment. The information presentation system 30 includes a terminal 10 and a server 20. Here, there may be a plurality of terminals 10, and the server 20 can perform processing for information presentation in parallel with each other. Hereinafter, the terminal 10 will be described as an arbitrary one representing these terminals that can exist. The terminal 10 and the server 20 communicate with each other via a network such as the Internet.

  The terminal 10 includes an imaging unit 1, a recognition unit 2, a presentation unit 3, a control unit 4, a terminal side transmission unit 11, and a terminal side reception unit 12. The server 20 includes a storage unit 5, a drawing unit 6, a server side receiving unit 21, and a server side transmitting unit 22.

  FIG. 2 is a sequence diagram of the operation of the information presentation system 30 according to an embodiment. Hereinafter, the outline of each part in FIG. 1 will be described while explaining each step in FIG. 2.

  First, the overall outline of the sequence of FIG. 2 is as follows. As shown on the left side of FIG. 2, when the sequence is roughly divided, it can be divided into four step groups S10, S20, S30, and S40 executed in this order. The entire L1 of the step groups S10 to S40 is a unit that is repeated as a loop process. Hereinafter, each of the step groups S10, S20, S30, and S40 in which the entire L1 is repeated as a loop process will be described.

  The step group S10 includes steps S11, S12, and S13 performed in this order, and the imaging by the imaging unit 1 is mainly performed. In step S11, the control unit 4 instructs the imaging unit 1 to perform imaging. In step S12, the imaging unit 1 performs imaging according to the instruction to obtain a captured image. In step S13, the imaging unit 1 outputs the obtained captured image to the control unit 4.

  The step group S20 includes steps S21, S22, and S23 performed in this order, and recognition is mainly performed by the recognition unit 2. In step S21, the control unit 4 outputs the captured image obtained in step S13 to the recognition unit 2. In step S22, the recognition unit 2 recognizes the type of the imaging target captured in the output captured image, and calculates the relative position and orientation of the imaging target with respect to the imaging unit 1. In step S23, the recognition unit 2 outputs the target information to the control unit 4 using the recognized and calculated type and position and orientation of the imaging target and the time information included in the captured image as target information.

  Note that the time information included in the captured image is information associated with the captured image by adding the imaging time as a time stamp when the imaging unit 1 captures an image in step S12.

  The step group S30 includes steps S31, S32, S33, and S34 performed in this order, and drawing is mainly performed in the drawing unit 6. In step S31, the control unit 4 of the terminal 10 transmits the target information obtained in step S23 to the server 20 via the terminal-side transmission unit 11 and the server-side reception unit 21, and the position and orientation of the imaging target among the target information Information and time information are output to the drawing unit 6, and information on the type of imaging target and time information among the target information are output to the storage unit 5.

  In step S32, the storage unit 5 reads related information corresponding to the type of imaging target received in step S31 from the information stored in the storage unit 5, and outputs the related information to the drawing unit 6. In step S33, the drawing unit 6 draws the related information according to the position and orientation, using the position and orientation information of the imaging target received in step S31 and the related information received in step S32 as inputs. Generate information. In step S34, the drawing unit 6 transmits the generated drawing information via the server-side transmission unit 22 and the terminal-side reception unit 12, so that the server 20 outputs the drawing information to the control unit 4 of the terminal 10.

  When drawing information is generated in step S33, the drawing information is associated with the time information received by the drawing unit 6 in step S31 (time information of the captured image in which the position and orientation used for the drawing are calculated). . In step S34, drawing information associated with the time information is transmitted to the terminal 10.

  The step group S40 includes steps S41 and S42 performed in this order, and the presentation in the presentation unit 3 is mainly performed. In step S41, the control unit 4 generates presentation information by superimposing the drawing information received in step S34 on the captured image received in step S13, and then transmits the generated presentation information to the presentation unit 3. Output. In step S42, the presentation unit 3 presents the received presentation information.

  In addition, when superimposing drawing information and a captured image in step S41, the thing with which the time information matched with each corresponds is superimposed.

  Here, the series of steps shown in FIG. 2 is performed on the captured image P (t) at each time t acquired in step S12, and the loop unit process L1 is repeated, so that the series of times t, t + Processing is sequentially performed on the captured images P (t), P (t + 1), P (t + 2),. Here, each piece of information corresponding to the captured image P (t) at time t is subject information O (t) including position / posture C (t) and subject type K (t), related information R (t), drawing Information G (t) and presentation information D (t) are assumed.

  In addition, each said time t is the time information linked | related with the captured image demonstrated in step S23.

  In one embodiment, the loop unit process L1 can be performed synchronously at time t as described above. That is, all the information handled in the loop unit process L1 can be information corresponding to the captured image P (t) at time t acquired in step S12 as described above.

  On the other hand, when the loop unit process L1 is performed synchronously at time t as described above, the loop unit process L1 must be performed at each time interval required for the completion of the following process. That is, the process L1 can be performed only at time intervals obtained by adding the time required for completing steps S12, S22, S33, S42, etc., the time required for data transmission / reception via the network in steps S31, S34, and the like. . Therefore, if the time interval is large, the rate of the presentation process in the presentation unit 3 becomes rough, which is not preferable when augmented reality display or the like is performed in real time.

  Therefore, in order to improve the rate of the presentation process, the loop unit process L1 is divided into the following first loop process L11 and second loop process L12, and then each loop process L11, L12 (in FIG. 2). 2 may be performed in an embodiment in which each of them is performed independently (not shown). That is, as the first loop process L11, the terminal 10 side continuously performs a process at a predetermined interval using the step groups S10, S20, and S40 (and the transmission process of step S31 and the reception process of step S34) as a loop process unit. . Independently of the process L11, as the second loop process L12, the server 20 side continuously performs a process at a predetermined interval with the step group S30 as a loop process unit. The predetermined interval may not be constant, and may be different between the terminal 10 and the server 20.

  The second loop processing L12 on the server 20 side is preferably started as soon as the target information is received from the terminal 10 side, and is processed as fast as the computing resources of the server 20 allow. On the other hand, in the first loop process L11 on the terminal 10 side, the step group S40 is performed as a process according to whether or not the drawing information is received from the server 20 side, and the captured image is buffered (temporarily stored) as follows. It is preferable to perform processing.

  That is, in the loop processing L11, the control unit 4 buffers the captured image P (t) for a certain amount of time (particularly, a certain amount of time considering the network delay of information transmission / reception between the terminal 10 and the server 20). As soon as the corresponding drawing information G (t) is obtained from the drawing unit 6 in the loop processing L12 of the server 20, the corresponding presentation information D (t) is generated and presented to the presentation unit 3. Good. In this way, the loop processing L11 and L12 are performed independently of each other, whereby processing suitable for the respective computing resources of the terminal 10 and the server 20 (and the network delay between the terminal 10 and the server 20) is possible, and the time Since the information t is shared between the terminal 10 and the server 20, the terminal 10 can generate appropriate presentation information D (t).

In one embodiment, when performing step group S40 in each N-th time (N = 1, 2,...) Of loop processing L11, there are a plurality of drawing information G (t) that have been received from the server 20 side. The presentation information D (t _latest ) may be presented to the presentation unit 3 by superimposing the drawing information G (t _latest ) corresponding to the _latest time t _latest on the corresponding captured image P (t _latest ). .

In this case, the drawing information {G (t) | t ≦ t _latest } before the latest time t _latest for which the presentation information D (t _latest ) is generated is the N + 1th step group S40 in the loop processing L11. You may make it exclude from the process target in. Similarly, the picked-up image P (t) held by buffering in the control unit 4 is also targeted at the time after the latest time t _latest (t> t _latest ) at the N + 1th time of the loop processing L11. You can do it.

  Therefore, in the one embodiment, when the presentation unit 3 presents the presentation information D (t) by the buffering in the step group S40 of the Nth loop process L11, the step group in the Nth loop process L11 There is a relationship that the captured image at the current time acquired in S10 is P (t + n) (where n ≧ 0). That is, on the terminal 10 side, the step group S10, S20, S40 is performed in the loop processing L11 at each current time t, but the target of executing the step group S40 at the current time t is any past time The captured image P (tn) that was captured and acquired in step S12 of the loop processing L11 of tn and buffered and held until the current time t, and newly received from the server 20 in the loop processing L11 of the current time t The presentation information D (tn) generated from the drawn drawing information G (tn).

  In addition, the following situation may occur depending on fluctuations in network delay and the like. That is, the drawing information G (t) is received from the server 20 in the step group S40 in a certain N-th loop processing L11 and the presentation information D (t) is generated, but the next N + 1-th loop processing L11 When trying to implement step S40 in the above, new next drawing information G (t + 1) (or G (t + m) may be set as m ≧ 2) is not received from the server 20 side Is the situation.

  When this situation occurs, the latest drawing information (excluding drawing information already used to generate the presentation information) does not exist on the terminal 10 side, so that the presentation process in the (N + 1) th step group S40 cannot be performed. turn into. Therefore, as will be described in detail later, the latest drawing information G (t + 1) to be received from the server 20 side is generated by interpolation from the drawing information G (t) used for the latest (Nth step group S40). Thus, the presentation information D (t + 1) may be generated by interpolation on the captured image P (t + 1) and presented to the presentation unit 3.

  FIG. 3 shows the presentation information D that is generated and presented at the time N = 1, 2, 3,... 6 of the loop processing L11 in the terminal 10 (time t = 1, 2, 3,..., 6 respectively). A schematic example of (t), held buffering data Buffer (t), a captured image P (t) to be captured, and drawing information G (t) received from the server 20 is shown in a table format.

  In the example of FIG. 3, the loop processing L11 of N = 1, 2, 3 is equivalent to immediately after the start of the loop processing, and the target information O (1) transmitted as corresponding to the first captured image P (1) (FIG. 3) The response G (1) from the server 20 to (not shown in FIG. 3) has not been received. In other words, since the drawing information G (t) has not been received from the server 20 at the N = 1, 2, 3rd time, the buffering data Buffer (t) is sequentially accumulated, but the presentation information D (t) is generated. Can not do it. By receiving from the server 20 the drawing information G (1) corresponding to the captured image P (1) for the first time in the N = 4th loop processing L11, and superimposing these to generate the presentation information D (1), The captured image P (1) used for the generation is deleted from the buffering data Buffer (t) in the next N = 5th loop processing L11. Similarly, N = 5 and 6th reception of drawing information G (2) and G (3), respectively, are superimposed on the captured images P (2) and P (3) and presented information D (2) and D ( 3) is generated, and the captured images P (2) and P (3) already used in the buffering data Buffer (t) of the next loop processing L11 are deleted. In FIG. 3, the processing after N = 7 is omitted, but the same processing is continued as long as the drawing information G (t) is received.

  The details of each unit in FIG. 1 that operates in the operation sequence of each embodiment in FIG. 2 will be described below in the order of each unit of the terminal 10 and each unit of the server 20.

  In Steps S11, S12, and S13 of FIG. 2, the imaging unit 1 receives an instruction from the control unit 4 and images the imaging target, and outputs the obtained captured image to the control unit 4. As the imaging unit 1, a digital camera often provided as a standard in recent portable terminals can be used.

  In steps S21 and S22 in FIG. 2, the recognition unit 2 inputs the captured image from the control unit 4, recognizes the type of the imaging target from the captured image, and also determines the relative position and orientation of the imaging target with respect to the imaging unit 1. Calculate Further, in step S23 in FIG. 2, the recognition unit 2 uses the type, position, and orientation of the imaging target and time information associated with the captured image as target information, and outputs the target information to the control unit 4.

  Here, for an imaging target whose type is a recognition target, one or more types of predetermined targets (for example, a square marker disclosed in the following Non-Patent Document 2 and known in the augmented reality display field, etc.) are set in advance. The feature information on the image is extracted in advance and registered in the recognition unit 2. The recognition unit 2 extracts feature information from the captured image and performs matching with the pre-registered feature information, thereby specifying the imaging target captured in the captured image, as well as the relative position and Estimate posture.

Non-Patent Document 2 Kato, M. Billinghurst, Asano, Tachibana, "Augmented reality system based on marker tracking and its calibration", Transactions of the Virtual Reality Society of Japan, vol.4, 20
no.4, pp.607-617, 1999.

  The feature information for an image has a property that is invariant to any one of rotation and enlargement / reduction, projective change (distortion due to projective transformation), or any combination thereof, such as a well-known SIFT feature amount or SURF feature amount. Then, a local feature amount calculated based on a relative luminance gradient in a local region of the image can be used. Alternatively, a well-known ORB feature amount having the same property may be used. The SIFT feature value is disclosed in Non-Patent Document 3 below, the SURF feature value is disclosed in Non-Patent Document 4 below, and the ORB feature value is disclosed in Non-Patent Document 5 below.

Non-Patent Document 3 “DGLowe, Distinctive image features from scale-invariant key points, Proc. Of Int. Journal of Computer Vision (IJCV), 60 (2) pp.91-110 (2004)”
Non-Patent Document 4 “H. Bay, T. Tuytelaars, and LVGool, SURF: Speed Up Robust Features, Proc. Of Int. Conf. Of ECCV, (2006)”
Non-Patent Document 5 “Ethan Rublee, Vincent Rabaud, Kurt Konolige, Gary R. Bradski: ORB: An efficient alternative to SIFT or SURF. ICCV 2011: 2564-2571.”

  For the matching between the feature amounts (feature information) when the recognition unit 2 specifies the imaging target, each known method in the field of image recognition can be used. Further, a relation of plane projective transformation for mutually converting the coordinate information on the image in the feature information of the imaging target registered in advance and the coordinate information on the captured image of the feature information extracted and matched from the captured image is obtained. Thus, the recognition unit 2 can calculate the relative position and orientation between the imaging target and the imaging unit 1. For the calculation of the position and orientation, a well-known method in the augmented reality display field or the like as disclosed in Non-Patent Document 2 described above can be used.

  The terminal side transmission unit 11 transmits the target information obtained by the recognition unit 2 and output to the control unit 4 to the server side reception unit 21 of the server 20 in step S31 of FIG.

  The terminal-side receiving unit 12 receives drawing information as a response to the target information transmitted by the terminal-side transmitting unit 11 from the server-side transmitting unit 22 of the server 20 and outputs it to the control unit 4 in step S34 of FIG. To do.

  The presentation unit 3 inputs and presents the presentation information generated by superimposing the drawing information and the captured image from the control unit 4 in steps S41 and S42 in FIG. In one embodiment, the presentation may be an augmented reality display on the captured image in accordance with the recognized imaging target. For the presentation unit 3, a display that is normally provided in a mobile terminal or the like can be used.

  In steps S11 and S13 of FIG. 2, the control unit 4 inputs a captured image associated with time information at the time of imaging from the imaging unit 1 that has given the instruction, and in step S21, the captured image is input to the recognition unit 2. Is output.

  Here, as described above as the buffering process, in one embodiment, the captured image is held in the control unit 4 for a certain period of time. The predetermined time may be set to a length that allows for the time required for processing in the recognition unit 2, transmission from the terminal 10 to the server 20, processing in the drawing unit 6, transmission from the server 20 to the terminal 10, and the like. desirable.

  The set length is the maximum size (the maximum number of captured images) to be buffered. In the embodiment in which captured images (and captured images older than this) already used to generate the presentation information are sequentially deleted from the buffering target, depending on the reception status of the drawing information from the server 20 side, etc. The number of captured images that are actually buffered changes over time in a range equal to or less than the maximum number.

  The control unit 4 also inputs the target information recognized by the recognition unit 2 in step S23 of FIG. 2, and outputs the input target information to the terminal side transmission unit 11 in step S31. The control unit 4 further inputs drawing information from the terminal-side receiving unit 12 in step S34 of FIG. 2, and in step S41, the time information associated with the input drawing information matches the captured image whose time information matches. Presentation information is generated by superimposing the input drawing information and output to the presentation unit 3.

  That is, as described above as the buffering process, the input drawing information G (t) includes time information as time t, so that by superimposing on the captured image P (t) where the time information matches, Presentation information D (t) corresponding to time t is generated. The actual time when the presentation information D (t) is generated is generally time t + n (n ≧ 0), and the captured image P (t + n) is obtained in the imaging unit 1, but the control unit In 4, the captured image P (t) is held by buffering the captured image for a certain time.

  Here, when the drawing information is superimposed on the captured image, since it is drawn according to the position and orientation of the imaging target in the drawing unit 6 described later, it is simply superimposed according to the position information included in the drawing information. Can only be realized. Therefore, the superimposition processing can be performed at high speed without requiring a large calculation load in the terminal 10.

  As an embodiment of the superimposition, in order to reduce the transmission time of drawing information (time required for step S34 in FIG. 2), the drawing information resolution (size) is reduced when drawing by the drawing unit 6. Alternatively, the drawing information may be converted into the original resolution by the control unit 4 and then superimposed on the captured image. In this case, the drawing unit 6 associates the information about the range occupied by the drawing information at the original resolution with the image coordinates and the information on how much the resolution is reduced from the original, and then draws the drawing information drawn at a small resolution. What is necessary is just to transmit to the terminal 10 side. Note that, by reducing the resolution of the drawing information drawn by the drawing unit 6 of the server 20, an effect of reducing the drawing time in the drawing unit 6 can be obtained in addition to the reduction of the drawing information transmission time.

  Alternatively, as an embodiment related to the above-described one embodiment, the resolution of the drawing information and the compression rate (encoding for communication) according to the throughput of communication (communication in step S34 (or S34 and S31 in FIG. 2)). (Compression rate) may be controlled. The communication throughput may be monitored by the control unit 4 of the terminal 10, and the rendering unit 6 may be instructed by adding the resolution and / or compression rate corresponding to the throughput to the transmission information in step S31. The drawing unit 6 itself may monitor the throughput and set the resolution and / or compression rate.

In one embodiment, as mentioned as the interpolation process in the buffering process described above, the following may be performed. That is, in a period in which there is no drawing information G (t) that matches any time information t of the captured image P (t) buffered for a certain period of time (period in which the terminal 10 has not received from the server 20), The interpolation processing may be performed. Here, the captured image P (t) buffered as described above is obtained from the server 20 with the corresponding drawing information G (t _latest ) acquired at the latest time t _latest and the corresponding presentation information D (t _latest ). The time information t (t> t _latest ) after the one that is actually generated is _linked .

In the interpolating process, first, the recognition unit 2 regards the target information O (tn) at the latest time (in order to clarify the relationship with the target time t within the period, the latest time t _latest = tn with n> 0) A relationship H (tn, t) for converting the calculated relative position and orientation C (tn) into the position and orientation C (t) of the target information O (t) at the time t to be interpolated is obtained.

  Next, the interpolated drawing information G ′ (t) is generated by applying the conversion H (tn, t) to the corresponding drawing information G (tn) at the latest time, and is superimposed on the captured image P (t). The presentation information D (t) is generated by combining them. By the interpolation process, it is possible to generate the presentation information D (t) at the time t while suppressing the apparent shift between the captured image P (t) and the drawing information G (t−n).

  Here, the transformation H (tn, t) can be accurately obtained as a planar projective transformation, similar to the calculation of the position and orientation in the recognition unit 2, but the approximation is made to speed up the processing in the control unit 4. As a relationship of movement / enlargement / reduction on the image.

  It should be noted that for which time t of the captured images P (t) buffered for a certain time, the interpolation processing is performed at a predetermined position within a certain time range (for example, the latest time, the oldest time, or the middle Or the like at a predetermined time such as t).

  In the above, each part of the terminal 10 of FIG. 1 was demonstrated. Next, each part of the server 20 in FIG. 1 will be described.

  The server-side receiving unit 21 receives the information transmitted by the terminal-side transmitting unit 11, that is, the target information obtained by the recognizing unit 2 in step 31 of FIG. Of the received target information, the type of imaging target and time information are output to the storage unit 5. Of the received target information, the position and orientation information of the imaging target and the time information are output to the drawing unit 6.

  The storage unit 5 stores related information according to the imaging target, reads the related information according to the type of imaging target received at step S31 in step S32 of FIG.

  The relevant information is equivalent to the original data so that the presentation unit 3 finally superimposes it as drawing information on the captured image, and is characteristic information of the imaging target (the same as that used in recognition of the recognition unit 2). An administrator or the like prepares in advance as a predetermined target defined in the reference coordinate system defined in the feature information) and stores it in the storage unit 5.

  For example, when the imaging target is a square marker as disclosed in Non-Patent Document 2 described above, a predetermined three-dimensional object in a reference coordinate system defined by the square marker (including shape, color, pattern, etc. in its definition) For example, related information can be prepared in a format such as CAD data, which is a well-known format. Since the related information can be defined for each type of imaging target, a desired presentation process can be performed in the presentation unit 3 according to the application.

  The drawing unit 6 receives the relevant information from the storage unit 5 and the position and orientation information from the server-side receiving unit 21 and draws the relevant information according to the position and orientation in step S33 of FIG. And the time information input from the server-side receiving unit 21 is associated with the drawing information.

  The drawing unit 6 further outputs the drawing information to the server side transmission unit 22 in step S34 of FIG. Note that the output drawing information is associated with time information, includes information on the range when superimposed on the captured image, and scale information indicating the degree of reduction when the resolution is reduced. Also included.

  Here, the generation of the drawing information by drawing the related information according to the position and orientation means that the relevant information as the original data prepared in the reference coordinate system is the position and orientation (of the imaging unit 1). This corresponds to calculating and determining how it will look when viewed (as a captured image). That is, it corresponds to rendering the related information as the above-mentioned original data according to the position and orientation (three-dimensional rendering). The calculation at the time of generation is well known in the field of three-dimensional computer graphics.

  At this time, when the presentation information is finally generated by superimposing the captured image on the control unit 4 of the terminal 10, the drawing unit 6 draws only the related information so that the presentation information can be generated only by the superposition. The target of. Therefore, the processing time can be shortened by not drawing the entire screen corresponding to the captured image, and the transmission time can also be shortened by minimizing the amount of drawing information transmitted to the terminal 10. In addition, when only the relevant information is drawn, information on what coordinate range is occupied when finally superimposing on the captured image is also calculated, so that the captured image Can be omitted.

  Depending on the relationship between the pre-defined related information and the position and orientation of the drawing target, as a result of drawing the relevant information according to the position and orientation, it may happen that the entire screen corresponding to the captured image is accidentally occupied. It may be. Such a case can also be regarded as a special case of drawing as a part of the screen.

  In one embodiment, in order to finally present the high-quality presentation information when the presentation unit 3 displays augmented reality, the drawing unit 6 displays drawing information corresponding to the position and orientation of the related information in consideration of the environment map. You may make it draw. In this case, in the environment where the imaging unit 1 captures an image, the arrangement, color, intensity, and the like of the surrounding light sources are acquired in advance as an environment map, and the environment map is reflected according to the position and orientation and the shape of the related information. It is sufficient to draw the drawing information. Note that a technique considering the environment map is well known in the field of three-dimensional computer graphics and the like.

  In one embodiment, as mentioned in the description of the control unit 4, the drawing unit 6 reduces the resolution (size, number of constituent pixels) of the drawing information in order to reduce the drawing time and transmission time of the related information. Then, the drawing information may be generated, and the scale information and the drawing information regarding how small the drawing information may be transmitted to the control unit 4. The control unit 4 can reconstruct the drawing information by converting the received drawing information to return to the original resolution according to the scale information. Alternatively, the resolution and compression rate of the drawing information can be controlled according to the communication throughput as described above. Thereby, the effect of shortening the transmission time of drawing information is also acquired.

  In the related information stored in advance in the storage unit 5, a predetermined resolution (original resolution) corresponding to the size of the captured image to be superimposed and the position and orientation at the time of drawing is defined in advance. When the resolution is set to be small as described above, the original resolution is set as a reference, and the resolution is set to be smaller than the reference.

  Here, how much resolution is actually used with respect to the original resolution (scale s = 1) is determined by specifying the scale s as a ratio between the resolutions in the range of 0 <s ≦ 1. Can be determined. At this time, as described above, the scale s may be automatically designated by a predetermined function formula for the throughput of communication between the terminal 10 and the server 20, or a user (the user who uses the information presentation system 30 ( The specification of the scale s may be received manually from an administrator). When the scale s is designated by the communication throughput, the throughput calculated by performing test communication between the terminal 10 and the server 20 in advance may be used in a fixed manner, or information presentation by the information presentation system 30 may be performed. However, the throughput may be calculated sequentially, and the throughput updated each time may be used.

  FIG. 4 shows the related information prepared and stored for each imaging target in the storage unit 5 as described above, the drawing information obtained by drawing the related information according to the position and orientation in the drawing unit 6, and the control. FIG. 4 is a diagram showing a typical example of presentation information obtained by superimposing drawing information on a captured image in a unit 4.

  In FIG. 4, [1] is an example of the site F that the imaging unit 1 captures, and an exhibition stand S is disposed on the ground G, and a predetermined imaging target M (for example, a well-known square marker) is disposed on the upper surface thereof. Has been. [1] also shows the positions and orientations C (t1), C (t2), and C (t3) when the imaging unit 1 images the scene at times t1, t2, and t3. [2] is an example of the related information R corresponding to the imaging target M. For the imaging target M as a square marker, the related information R is prepared in advance as a rectangular parallelepiped having the square portion as a bottom surface.

  In [2], the related information R is drawn as if it were a real object in the field F. Actually, the data defined in the reference coordinate system defined by the imaging target M is the related information R. When the presentation unit 3 finally displays the augmented reality, the related information R is drawn and presented as if the real object as shown in [2] exists.

  The captured images obtained by the imaging unit 1 imaging at the positions and orientations C (t1), C (t2), and C (t3) at the times t1, t2, and t3 in [1] in FIG. The captured images are shown as P (t1), P (t2), and P (t3). These are images in which the imaging target M is viewed from the front left side, the front side, and the front right side. Further, the drawing information obtained by drawing the related information R by the drawing unit 6 in accordance with the position and orientation C (t1), C (t2), and C (t3) in [1] is the drawing information G (t1) in [4], respectively. , G (t2), G (t3). Then, by superimposing the drawing information G (t1), G (t2), G (t3) of [4] on the captured image P (t1), P (t2), P (t3) of [3] The presentation information generated in the control unit 4 is shown as presentation information D (t1), D (t2), and D (t3) in [5], respectively.

  As illustrated in [4], the drawing information G (t1), G (t2), and G (t3) obtained by drawing the related information R configured in a rectangular parallelepiped shape according to the position and orientation are the imaging of [3]. Since it includes information on which range of the images P (t1), P (t2), and P (t3), it is superimposed as it is without positioning, so that the presentation information D (t1), [5] D (t2) and D (t3) can be generated. In addition, the related information R drawn by the drawing information G (t1), G (t2), and G (t3) is configured as only a partial range without occupying the entire range of the captured image. (Regarding the region other than the drawn related information R, only the information that the captured image is displayed as it is without being superimposed (overwritten) on the captured image is defined in the rendering information. The amount of data as image data is zero because it is a region that is

FIG. 5 is a diagram schematically illustrating the drawing information with a reduced resolution. The drawing information G (t1) _[[ 1] in which the resolution of the drawing information G (t1) in FIG. 4 is reduced to half (scale s = 1/2) _{. 1/2]} is shown. Even when using the drawing information G (t1) _[1/2] , the drawing information G (t1) depicting the related information R at the original resolution is shown as the information G ′ (t1) drawn in dotted lines. Information on the range occupied in the captured image is transmitted to the control unit 4. In the control unit 4, the drawing information G (t1) _[1/2] is doubled according to the scale information (s = _1/2 ) to return to the original resolution, and then specified by the information G ′ (t1) Then, it is arranged in a range within the captured image to reconstruct what corresponds to the original drawing information G (t1).

  The server-side transmission unit 22 inputs the drawing information drawn by the drawing unit 6 in step S34 in FIG. 2 and transmits the drawing information to the terminal-side receiving unit 12 of the terminal 10.

  In the above, each part of FIG. 1 was demonstrated. The present invention can produce the following effects. Since the drawing unit 6 is a configuration provided in the server 20, when drawing is performed in consideration of the position and orientation in the related information, even if the drawing is processing that is generally computationally expensive considering the environment map as described above, the terminal There is no computational load on the 10 side. In addition, since data transmission / reception between the server 20 and the terminal 10 does not always transmit / receive the entire image, only necessary information is transmitted / received, so the transmission amount is suppressed, and a large delay due to a huge transmission amount is not caused. Does not occur. Further, by absorbing the minimum delay that occurs by the buffering process described above, the presentation process in the presentation unit 3 can be realized almost in real time as a user feeling.

  The present invention can also be provided as each program that causes a computer to function as the terminal 10 or the server 20. The computer can adopt a known hardware configuration such as a CPU (Central Processing Unit), a memory, and various I / Fs, and the CPU instructs the computer to correspond to the functions of each part of the terminal 10 or the server 20. Will be executed.

  DESCRIPTION OF SYMBOLS 10 ... Terminal, 20 ... Server, 30 ... Information presentation system, 1 ... Imaging part, 2 ... Recognition part, 3 ... Presentation part, 4 ... Control part, 11 ... Terminal side transmission part, 12 ... Terminal side reception part, 5 ... Storage unit, 6 ... drawing unit, 21 ... server side receiving unit, 22 ... server side transmitting unit

Claims (8)

An information presentation system comprising a terminal that presents presentation information and a server that communicates with the terminal,
The terminal
An imaging unit that performs imaging and obtains a captured image;
A recognition unit that recognizes a predetermined imaging target from the captured image, calculates a position and orientation relative to the imaging unit, and uses the type of the recognized imaging target and the calculated position and orientation as target information;
A terminal-side transmitter that transmits the target information to the server;
A terminal-side receiving unit that receives drawing information drawn by the server from the server;
A controller that generates presentation information by superimposing the received drawing information on the captured image;
A presentation unit for presenting the generated presentation information,
The server
A storage unit that stores relevant information that is prepared in advance for each imaging target and configured for predetermined augmented reality display;
A server-side receiving unit that receives target information transmitted from the terminal-side transmitting unit;
A drawing unit that reads the related information corresponding to the imaging target in the received target information from the storage unit and generates the drawing information by drawing the read related information according to the position and orientation in the received target information; ,
And a server-side transmitter that transmits the drawn drawing information to the terminal.   The recognition unit extracts, from the captured image, an image feature amount that has an invariable property with respect to any one or any combination of enlargement / reduction, rotation, luminance change, projective change, and a plurality of types of predetermined imaging targets. The information presentation system according to claim 1, wherein the recognition and the calculation are performed by matching with an image feature amount having the property extracted in advance for each of the images. The imaging unit, after associating time information at the time of each imaging, continuously acquires captured images at a predetermined rate on the time axis,
In the recognizing unit, after associating time information associated with the imaging, the target information,
The drawing unit generates the drawing information after associating the time information associated with the target information,
In the control unit, the captured image associated with the time information is buffered for a certain amount of time, and the time information associated with the buffered captured image is associated with the time information associated with the received drawing information. The information presentation system according to claim 1, wherein the presentation information is generated by superimposing the received drawing information on a matching captured image.   The said control part produces | generates the said presentation information by superimposing the drawing information with the newest time information linked | attached on the said captured image, when the received said drawing information exists in multiple numbers. Information presentation system described. In the control unit, after the presentation information is generated by superimposing the drawing information with the latest associated time information on the captured image, new drawing information is not received in the terminal-side receiving unit.
Conversion relationship between the position and orientation calculated by the recognition unit in the acquired captured image in the period and the position and orientation calculated by the recognition unit in the captured image on which the latest drawing information is superimposed Seeking
The said presentation information is produced | generated by superimposing what changed the said latest drawing information by the said conversion relation with respect to the said captured image acquired in the said period. Information presentation system. The storage unit is prepared in advance for each imaging target, stores related information configured for predetermined augmented reality display realized by superimposing a predetermined target,
The drawing unit draws the related information as an image that the predetermined target occupies in the captured image at the position and orientation in the received target information, thereby associating information on a range occupied in the captured image with the information Generate drawing information,
6. The information presentation system according to claim 1, wherein the control unit superimposes the drawing information according to the information of the linked range when superimposing the drawing information. The storage unit is prepared in advance for each imaging target, configured for a predetermined augmented reality display, stores related information given at a predetermined resolution,
When generating the drawing information by drawing the read related information in the drawing unit, the predetermined resolution according to a scale determined based on a communication throughput between the terminal and the server or a scale specified by a user. 7. The information presentation system according to claim 1, wherein related information given at the following resolution is used. The server-side transmission unit transmits the drawing information including the scale information to the terminal,
The control unit uses the related information given at a resolution equal to or lower than the predetermined resolution to return the drawn drawing information to the predetermined resolution according to the scale information, and then generates the presentation information. The information presentation system according to claim 7.