JP2018106611A - AR information display device and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an AR information display device and program for intuitively achieving guidance to a predetermined target object from among a plurality of objects.SOLUTION: An AR information display device 20 for presenting a display of guidance from a plurality of objects to a predetermined target object is provided as including: an imaging unit 1 for obtaining a captured image by performing photographing; a recognition unit 2 for recognizing each of the plurality of objects from the captured image; a generation unit 3 for generating guidance information to the target object on the basis of the object recognized by the recognition unit 2 when the target object is not recognized by the recognition unit 2; and a display unit 4 for displaying the guidance information.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an AR information display device and a program for intuitively realizing guidance from a plurality of objects to a predetermined target object.

  As a conventional technique for realizing user assistance by guidance or guidance using AR (augmented reality) technology, for example, the following is available.

  This type of device disclosed in Patent Document 1 (Title of Invention: Product Information Providing Terminal Device and Product Information Providing System) provides real-time optimal product information on products displayed in stores using AR technology from captured images. The product information providing terminal device for providing to the customer, considering that the power consumption is efficiently suppressed, when the customer is not in a moving state or when the customer's head is stopped Product information was provided.

  This type of device disclosed in Patent Document 2 (Title: User Interface for Head Mount Display) acquires the movement of the head of a user wearing an HMD (Head Mount Display) with a sensor, and the user faces forward. The first screen consisting of the field of view when the user is facing down, the second screen consisting of a map containing the field of view and the user's position (overhead view) when the user looks down, and persistent data independent of the physical objects in the field of view Is a user interface technology for HMD.

Patent No. 5656457 Patent No.5909034

  However, in the conventional techniques as described above, it has not always been possible to effectively assist the user in reaching the target object. In Patent Document 1, only information related to products directly corresponding to the instruction information acquired by the terminal device is displayed. Therefore, when a large number of products are lined up, the user searches individual instruction information to find a target product. It was difficult. Further, in Patent Document 2, the map displayed in the field of view of the HMD depicts a physical surface on which the user is located based on sensor information, and navigation information is provided based on the current location of the user. It was not always intuitive.

  In view of the above problems, the present invention intuitively guides to a predetermined target object by changing the mode of guidance information including the position of the starting point and the position of the end point when searching for the target object from a plurality of objects. It is an object of the present invention to provide an AR information display device and a program that can be realized.

  In order to achieve the above object, the present invention provides an AR information display device that performs display that guides a plurality of objects to a predetermined target object, the photographing unit that performs photographing and obtains a photographed image, and the photographing A recognition unit that recognizes each of the plurality of objects from an image, and guidance information to the target object based on the object that is recognized by the recognition unit when the target object is not recognized by the recognition unit And a display unit for displaying the guidance information. Further, the present invention is a program that causes a computer to function as the AR information display device.

  According to the present invention, the guidance information is made more intuitive by changing the mode of the guidance information to the target object based on the target recognized by the recognition unit, the situation at the time of recognition, and the operation purpose. Can be generated and displayed.

It is a functional block diagram of the AR information display device according to an embodiment. It is a functional block diagram of the production | generation part which concerns on one Embodiment. It is a schematic diagram for demonstrating the distinction between Case 2 and Case 3 determined by the determination unit. It is a figure which shows the typical example of guidance information. It is a flowchart of operation | movement of the AR information display apparatus which concerns on one Embodiment. It is a figure for demonstrating 2nd embodiment of the arrow as guidance information. It is a figure for demonstrating 2nd embodiment of the arrow as guidance information. It is a figure for demonstrating 3rd embodiment of the arrow as guidance information. It is a figure which shows the distinction of the arrow (and the set end point) in case 3 and case 2 are applicable. It is a figure which shows the example of arrangement | positioning etc. in order to demonstrate the production | generation of guidance information. It is a figure which shows the schematic example of embodiment which comprises guidance information with an overhead view. It is a figure which shows the example which covers and displays the area | region of the object which is not recognized on an overhead view as an example corresponding to the example of FIG. FIG. 10 is a diagram illustrating a schematic example of an embodiment in which the end point is always determined as the midpoint of the target object when the guidance information is configured by an overhead view.

  FIG. 1 is a functional block diagram of an AR information display device according to an embodiment. The AR information display device 20 includes an imaging unit 1, a recognition unit 2, a generation unit 3, a display unit 4, and a storage unit 5. The general functions of the respective units 1 to 5 are as follows.

  The photographing unit 1 shoots the real world that is a target for which AR display is performed by a user's photographing operation, and outputs the obtained photographed image to the recognition unit 2. As indicated by the branched dotted line L1, in some embodiments, the captured image obtained by the imaging unit 1 is output not only to the recognition unit 2 but also to the generation unit 3. The shooting unit 1 continuously shoots images (moving images) at a predetermined rate on the time axis, and the shot images as frame images at each time are sent to the recognition unit 2 (and the generation unit 3 in some embodiments). Is output. As hardware for realizing the photographing unit 1, a normal digital camera can be used.

  The recognition unit 2 analyzes the photographed image obtained by the photographing unit 1, recognizes each of a plurality of objects registered in advance in the storage unit 5 from the photographed image, and estimates the position and posture in the real world. Then, the recognition result (including the estimated position and orientation) is output to the generation unit 3. For the processing by the recognition unit 2, existing technology in the AR technical field can be used.

  In other words, with regard to recognition processing, for example, by using local feature amounts defined around feature points such as SIFT feature amounts, it is possible to determine which target exists in which part of the captured image, that is, each target. The coordinate arrangement in the screen coordinate system can be specified. Here, the local feature amount of each target is registered in the storage unit 5 in advance, the recognition unit 2 extracts the local feature amount from the captured image, and collates with the local feature amount of each target stored in the storage unit 5 This makes it possible to recognize which target is present in the captured image. Furthermore, it is possible to obtain a homography matrix for converting the coordinate position of a predetermined target point that has been recognized from the world coordinate system to the screen coordinate system. In order to obtain the homography matrix, the storage unit 5 stores in advance the positions of four or more points in a predetermined world coordinate system for each target. As is well known in the AR field, obtaining a homography matrix is equivalent to estimating the position and orientation of an object that can be recognized with respect to the camera constituting the imaging unit 1.

  Based on the recognition result obtained by the recognition unit 2, the generation unit 3 acquires or estimates the position of a predetermined point of the target target in the screen coordinate system and generates guidance information. At that time, when the target object cannot be recognized, a homography matrix of a predetermined point of the recognized object existing in the vicinity of the target object is used. By converting the position information of the predetermined point of each target stored in the storage unit 5 in the world coordinate system using the homography matrix, the coordinate position of the predetermined point of the target target in the same situation is estimated. be able to. The generation unit 3 generates guidance information based on the acquired or estimated target target position and outputs the guidance information to the display unit 4. The display unit 4 displays the guidance information obtained by the recognition unit 2 as an AR display for the user.

  Particularly in the present invention, when the target unit is not recognized by the recognition unit 2, that is, when the target unit is not photographed by the photographing unit 1 (note that the target target is photographed but is recognized due to the influence of noise or the like). In some cases, it may not be recognized due to a recognition processing error in part 2, but this is not considered here). Based on an object other than the target object recognized in recognition part 2, generation part 3 generates guidance information. As the guidance information is displayed by the display unit 4 as AR, intuitive guidance for the user can be realized. An example of the guidance information is AR display information using an icon such as an arrow or a thick frame surrounding the target object. Details thereof will be described later.

  Here, the display unit 4 can display the guidance information as being associated with the captured image obtained by the imaging unit 1 (positional relationship or the like) by various existing display modes used in the AR technology. .

  In one embodiment, as indicated by a dotted line L1 in FIG. 1 (and FIG. 2 to be described later), the photographed image obtained by the photographing unit 1 is received by the generating unit 3, and the generating unit 3 displays an AR for the photographed image. After superimposing the guidance information, the information may be output to the display unit 4, and the display unit 4 may display the superposed guidance information on the captured image. In this case, a liquid crystal display or other normal display can be used as hardware for realizing the display unit 4. Also in this case, the AR information display device 20 can be realized by, for example, an information terminal device to which a tablet or other camera and display are attached.

  In another embodiment, the AR information display device 20 may be realized as a head mounted display or the like, and a see-through display used in the head mounted display or the like may be used as hardware for realizing the display unit 4. In this case, according to the existing technology, the field of view of the captured image obtained by the photographing unit 1 is associated with the field of view of the user through the see-through display, and the user sees through the see-through display with the see-through display. Guide information as AR information can be superimposed and presented to the user. In this case, the flow of outputting the captured image obtained by the imaging unit 1 to the generation unit 3 indicated by the dotted line L1 in FIG. 1 (and FIG. 2 described later) is omitted.

  Note that, in some embodiments, as indicated by a dotted line L3 in FIG. 1 (and FIG. 2 described later), the generation unit 3 recognizes instruction information according to the situation determination when the guidance information is generated. The recognition unit 2 may perform recognition processing according to the instruction information. For example, for a specific target that has already been recognized by the recognition unit 2 at a certain point in time, tracking is performed by tracking using, for example, template matching at a later point in time, and recognition using a local feature amount is omitted. The instruction information to that effect may be notified from the generation unit 3 to the recognition unit 2.

  The storage unit 5 stores information necessary for recognition processing and position / orientation estimation processing of each target in the recognition unit 2 and guidance information generation processing in the generation unit 3 as registered in advance by an administrator or the like. In addition, information necessary when the recognition unit 2 and the generation unit 3 perform processing is provided as stored information.

  For example, as described above, the recognition unit 2 recognizes each of a plurality of objects (one of which is a target object) from the captured image and estimates the position and orientation. The storage unit 5 stores a quantity (such as a local feature quantity) and coordinate position information in predetermined world coordinates.

  In one embodiment, the position and orientation of each object i (i = 0, 1, 2,..., Where i is used as the object identifier) to be registered and stored in the storage unit 5 is AR. Similar to the registration of square markers and the like well known in the technical field, the positions of the four points of the object i in the world coordinate system ((x0i, y0i, z0i), (x1i, y1i, z1i), (x2i, y2i, z2i), (x3i, y3i, z3i)). (Note that, as is well known, the positions of the four points do not necessarily need to form a square, and the homo-growth transform converts the coordinates of the four points detected on the image and the four points to be registered with each other. You can register any 4 points that can be calculated, and you can register more than 4 points.)

  In addition, for each object i, the priority (ε0, ε1, ε2, ε3, ..., εi, ...) to become a starting point candidate (the starting point will be described later), and other starting points Various threshold values for selecting candidates can also be registered in the storage unit 5 in advance. In addition, the details of the individual storage information registered and stored in the storage unit 5 will be described as appropriate in the detailed description of the processing contents of the recognition unit 2 and the generation unit 3.

  FIG. 2 is a functional block diagram showing individual functions of the generation unit 3 according to an embodiment. The generation unit 3 includes a determination unit 31, a target estimation unit 32, an end point determination unit 33, a start point determination unit 34, a current state holding unit 35, and an information generation unit 36. The general functions of the units 31 to 36 are as follows.

First, the determination unit 31 determines which of the following cases the recognition result obtained from the recognition unit 2 corresponds to.
(Case 1) ... At least one target is recognized in the photographed image, and no target target exists among the recognized targets.
(Case 2) ... At least one target is recognized in the photographed image, the target target exists in the recognized target, and the target target exists within the display range of the display unit 4. doing.
(Case 3) ... At least one target is recognized in the photographed image, and the target target is present in the recognized target, and the target target is within the display range of the display unit 4. not exist.
(Case 4) ... No object is recognized in the photographed image.

  FIG. 3 is a schematic diagram illustrating an example for explaining the distinction between the case 2 and the case 3. In FIG. 3, the shooting range R1 by the shooting unit 1 is shown as a rectangular range of the shot image, and the display range R4 by the display unit 4 (that is, the range in which the guide information generated by the generation unit 3 can be displayed) is the rectangle. By being included within the range, it is shown as a range narrower than the rectangular range (in the screen coordinate system). In this way, the configuration in which the shooting range R1 is wider than the display range R4 is implemented when, for example, the AR information display device 20 is mounted by a head mounted display, and the display unit 4 is mounted as its see-through display. This is a configuration that can be applied depending on the mode.

  In the example of FIG. 3, there is one recognized object O2 within the display range R4 (at the same time within the shooting range R1), and one is recognized outside the shooting range R1 and outside the display range R4. An example in which the target O3 exists is shown. If target O2 is the target target, it falls under Case 2. On the other hand, if the target O3 is the target target, it corresponds to Case 3.

  As is clear from the example of FIG. 3 described above, unlike the example of FIG. 3, if the shooting range R1 and the display range R4 coincide, the situation corresponding to case 3 occurs. There is no. Therefore, for example, when the AR information display device 20 is mounted with a tablet or the like having a camera and a display that displays the entire area of the camera image, and mounted so that the shooting range R1 and the display range R4 match, The determination unit 31 may omit determination as to whether or not the case 3 is applicable.

  The information generation unit 36 generates guidance information according to the recognition results of the cases 1 to 4 and the recognition unit 2 determined by the determination unit 31 and outputs the guidance information to the display unit 4 (as a final output in the generation unit 3). To do. The target estimation unit 32, the end point determination unit 33, the start point determination unit 34, and each of the units 32 to 35 (any or all of them) of the current state holding unit 35 are required when the information generation unit 36 generates the guidance information. Various elemental processes are performed according to the recognition results of cases 1 to 4 and the recognition unit 2 determined by the determination unit 31, and the processing results are provided to the information generation unit 36. The information generation unit 36 can generate guidance information corresponding to the situation based on the provided processing result.

  The outline of the processing contents of the units 32 to 35 is as follows. As will be described later, various embodiments are possible for the embodiment in which the information generation unit 36 generates the guidance information based on the processing contents of the units 32 to 35.

  In the target estimation unit 32, in the case of Case 1, the position of the target object that does not exist in the captured image (the position of the screen coordinate system outside the range of the captured image) other than the target object that exists in the captured image, Estimate based on objects that have been successfully recognized.

  The end point determination unit 33 determines the end point position of the arrow constituting the guidance information (the position of the arrow tip in the arrow) based on the estimation result by the target estimation unit 32 and the like. The start point determination unit 34 determines the start point position (the position of the root (the opposite side of the arrow tip) of the arrow) that constitutes the guidance information.

  In the embodiment in which the end point determination unit 33 determines the end point, and the start point determination unit 34 determines the start point, the information generation unit 36 uses an arrow (with the display unit 4) with the determined end point and start point as an arrow tip and a root, respectively. “An image of an arrow” displayed according to each aspect, but hereinafter simply referred to as “arrow”) can be generated and used as guidance information for guiding to a target object.

  The current status holding unit 35 is used in the embodiment when the information generating unit 36 configures the guidance information with the overhead view, and the current user in the overhead view (that is, the imaging unit at the current time) 1), which is captured by 1 and recognized by the recognition unit 2, is stored as current information and updated in real time. In the embodiment, the information generation unit 36 refers to the current state information, and displays the target currently recognized by the user separately from the target not currently recognized by the user. Information of the overhead view reflecting the range can be generated, and guidance information based on the information can be generated.

  FIG. 4 shows a schematic example of guidance information generated and displayed by the AR information display device 20 configured as shown in FIGS. 1 and 2 according to the recognition status of the target in the captured image. FIG.

  In FIG. 4, as a schematic example of the real world W to be guided by AR display, a plurality of objects to be recognized O11 to O63 are arranged in front of the shelf R on the upper side. It is. And, in the real world W, a schematic example of the guidance information G1, G2 in the case where the user U who is shooting using the AR information display device 20 is shooting at the position P1 and the case where the user U is shooting at the position P2, It is shown on the lower side in distinction from the real world W on the upper side. That is, the lower side shows not a schematic example of the real world W itself but a schematic example of guidance information generated with respect to the real world W. The user U who is in the real world W shown in the upper side in FIG. 4 faces the shelf R and is shooting using the AR information display device 20, and the guidance obtained on the AR information display device 20 by the shooting. Information G1 and G2 are shown on the lower side of FIG. 4 in distinction from the real world W on the upper side. (Note that in FIG. 4, the lower side is not for showing the real world W, and the user U is not shooting toward the lower side.)

  FIG. 4 shows an example in which a total of 18 objects O11 to O63 existing in the real world W on the upper side are arranged in a substantially grid pattern on the front surface of a substantially flat shelf R. For convenience in explaining these positional relationships, the i-th (1 ≦ i ≦ 6) in the horizontal direction (horizontal direction from left to right) and the j-th in the vertical direction (vertical direction from top to bottom) The symbol “Oij” is given to the object at the position of 1 ≦ j ≦ 3).

  Each of the objects O11 to O63 can be distinguished and recognized by the recognition unit 2 by registering the feature information and the position and orientation in a predetermined world coordinate system common to each other in the storage unit 5. In addition, a predetermined one of the targets O11 to O63 is a target target to which the user is to be guided, and identification information indicating that the target target is a target target is also registered in the storage unit 5 separately from other targets that are not target targets. It may be. In the example of FIG. 4, it is assumed that the target O62 located at the rightmost end and in the middle is registered as the target target. In addition to pre-registering the target object in the storage unit 5, the user, the user's instructor, or the like may be set on the spot.

  According to the AR information display device 20 of the present invention, it is possible to guide the user intuitively and efficiently from the presence of a large number of objects O11 to O63 to the target object O62 as in the schematic example of FIG. .

  In FIG. 4, first, the user U taking a picture from the position P1 at the first time takes a photographed image PC1 as shown on the lower side. The captured image PC1 includes at least one of the objects O11 to O63, but does not include the target object O62, and therefore corresponds to case 1. In this case, the guidance information G1 is generated by expressing that the arrow A1 is generated and the target object has not yet been reached. Note that a specific example of the expression that the target object has not been reached is not particularly shown in FIG. 4, but the expression can be used as a text or other AR message or arrow A1 generation mode (color, shape, etc.) It is. The arrow A1 is generated with the vicinity of the target O22 as the start point and the vicinity of the target O42 as the end point. Therefore, the guidance information G1 can effectively inform the user U at the position P1 that the target object is not currently visible, but if the user goes to the right side, the target object is present.

  Next, the user U guided to the guidance information G1 and moved to the right position P2 captures the captured image PC2 as shown on the lower side. Since the captured image PC1 includes at least one of the targets O11 to O63 and also includes the target target O62, this corresponds to case 2. (Here, for convenience of explanation, it is assumed that the AR information display device 20 is mounted on the above-mentioned tablet or the like and there is no case 3.) In this case, the arrow A2 is generated and the target object O62 has already been reached. The guidance information G2 is generated by generating a highlight display B2 such as enclosing that a thick frame as shown in the figure. The arrow A2 is generated with the vicinity of the target O42 as the start point and the vicinity of the target object O62 as the end point. Therefore, the guidance information G2 effectively indicates that the user U who has been at the position P2 has already reached the target object O62 by moving to the right side, and which target in the image PC2 is the target object O62. Can be communicated to. In the above-described guidance information G1, it is also possible to express that the target object O62 has not been reached by distinguishing the display mode that the highlighted display B2 is not generated as in the guidance information G2.

  As described above, the AR information display device 20 of the present invention can achieve the following effects. In other words, even when the target target is not yet recognized, it is possible to intuitively guide the user to the target target by using what is recognized by other targets that are not target targets. is there.

  In the example of FIG. 4, it is assumed as an example of an ideal case that all the objects in the captured images PC1 and PC2 are recognized, but actually at least a part of them is recognized. If this is the case, the present invention can provide an effect of intuitively guiding the user to the target object, as in the example of FIG. Examples of situations where only some of the objects in the captured image are recognized will be introduced in FIGS.

  As a practical example suitable for use of the AR information display device 20 of the present invention in which the situation shown in FIG. As an example, when displaying or picking up merchandise, the merchandise is fixedly arranged for a certain period of time, so that information indicating the relative positions of individual objects can be measured and registered in the storage unit 5 in advance.

  Specifically, for example, a case is considered in which each of a plurality of drawers in a display of goods or the like is arranged, and a user inputs or extracts an item from a target drawer. Assuming that the objects O11 to O63 in the example of FIG. 4 are drawers, when the drawers are arranged in a wide range as in the example of FIG. 4, first (when there is a user at the position P1), the user draws the drawer from which direction and position I have no idea what to look for. Therefore, even if the target drawer cannot be directly recognized, it is convenient for the user if some information for reaching the target object is provided. Therefore, in the present invention, the relative position of all the objects (these can be expressed by the relative position and orientation of the objects obtained from the position and orientation of each object registered in the common coordinate system) in the storage unit 5. By registering, when an object outside the target can be recognized, the position of the target object can be estimated from the recognized object and the user's line of sight can be guided. That is, as in the example of FIG. 4, the user U can be guided from the line of sight at the position P1 to the line of sight at the position P2.

  As a similar example, a maintenance operation of a server room in which a large number of servers are arranged in a rack can be mentioned. The present invention is suitable for any other similar situation.

  In the example of FIG. 4, the description has been made with the images PC1 and PC2. However, in the case of the user who is provided with the corresponding guidance information G1 and G2, in the case of provision by the display unit 4 including a see-through display, It will be perceived as scenery PC1 and PC2.

  In the following description, unless otherwise specified, the guidance information generated by the generation unit 3 (the information generation unit 36 in FIG. 2) refers to a captured image when the display unit 4 is a normal liquid crystal display or the like. Means that if the display unit 4 is configured with a see-through display, etc., it means that the content is superimposed on the scenery. These shall be explained without particular distinction under certain assumptions. However, in case 3 described above, the guidance information is specialized for the see-through display.

  FIG. 5 is a flowchart of the operation of the AR information display device 20 according to an embodiment.

  First, the overall structure of FIG. 5 will be described. Steps S12, S13, and S21 correspond to steps for distinguishing the above-described case 1 to case 4 by the determination unit 31. And when it corresponds to case 1, a series of steps S14-S18 are performed, when it corresponds to case 2, a series of steps S26-S28 are performed, and when it corresponds to case 3, a series of steps S36-S38 are performed. If it is executed and falls under Case 4, a series of steps S47 to S48 are executed.

  In addition, by applying the flow of FIG. 5 to the captured image at each time captured by the capturing unit 1, it is determined whether the captured image at each time corresponds to Case 1 to Case 4. Appropriate guidance information corresponding to the case to be generated is generated and displayed. In this way, the user who uses the AR information display device 20 is provided with guidance information generated (and updated) in real time by performing real-time imaging. Here, as for the frame rate of shooting by the shooting unit 1 and the rate at which the guidance information is generated and displayed in accordance therewith, a desired setting may be used. Guidance information may be generated and displayed sequentially for all the captured images of the video imaged by the imaging unit 1, or the information obtained by thinning out the rate at a predetermined rate from the video imaged by the imaging unit 1 as the generation target of the guidance information. In addition, guidance information may be displayed.

  In the following, details of each part of the AR information display device 20 (particularly, details regarding generation and display of guidance information) will be described while explaining the steps of FIG. Note that the flow of FIG. 5 is branched from case 1 to case 4 as described in the overall structure above, but the description of each step is in this order (case 1 → case 2 → case 3 → case 4). I will do it.

  In step S11, the photographing unit 1 performs photographing at the current time to obtain a photographed image, and the recognition unit 2 performs the above-described recognition processing on the photographed image at the current time and is stored in the storage unit 5 in advance. The process proceeds to step S12 after obtaining a recognition result by identifying which of the plurality of predetermined objects is being photographed and estimating the position and orientation of the object being photographed.

  In step S12, the determination unit 31 determines whether or not at least one target (a target stored in advance in the storage unit 5) is recognized in the recognition result in step S11. That is, if at least one target has been recognized, the process proceeds to step S13. If the determination is negative, that is, if no recognized target exists, the process proceeds to step S47.

  In step S13, the determination unit 31 determines whether there is a target object among the recognized objects in the recognition result in step S11. If the determination is affirmative, that is, if the target object exists. The process proceeds to step S21. If the determination is negative, that is, if the target target does not exist, the process proceeds to step S14.

  As described above, one of a plurality of predetermined objects stored in the storage unit 5 is set and registered as a target object in advance, so that the determination unit 31 performs step by referring to the registration information. S13 can be determined.

  Hereinafter, Steps S14 to S18 that correspond to Case 1 will be described.

  In step S14, the target estimation unit 32 selects an estimation source target as a foothold for estimating the position of the target target outside the range of the captured image from the targets that have been successfully recognized in step S11. The process proceeds to step S15.

  Various types of embodiments of the estimation source target selection process by the target estimation unit 32 in step S14 are possible as follows.

  In the first embodiment, the distance from the target object calculated by referring to the position coordinates of each target in the predetermined world coordinate system stored in the storage unit 5 is minimized in the world coordinate system. The recognized object can be selected as the estimation source object. For example, if the spatial coordinates of the four directions (four corners) of each object are registered as the position and orientation as described above, a predetermined point (for example, the center of gravity) calculated from the four points is used as the position of each object. Can be referred to as

  In the second embodiment, the error in the position and orientation estimated when the object is recognized in step S11 (for example, a predetermined type of error in numerical calculation when calculating the planar projection transformation matrix) is minimized. The recognized object can be selected as the estimation source object.

  In the third embodiment, the recognized target that maximizes the total score based on the distance in the world coordinate system calculated in the first embodiment and the error calculated in the second embodiment is set as the estimation source target. You can choose. The total score may be calculated using a predetermined evaluation formula that is larger as the distance is smaller and larger as the error is smaller.

  In step S15, the target estimation unit 32 further estimates the position occupied by the target object in the screen coordinates based on the estimation source object selected in step S14, and then proceeds to step S16. Since there is no target object in the photographed image, the estimated position is photographed (unless a large error in position and orientation calculation due to noise or the like has occurred, that is, in a normal case) The position is out of the range occupied by the captured image in the screen coordinates of the image.

The estimation of the screen coordinates of the target object in step S15 may be performed as follows. Here, the expression for the mathematical formula is defined as follows. Assume that the estimation source object is identified by an index i [estimation source], and the target object is identified by an index i [target]. In addition, it is assumed that the planar projection transformation matrix representing the position and orientation of the estimation source object i [estimation source] on the captured image is obtained as an _actual measurement value as H _{(measurement) i [estimation source] in} the recognition processing in step S11. . In addition, the screen coordinates in the captured image of the object i (an index of an arbitrary object is i) are q _i , and the world coordinates registered in advance in the storage unit 5 of the object i are Q _i .

In the first embodiment, by assuming that H _{(actual measurement) i [estimator]} obtained as the actual measurement value directly represents the position and orientation of the target object i [target] as it is, the target object i [target ] Screen coordinates q _{i [target]} can be obtained as follows.
q _{i [target]} = H _{(actual measurement) i [estimator]} Q _{i [target]}

In the second embodiment, the above approximate application is not performed, and the screen coordinates q _{i [target]} of the target object i [target] can be obtained as follows.
q _{i [target]} = T (i [estimator] → i [target]) H _{(actually measured) i [estimator]} Q _{i [target]}
Here, T (i [estimation source] → i [target]) is the coordinate Q _{i [estimation} ] of the estimation source target i [estimation source] in the world coordinates registered in advance in the storage unit 5 as follows. _{The original]} can be obtained as a matrix for converting the coordinates Q _{i [target ]} of the target object i [target].
Q _{i [target]} = T (i [estimator] → i [target]) Q _{i [estimator]}

  In step S16, as various information necessary for the information generation unit 36 to generate guidance information in the next step S17, the start point determination unit 34 determines the start point, and the end point determination unit 33 determines the end point. To step S17.

  In step S16, the start point determination unit 34 can determine the start point whose position is defined on the two-dimensional display area by the display unit 4 in any of the following embodiments.

In the first embodiment, the distance from the estimation source target camera (camera as hardware constituting the photographing unit 1) selected in step S14 is γ, the user's line-of-sight distance is γ0, and a predetermined threshold for determination is set. As α0, the starting point can be determined according to the following cases (1) and (2). The distance γ can be obtained from the position and orientation obtained by the recognition unit 2 with respect to the estimation source object. Further, the value of the line-of-sight distance γ0 may be stored as a fixed value in the storage unit 5, or the average value of the distances obtained from the objects recognized by the recognition unit 2 and the position and orientation are obtained, or the recognized objects Of these, the distance obtained for the object closest to the center on the captured image may be used.
(1) When | γ−γ0 | ≦ α0, the viewpoint (the viewpoint obtained in the screen coordinate system) is set as the starting point. Here, the viewpoint may be set in advance as a predetermined point (for example, the center) in the display area of the display unit 4, or is acquired by the line-of-sight sensor when the line-of-sight sensor can be used when using an HMD or the like. It may be a different position.
(2) If | γ−γ0 |> α0, the starting point is the midpoint of the recognized object (midpoint in the screen coordinate system) at the shortest distance from the viewpoint in the world coordinate system. Here, the midpoint of the object may be set in advance as a predetermined point (for example, the center of gravity) in the area occupied by the object. As for the viewpoint position in the world coordinate system, the fixed position may be stored in the storage unit 5 as in the case of the line-of-sight distance γ0 described above, or the recognition unit 2 recognizes the position and orientation. You may use the average value of the position obtained in each object, or the world coordinate position obtained in the object at the screen coordinate position closest to the center on the captured image among the recognized objects.

  That is, in the first embodiment, in the case of (1) and (2), the position in the world coordinate system of the estimation source object that is an object close to the target object is near and far from the position in the world coordinate system of the line of sight Dividing. If the distance is close, it is determined that the line-of-sight position is useful as a guide for guiding the user as it is (start point of the arrow). By setting to, guidance is made more reliable.

In the second embodiment, the above case classification is not performed, and the start point can be determined according to any of the following settings (3) to (7).
(3) The setting in the case of (1) of the first embodiment is always adopted. That is, the viewpoint is the starting point. With respect to the viewpoint, the same predetermined point as described in the first embodiment or a point obtained by a line-of-sight sensor may be used.
(4) The setting in the case of (2) of the first embodiment is always adopted. That is, the midpoint of the recognized object at the shortest distance from the viewpoint in the world coordinate system is set as the starting point. The midpoint is a predetermined point similar to that described in the first embodiment, and the same applies to the following description.
(5) The midpoint of the estimation source target selected in step S14 is set as the start point.

  (6) Among the objects recognized in step S12, the middle point of the object having the highest saliency due to the difference in features from the surrounding area is set as the starting point. Here, regarding the calculation of the saliency, an existing method in the image processing field such as comparison of average colors, color histograms, shapes and orientations may be used. In this embodiment, it is assumed that the recognition unit 2 performs the calculation of the saliency in the above step S12 as an additional process for the recognized object.

  (7) For each target i, the priority degree εi is registered in the storage unit 5 as described above, and among the targets recognized in step S12, the middle point of the target with the highest priority εi is set as the starting point. . The priority level may be registered in the storage unit 5 as a table in advance as a table reflecting the user attributes (for example, by giving a higher priority level to an object familiar to the user). What is automatically generated or corrected using the operation history indicating the guidance effect may be registered in the storage unit 5.

  In the calculation of the saliency in the above (6), the saliency may be calculated by referring to the same priority level εi as used in (7). That is, according to (6), when calculating the saliency of the object due to the difference in features from the surroundings, the same priority degree as that registered for each object i in the recording unit 5 in (7) By referring to ε i, the priority degree ε i may be used as a weight when calculating the saliency of each object.

  The use of the priority order reflecting the above-described user attributes is not limited to the situation where the user to be guided exists at the site where the photographing unit 1 is photographing as shown in the schematic example of FIG. It is also suitable in a situation where guidance information is provided to a remote user who exists and plays a role of giving an instruction to a shooting user on site. That is, from the standpoint of a remote user, the viewpoint of the shooting user is not always important, and it may be preferable to set a starting point that reflects the attributes of the remote user.

  In step S16, the end point determination unit 33 can determine the end point whose position is defined on the two-dimensional display area by the display unit 4 in any of the following embodiments.

  In the first embodiment, a straight line (directional half line) extending from the start point determined by the start point determination unit 34 to the target target screen coordinates estimated by the target estimation unit 32 in step S15 is displayed. The position that intersects the boundary (outer frame portion) of the display area by the section 4 can be determined as the end point.

  In the second embodiment, it is possible to determine, as an end point, a position that returns to the inside of the display area on the straight line by a predetermined amount from the intersecting position obtained in the first embodiment.

  In the third embodiment, the midpoint of the estimation source target selected in step S14 can be the end point. In this case, it is assumed that the start point determination unit 34 applies other than the embodiment (5) so that the end point does not coincide with the start point. In addition, the third embodiment is a premise embodiment that serves as a general indication that the position of the estimation target object is directed to the position of the target object outside the display area due to the presence of a large number of objects. In the embodiment, it is preferable to adopt an embodiment in which the start point is determined approximately at the center of the display area.

  In the fourth embodiment, the midpoint of the object recognized by the recognition unit 2 that is closest to the end point determined by the first or second embodiment on the screen coordinates can be determined as the end point. Here, the distance (the distance on the screen coordinates) between the end point determined by the first embodiment or the second embodiment and the object recognized by the recognition unit 2 that is closest to the end point is a predetermined threshold value. The fourth embodiment may be applied to the following cases, and if it is larger than the threshold, the end point determined in the first embodiment or the second embodiment may be adopted instead.

  In step S17, the information generation unit 36 generates guidance information as an arrow superimposed on the captured image or landscape, and then proceeds to step S18. Here, the arrows constituting the guidance information can be generated by setting the start point and the end point determined in step S16 as the start point and end point of the arrow, respectively. The generated arrow information may be stored in advance in the storage unit 5 as shape model information or the like that automatically determines the shape of the arrow when a start point and an end point are given. In step S18, the display unit 4 displays the generated guidance information as an arrow superimposed on the captured image or landscape.

  With respect to generation and display of guidance information in steps S17 and S18, the following embodiments are possible.

  In 1st embodiment, you may make it produce | generate and display guidance information including the information which tells a user that the present condition corresponds to case 1 besides the arrow. For example, although the target object has not yet been reached, information indicating that the target object is closer to the direction of the arrow may be given as text information, or the direction of the arrow to be displayed (case 2) (Different) may be a predetermined mode.

  6 and 7 are diagrams for explaining a display mode of an arrow in the second embodiment. As shown in FIG. 6, in the real world W, a plurality of objects O11 to O63 similar to the example of FIG. 4 are arranged differently from the example of FIG. 4, that is, a part thereof is arranged on the front surface PR1 of the shelf R1. The remaining part is disposed on the front surface PR2 of the shelf R2 that is further away from the shelf R1 in the depth direction D3 when viewed from the user U taking the image at the position P3.

  Images PC3 taken at position P3 in FIG. 6 are shown in FIG. 7 as [1] to [4], respectively. In [2] to [4], examples of arrow display modes according to the second embodiment are arrows A32, Shown as A33, A34. Note that in the image PC3, drawing of objects other than the objects O11 to O63 such as the boundaries of the shelves R1 and R2 is omitted.

  That is, the second embodiment is an embodiment that is possible based on the embodiment in which the determination of the start point and the end point is determined as the midpoint of any target, and the distance from the camera position of the target determined as the start point Depending on whether or not the difference (absolute value) from the target camera position determined as the end point is within a predetermined threshold, the display mode of the arrow is changed. As is well known, since the information on the distance from the target camera position is known when the position and orientation of the target is estimated as a homography matrix in the recognition unit 2, the information on the known distance is referred to. Thus, it is possible to determine whether or not it is within the predetermined threshold.

  An arrow A32 shown in [2] in FIG. 7 and an arrow A33 shown in [3] display a thin line arrow as the first mode because the distance difference is within the threshold value. That is, since the arrow A32 has a start point at the midpoint of the object O32 and an end point at the midpoint of the object O22, both are located on the front surface PR1 in FIG. Similarly, the arrow A33 has a start point at the midpoint of the object O43 and an end point at the midpoint of the object O52, both of which are on the front surface PR2 in FIG. On the other hand, the arrow A34 shown in [4] of FIG. 7 displays a thick line arrow as the second aspect because the distance difference exceeds the threshold value. That is, the arrow A34 has the start point at the midpoint of the object O32 and is on the front surface PR1 in FIG. 6, the end point is at the midpoint of the target O42 and is on the front surface PR2 in FIG. Since one side is on the front surface PR2 that is separated in the depth direction D3, the distance difference is large.

  In the second embodiment, that is, when the distance from the camera position changes greatly between the start point and the end point of the arrow, the direction of depth or the like is displayed to the user by displaying the arrow in distinction from the case where the change is small. It is possible to effectively guide the user to be aware of the presence or absence of a distance change. In the embodiment in which at least one of the start point and the end point is not determined as the midpoint of the target, the start point or the end point is determined by the recognized world coordinate of the target closest to the at least one of the start point or the end point on the screen coordinates. The position from at least one camera may be calculated as an approximate value, and the second embodiment described above may be applied.

  FIG. 8 is a diagram for explaining the third embodiment. In FIG. 8, the same objects O11 to O63 as in FIG. 4 are arranged on the shelf R in the real world W as in FIG. 4, and the user located in front of the shelf takes a picture facing the front of the shelf. Thus, it is assumed that a captured image PC4 (or scenery PC4) as shown in [1] in FIG. 8 is obtained. (However, it is assumed that the actual captured image can be captured only in a part of the area shown in the PC 4 as in FIG. 4. That is, the PC 4 is assumed to be a virtual panoramic image.) FIG. The target object is assumed to be the object O61 at the upper right end.

  In this case, if an arrow is displayed in the same manner as described above without applying the third embodiment, the arrow as shown in [2] is sequentially provided as guidance information to the user according to the manner of shooting by the user. That is, first, an arrow A41 from the target O13 to the target O32 is provided to the user who is shooting the lower left area of the shelf, and the user guided by the arrow A41 moves the shooting position to the upper right side, and then Is provided with an arrow A42 from the target O32 to the target target O61. In this way, the user can reach the target object O61.

  However, when the objects O11 to O63 are arranged in a grid pattern, that is, in a case where the objects are arranged in a grid pattern, an arrow that crosses diagonally as in [2] Can somewhat confuse the user's intuitive guidance to the target. In particular, when the distance between the objects is wide or there are a certain number of objects or more, the confusion is considered to become more prominent.

  Therefore, in the third embodiment, when the arrow to reach the target object O61 from the target O13 that is recognized at a certain time and whose midpoint is set as the start point is inclined as shown in [2], Instead of this, arrows A43, A44, and A45 that linearly advance on the lattice points can be sequentially given as in [3]. That is, the arrow A43 extends from the target O13 to the target O43 in the substantially horizontal and horizontal direction, the arrow A44 extends from the target O43 to the target in the horizontal and horizontal direction O63, and the arrow A45 corresponds to the vertical and vertical direction from the target O63. In comparison with [2], guidance information that is easier for the user to grasp intuitively is realized. In addition, when applying 3rd embodiment, it is preferable to employ | adopt embodiment which determines the embodiment which determines a start point and an end point as a midpoint of either object.

  In the third embodiment, information on the positional relationship between the objects i is given in advance to the storage unit 5, and at a certain point, when the center point reaches the target object linearly from the target set as the starting point, [2 ], If it is determined that a “diagonal” arrow is generated, correct it to a combination of two horizontal and vertical arrows that do not become “diagonal” and display either of them as guidance information Is repeated so that the user reaches the final target. At this time, the end point of the arrow displayed at the nth repetition may be the start point of the arrow displayed at the next n + 1th time.

  In the third embodiment, after the start point determination unit 34 determines the start point by any one of the methods described above (the method of the first embodiment or the method of any one of (2) to (7) of the second embodiment), Guidance information can be generated using horizontal and vertical arrows.

  FIG. 10 is a diagram illustrating an example of arrangement and the like for explaining generation of the guidance information. As illustrated, it is assumed that the same objects O11 to O63 as those in FIG. 4 and the like are arranged, the midpoint of the object O23 is determined as the start point, and the target object is O61. Here, as shown in FIG. 10, the position is described with coordinates (x, y) such that the + x direction is rightward and the + y direction is upward.

  Specifically, in the third embodiment, for example, guidance information by a horizontal arrow and a vertical arrow can be generated as follows. The midpoint position of the target at the start point (target O23 in the example of FIG. 10) is (i0, j0), and the midpoint position of the target target (target O61 in the example of FIG. 10) is (i1, j1). If there is a relationship of i1 ≥ i0 and j1 ≥ j0, when the horizontal arrow is displayed first, the position of the horizontal arrow start point is (i0, j0), and the horizontal width of the display area Is Sw, if i0 + Sw / 2> i1, the end point position is (i1, j0). If i0 + Sw / 2 <= i1, the end point position is ((i0 + Sw / 2 ), j0). By repeating this (including the case where the number of repetitions is 1), that is, by repeating the movement of the line of sight by Sw / 2 in the x-axis direction, (i1, j0) appeared in the user's field of view. At this time, control is performed to change the arrow to be displayed from the horizontal direction to the vertical direction. The starting point of the vertical arrow at the moment of change is (i1, j0), and if the vertical width of the display area is Sh, if j0 + Sh / 2> j1, the end point is (i1, (j0 + Sh / 2 )), And if j0 + Sh / 2 ≦ j1, the end point is assumed to be (i1, j1) and may be repeated in the same manner as in the horizontal direction. In the above description, it is assumed that the starting point (i0, j0) is at the center of the display area. (Therefore, the end point of an arrow that moves more than Sw / 2 in the x-axis direction or more than Sh / 2 in the y-axis direction from the start point is outside the display area. Such arrows cannot be displayed.) When (i0, j0) exists out of the center of the display area, the arrow is displayed taking into account the movement for correcting the deviation from the center when the arrow is displayed for the first time. After that, it may be the same as described above.

  As is clear, the start point and end point positions (x, y) used in the description are fixed virtual screen coordinates (in the photographing unit 1) for expressing the arrangement of a plurality of objects in the real world. In addition to the acquired captured image, it means a position in a screen coordinate) when considering a sufficiently large captured image in which all of a plurality of objects are generally captured from the front. The position (x, y) is a different concept from the screen coordinates (where the origin moves in the real world as the user viewpoint moves) in the captured image acquired by the imaging unit 1. That is, in the screen coordinates on the captured image, for example, even if the starting point is a position (i0, j0) at a certain time, the position is (i0-Δx, j0-Δy) when the camera is moved by (+ Δx, + Δy). ), But in the above description, not the screen coordinates that move with the camera in this way, but the fixed virtual screen coordinates that the starting point is (i0, j0) regardless of the camera position Is used for explanation. Such virtual enemy screen coordinate information can be calculated from position coordinates of each target in the world coordinate system registered in the storage unit 5.

  In the above example, the horizontal arrow is first displayed repeatedly so that (i1, j0) appears in the field of view, and then the vertical arrow is displayed repeatedly to the target target (i1, j1). Similarly, the vertical arrow is first displayed repeatedly so that (i0, j1) appears in the field of view, and then the horizontal arrow is repeatedly displayed to display the target object ( i1, j1) may be reached. Whether to display the horizontal or vertical arrow first may be given a fixed setting in advance, or the positional relationship between the start point (i0, j0) and the end point (i1, j1), etc. Accordingly, it may be determined according to a predetermined rule. For example, if | i0-i1 | ≧ | j0-j1 |, that is, if the rectangle with the start point (i0, j0) and the end point (i1, j1) as one diagonal is a horizontally long shape, The predetermined rule (or the reverse predetermined rule) may be used such that the arrow is displayed first, and if not (when the rectangle is vertically long), the vertical arrow is displayed first. In the present embodiment, it is preferable that the positional relationship between the plurality of objects registered in the storage unit 5 is generally defined in a lattice point so that the movement in the horizontal direction and the vertical direction can be defined.

Alternatively, the determination may be performed as follows.
(1) When the distance between the recognized object and the target object is more than the predetermined value α1, or (2) When there are more than α2 objects between the recognized object and the target object, It is determined that the combination is the case of “horizontal” + “vertical”. Here, the number of objects between the recognized object and the target object in (2) is the number of objects and target objects that can be recognized on the overhead view stored in advance in the storage unit 5 in the embodiment described later. A straight line may be drawn between the two and the number of objects through which the straight line has passed may be obtained, or the number of objects between any two objects may be registered in the storage unit 5 in advance. The distance in (1) can also be calculated using information on the position of each target in world coordinates registered in the storage unit 5 in advance. The “recognized target” in the above (1) and (2) may be a target for which the start point at the time described with reference to FIG. 10 is set.

  As described above, steps S14 to S18, which are cases corresponding to case 1, have been described.

  In step S21, the determination unit 31 determines whether or not the target object determined to be present in step S13 is within the display area of the display unit 4 in the recognition result in step S11. If so, that is, if the target object exists inside the display area, the process proceeds to step S26, and if the determination is negative, that is, if the target object does not exist inside the display area, the process proceeds to step S36.

  Hereinafter, Steps S26 to S28 that correspond to Case 2 will be described.

  Since Steps S26, S27, and S28 of Case 2 are the same as Steps S16, S17, and S18 of Case 1, overlapping descriptions are omitted. However, since the target object is recognized in the case 2 and exists in the display area, the following processing different from the case 1 can be performed.

  First, in step S26, the end point determination unit 33 may determine the midpoint of the target object that has already been recognized as the end point. In addition, the determination of the start point by the start point determination unit 34 may be omitted, or the start point may be omitted when the determined start point is the midpoint of the target object.

  In addition, in the generation of guidance information by the information generation unit 36 and the display by the display unit 4 in steps S26 and S27, the target target set as the end point is the target as an additional process for the case 1 You may make it perform the display which shows. For example, text information indicating that it is the target target may be superimposed on the midpoint of the target target determined as the end point. In addition, an emphasis display or the like may be given using an icon such as a thick frame B2 superimposed on the target object O62 in the example of FIG.

  In addition, when the determination of the start point by the start point determination unit 34 is omitted, the arrow display may be omitted, and the guidance information may be configured only by information indicating that the target is the target as described above.

  Hereinafter, Steps S36 to S38, which are cases corresponding to Case 3, will be described.

  Since Steps S36, S37, and S38 of Case 3 are the same as Steps S16, S17, and S18 of Case 1, overlapping descriptions are omitted. However, since the target object is recognized and exists outside the display area in case 3, the following processing different from the case 1 can be performed.

  First, in step S36, the end point determination unit 33 is already recognized (not the target target screen coordinates estimated by the target estimation unit 32 in step S15 in case 1) when applying the first embodiment described above. If a line segment from the start point determined by the start point determination unit 34 is drawn to the midpoint of the target target, and the point where the line segment intersects the boundary (outer frame) of the display area by the display unit 4 is determined as the end point Good. Similarly, when the second embodiment described above is applied, a position where a predetermined amount is returned from the intersection to the inside of the display area on the line segment may be set as the end point. Further, in case 3, since the estimation source target obtained in step S14 in case 1 does not exist, the third embodiment described above may be excluded as an embodiment of the end point determination unit 33 in step S36.

  FIG. 9 is a diagram showing an example of an arrow in case 3 in [1] and an example of an arrow in case 2 in [2]. In [1], the target O2 in the display area R4 is set as the start point in the same configuration as in FIG. 3, but the target object O3 is located outside the display area R4 and in the imaging area R1, The arrow A5 can be displayed only up to the boundary of the display area R4, and the guidance information G5 is given. On the other hand, unlike [1], the shooting area R1 and the display area R4 are in the same relationship in [2], so that an arrow with the target O2 as the start point and the target object O3 as the end point can be displayed. Thus, guidance information G6 is given.

  In addition, in the generation of the guidance information by the information generation unit 36 and the display by the display unit 4 in steps S36 and S37, the target object already exists in the field of view (in the captured image) as an additional process for the case 1. You may make it display that it is. For example, text information indicating that the target object exists in the field of view may be superimposed on the end point or the vicinity thereof.

  Hereinafter, Steps S47 and S48 that correspond to Case 4 will be described.

  In step S47, the information generation unit 36 generates text information indicating that the object is not recognized at all as the guidance information, and in step S48, the display unit displays the information. In this way, it can be generated as guidance information that the object is not recognized at all, and the user can be urged to review the shooting location. Alternatively, steps S47 and S48 may be skipped so that the guidance information is not generated and displayed. Also in this case, since the guidance information is not displayed at all, it is possible to prompt the user to review the shooting location.

  As described above, according to the present invention, it is possible to eliminate the difficulty in recognizing a target object from a plurality of objects such as products and the high cost problem when using the sensor, and even if the target object cannot be recognized, It is possible to estimate the position of the target object on the display screen from the position and orientation information of the object that can be recognized, and provide guidance information for guiding to the target product or the like.

  Here, not only GPS and other special position sensors, but also using cameras and image recognition AR technology to recognize objects, not only reducing installation costs, but also accurate and detailed guidance information corresponding to changes in the situation Can be provided. In navigation using a special position sensor, the position of the user is used as a reference. However, according to the present invention, the position of the user's line of sight and the object that the user can recognize in the field of view according to the user's situation and work purpose. By providing the guidance information based on the position, guidance information appealing to the user's intuition can be provided.

  Hereinafter, additional embodiments of the present invention and other supplementary explanations will be given.

  (1) In steps S16 to S18 (and corresponding steps of cases 2 and 3) in case 1 of FIG. 5, in the above embodiment, the same applies to the start point and end point (only the end point in the embodiment and in some cases). ) And the guidance information was realized by directly superimposing an arrow on the photographed image or landscape. Instead, the start point and the end point are determined on the overhead view in which the arrangement of a plurality of targets is displayed as a model as a two-dimensional map, and exactly the same arrows and various messages (that the target object has been reached) May be displayed. In other words, instead of using a captured image or landscape in the above embodiment, an overhead view may be used and the guidance information may be configured by superimposing an arrow or the like thereon. When the overhead view is used, the user can grasp not only the field of view area but also the state of the entire object, so that the user can easily grasp the position of the target object.

  That is, in the embodiment using the overhead view, determination of an arrow or the like for performing the superimposed display can be performed in the same manner as the above embodiment, and the guidance information is generated and displayed in a form in which the superimposed display is performed on the overhead view. The only difference is that However, as an embodiment that considers using an overhead view, the current state holding unit 35 may be applied so that the user's recognition area can be identified on the overhead view. In addition, the embodiment for determining the start point and the end point can apply the embodiment for determining as the midpoint of any target, so that the start point and end point of the arrow can be set at the position of the target that is predefined on the overhead view. It is preferable to do so.

  FIG. 11 is a diagram illustrating a schematic example of an embodiment in which guidance information is configured by a bird's-eye view corresponding to the example of FIG. As shown on the upper side, overhead view information OV0 obtained by modeling the arrangement of the objects O11 to O63 in a two-dimensional arrangement is created in advance and registered in the storage unit 5. In the example of FIG. 4, the model can be given as a planar arrangement model when the shelf R on which the objects O11 to O63 are arranged is viewed from the front, and can be registered in the storage unit 5. Then, as shown on the lower side, the guidance information G10, G20 (as another embodiment corresponding to the example of FIG. 4) as shown on the lower side by the information OV1, OV2 displaying an arrow etc. on the overhead view. Guidance information) can be realized. That is, the guidance information G10 configured in the overhead view OV1 of FIG. 11 includes an arrow having the middle point of the target O22 as the start point and the middle point of the target O42 as the end point, like the guidance information G1 in FIG. Further, the guidance information G20 configured in the overhead view OV2 of FIG. 11 includes an arrow having a middle point of the target O42 as a start point and a middle point of the target O62 as the end point, similar to the guidance information G2 of FIG. It is highlighted with an icon indicating that it is a target object.

  In the superimposed bird's-eye view, the target that has entered the field of view and can be recognized at the present time (the target that has been shot by the shooting unit 1 and has been successfully recognized by the recognition unit 2) is displayed separately from other targets. In this way, the user may be able to grasp where the current field of view is on the overhead view. In order to enable the display to be grasped, the current status holding unit 35 holds the target that has been successfully recognized by the recognition unit 2 with respect to the current captured image, and updates it in real time. As an example of guidance information as an overhead view generated by the information generation unit 36 in response to the update result by the current state holding unit 35, in the example of the information OV1 and OV2 in FIG. The recognized object is displayed as a white display in distinction from the gray display of the unrecognized object.

  In particular, FIG. 11 shows an example of a situation in which not all objects in the captured image are necessarily recognized. That is, there are 12 objects O11 to O43 in the photographed image PC1, but in the information OV1, only half of them O21, O31, O22, O32, O23, O33 are recognized as having been successfully recognized. It is displayed. In addition, there are 12 objects from O31 to O63 in the captured image PC2, but in the information OV2, only half of them, O41, O51, O42, O52, O43, and O53, are recognized as being white. It is displayed.

  In the present invention, guidance information can be displayed even when the recognition unit 2 does not recognize some of the objects. Although the example of FIG. 11 is an example of an overhead view, even when the arrow is superimposed on the captured image or the field of view, the guidance information is displayed even if the recognition unit 2 can recognize only a part of the objects. Is possible.

  In addition to the mode of displaying each of the unrecognized objects in gray as in the example of FIG. 11, the entire area where the target is not recognized on the overhead view is displayed in gray, or similarly, The information generation unit 36 may generate the guidance information so that the entire region is displayed in a form covered with a mesh or the like. FIG. 12 shows a schematic example displayed in a form covered with a mesh. In FIG. 12, in [1] and [2], the objects that cannot be recognized in the overhead view OV1 and OV2 in FIG. 11 are displayed in gray instead of being displayed in gray. A schematic example is shown in which a target that is not covered is displayed with a mesh. In addition, any display method for distinguishing the recognized object and the unrecognized object (the object held by the current state holding unit 35 and the other object) on the overhead view may be used.

  In the embodiment in which the guidance information is configured by the overhead view, it is possible to display the arrangement of all objects on the overhead view, unlike the captured image and the actual landscape. As a change from the embodiment in which an arrow or the like is superimposed, the end point determination unit 33 may always set the end point as the midpoint of the target object. Thus, when the information generation unit 36 generates an arrow on the overhead view, the target object is always set as the end point of the arrow regardless of whether the end point exists within the user's view range (the range of the captured image). be able to. In this case, guidance information may be generated by expressing that the end point of the arrow is the target object.

  FIG. 13 is a diagram illustrating a schematic example of an embodiment in which the end point is always determined as the midpoint of the target object. In place of the guide information G10 and G20 in the embodiment of FIG. 11, guide information G15 and G25 as shown in [1] and [2] of FIG. 12 are generated in this embodiment. That is, in the guidance information G15 in [1] of FIG. 13, the target O62 outside the user's field of view is indicated by the thick frame display and is set as the end point of the arrow.

  (2) An embodiment in which the superimposition of an arrow or the like on the above-described overhead view and the superimposition of an arrow or the like on a captured image or a landscape is combined or switched according to the situation is possible. For example, when the distance of the recognized target that has the smallest distance from the camera exceeds the threshold (when the target is determined to be generally far), an overhead view to prompt the user to grasp the overall arrangement relationship You may make it superimpose on. When the distance is less than or equal to the threshold value, either a bird's eye view or direct superimposition or either of them may be applied by user selection. For example, as an example of presetting, direct superimposition (or a combination of this and a bird's-eye view) is used to prompt the user to understand the local positional relationship that the user is viewing when the distance falls below a threshold value. You may do it.

  The distance for performing the switching determination may be the distance of the object recognized as described above that has the minimum distance to the camera, or another distance may be adopted. For example, it may be the average of the distances between all recognized objects and the camera, or in case 1 (by steps S14 and S15 in FIG. 5) between the target object and the camera estimated based on the estimation source object. It may be a distance, or may be a distance between the target object whose position is directly obtained in cases 2 and 3 and the camera. Further, although the above distance is the distance in the world coordinates, the distance in the screen coordinates may be used instead.

  Furthermore, in the embodiment in which the guidance information is generated by combining the overhead view and the captured image, the target on the overhead view and the target on the captured image (including the field of view in the case of HMD; the same applies hereinafter) are associated with each other. In addition, icons may be superimposed at corresponding positions. In one embodiment, the position where the icon is superimposed can be determined as the closest to the position of the line of sight in the screen coordinates or the camera position in the world coordinates among the objects recognized on the captured image, and also corresponds to the overhead view. If the icon is superimposed on the position, the user does not have to hesitate to determine which object in the overhead view corresponds to the object in the field of view. That is, an icon is displayed as being the same target in both the overhead view and the captured image for a certain target, so that the user can intuitively grasp it. Moreover, you may use not only superimposition of an icon but the other arbitrary display modes which make the said object identifiable on a bird's-eye view and a picked-up image.

  Further, the determination of the predetermined target on which the icon or the like is superimposed may be an embodiment other than the determination as the target closest to the line of sight as described above, and the target midpoint is determined when the start point is determined by the start point determination unit 34. May be determined as the target in the embodiment determined as (for example, the target having the greatest saliency). Further, any one of the objects determined by the recognition unit 2 may be used. Furthermore, the icon or the like may be superimposed on two or more objects, but in this case, it is possible to distinguish between the icon on the overhead view and the icon on the captured image. Is preferably displayed. That is, it is preferable to give the icon and the like by changing the display mode so that the target ID can be identified. The icon and other information to be used may be registered in the storage unit 5 in advance.

  (3) In the above description, the guidance information is composed of arrows from the start point to the end point. However, the present invention is not limited to the arrow, and the start point (or the vicinity thereof) and the end point (or the vicinity thereof) can be arbitrarily recognized by the user. The guidance information may be configured according to the display mode. For example, an icon or other superimposition may be performed at the start point, and an icon or other superimposition may be performed at the end point. When performing display connecting the start point and the end point, any mode other than the arrow may be used. For example, a straight line (or elongated rectangle) may be drawn from the start point to the end point, and an animation display in which the pattern moves from the start point to the end point on the straight line may be given. In the embodiment in which the start point and / or the end point are set as the midpoint of the target, the start point and / or the end point can be obtained by making the set target region visible, for example, by surrounding it with a frame. May be displayed. The guidance information may be configured by combining making the region corresponding to the start point and / or the end point visible and making the relationship between the start point and the end point recognizable by the arrow or the like.

  (4) The present invention can also be provided as a program that causes a computer to function as the AR information display device 20. In this case, the AR information display device 20 of the present invention is configured by a computer having a general configuration including a CPU, a primary memory that provides a work area for the CPU, a secondary storage device that stores predetermined data, programs, and the like. 1 and 2 can be realized by a CPU that reads and executes a predetermined program corresponding to each function. In addition, any part or all of the units shown in FIGS. 1 and 2 may be implemented by dedicated hardware (such as a dedicated LSI) instead of being implemented by a general-purpose CPU executing a program. May be.

  (5) In the present invention, when all of the objects registered in advance in the storage unit 5 are generally arranged on one plane as in the example of FIG. 4, the objects are grouped as in the example of FIG. Divided (divided into target groups on shelf R1 and target groups on shelf R2), realizing a particularly suitable guidance display when objects belonging to each group are generally arranged in one plane However, the present invention can be applied even when there is no such restriction and the arrangement of each object is arbitrary in the three-dimensional space.

  (6) In the present invention, the position on the screen coordinates is obtained with respect to the start point and the end point for generating the arrows constituting the guidance information, and the arrows represent the two-dimensional directions on the screen coordinates. However, instead of this, a three-dimensional display arrow used in the existing AR technology or CG (computer graphic) technology may be used. In this case, the embodiment in which the start point and the end point are determined by the midpoint of the recognized object is adopted, and the position in the world coordinates of the midpoint of the object is used, so that the 3D including the representation in the depth direction A display arrow may be generated.

  20 ... AR information display device, 1 ... shooting unit, 2 ... recognition unit, 3 ... generation unit, 4 ... display unit, 5 ... storage unit

Claims (20)

An AR information display device that performs display that leads to a predetermined target object from a plurality of objects,
A shooting unit for shooting and obtaining a shot image;
A recognition unit for recognizing each of the plurality of objects from the captured image;
A generating unit that generates guidance information to the target target based on the target recognized by the recognition unit when the target target is not recognized by the recognition unit;
An AR information display device comprising: a display unit configured to display the guidance information. Each of the plurality of objects has a registered position and orientation in a common world coordinate system,
In the recognition unit, when recognizing each of the plurality of objects, estimation of the position and orientation captured in the captured image is also performed with respect to at least one of the recognized objects,
In the generation unit, one of the objects recognized by the recognition unit and the registered position and posture in the target object, and a position and posture estimated by the recognition unit with respect to the one object, The AR information display device according to claim 1, wherein the guidance information is generated based on the information.   3. Each of the plurality of objects includes registering the position and orientation in a common world coordinate system by registering a spatial coordinate of a predetermined point in each object. AR information display device.   The generation unit generates the guidance information to the target target based on the target recognized by the recognition unit as information for displaying that the guidance information is guided to the recognition position or the estimated position of the target target. The AR information display device according to any one of claims 1 to 3,   The generation unit determines a start point and an end point based on an object recognized by the recognition unit, and generates guidance information to the target object as information indicating that the start point is reached to the end point. An AR information display device according to any one of claims 1 to 4. In the generation unit,
One of the objects recognized by the recognition unit is determined as an estimation source object,
The starting point is
When the difference between the estimation source target and the camera constituting the photographing unit (γ) and the user's line-of-sight distance (γ0) is within a threshold, the predetermined user viewpoint on the photographed image is Decide
6. The AR information display device according to claim 5, wherein if the difference is not within a threshold value, the AR information display device is determined as a midpoint of the object recognized by the recognition unit at the shortest distance from the viewpoint in the world coordinate system. .   In the generation unit, the start point is determined as a predetermined user viewpoint on the photographed image, or as a midpoint of an object recognized by the recognition unit at a shortest distance from the viewpoint in the world coordinate system. The AR information display device according to claim 5.   The AR information display apparatus according to claim 5, wherein the generation unit determines the start point as a midpoint of a target having the highest saliency among the targets recognized by the recognition unit.   6. The AR according to claim 5, wherein the generation unit determines the start point as a midpoint in the highest priority degree registered in advance among the objects recognized by the recognition unit. Information display device. In the generation unit,
Of the objects recognized by the recognition unit, the one closest to the target object in screen coordinates is determined as an estimation source object,
10. The AR according to claim 5, wherein the screen coordinates of the target object are estimated based on the estimation source object, and the end point is determined based on the estimated screen coordinates. Information display device.   The AR information display apparatus according to claim 10, wherein the generation unit determines the end point on a line segment from the start point to the screen coordinate based on the estimated screen coordinate. In the generation unit,
If it is determined that a straight line from the starting point to the estimated screen coordinates crosses a plurality of objects diagonally,
The information indicating that the start point is reached and the end point is generated as being displayed divided into a plurality of times, and the information indicating that the horizontal direction and the vertical direction are sequentially displayed is displayed. 11. The method according to claim 10, wherein the information is generated by determining whether to display in the horizontal direction or the vertical direction first, and a start point and an end point for the display. The AR information display device described. In the generation unit,
Depending on whether or not the difference between the distance between the start point and the camera constituting the photographing unit and the distance between the end point and the camera constituting the photographing unit is within a predetermined threshold, the start point is reached to the end point. The AR information display device according to claim 5, wherein the information is generated by changing a display mode of information for displaying the effect. In the generation unit,
Based on the distance between the start point and the target object, or based on the number of objects existing between the start point and the target object, a display mode of information indicating that the start point is reached to the end point 12. The AR information display device according to claim 5, wherein the AR information display device is generated by changing.   The AR information display device according to claim 1, wherein the generation unit generates the guidance information as information associated with the captured image.   In the generation unit, the guidance information is used as information to be superimposed on the photographed image or a user's field of view corresponding to the photographed image and / or on a bird's-eye view expressing a predetermined arrangement relationship of the plurality of objects. The AR information display device according to claim 15, wherein the AR information display device generates the information to be superimposed.   When the target unit is recognized by the recognition unit, the generation unit generates the guidance information as information expressing that the target target already exists in the captured image. The AR information display device according to claim 1. The generating unit superimposes the guidance information on the captured image or first information to be superimposed on a user's field of view corresponding to the captured image, and an overhead view expressing a predetermined arrangement relationship of the plurality of objects. And generating the second information, and
The first information is generated so that the predetermined object recognized by the recognition unit in the field of view of the user corresponding to the captured image or the captured image can be identified, and the second information is displayed on the overhead view. The AR information display device according to claim 1, wherein the predetermined object is generated as being identifiable.   In the generation unit, a position is estimated based on a distance between a camera constituting the imaging unit and a target recognized by the recognition unit or a camera constituting the imaging unit and a target recognized by the recognition unit. Depending on the distance to the target object, the guidance information is superimposed on the captured image or information that is superimposed on the field of view of the user corresponding to the captured image or an overhead view that represents a predetermined arrangement relationship of the plurality of objects. 18. The AR information display device according to claim 1, wherein the information is generated by switching to one or both of the information to be processed.   A program causing a computer to function as the AR information display device according to any one of claims 1 to 19.