JP2017058657A - Information processing device, control method, computer program and storage medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve visibility of users upon conducting a projection display of presentation information onto objects.SOLUTION: In an information processing device provided with an imaging device and a display device, a marker ID and position of a marker included in a shot image obtained through the imaging device are recognized to determine whether one or more markers are included in the shot image (S203). When the marker is included in the shot image (S203:Y), a relevant information acquisition unit is configured to acquire relevant information corresponding to the marker ID (S204). In this case, a distance between the information processing device and the marker is obtained from a size of the marker included in the image (S212). When the distance is short, as to each of a plurality of markers, presentation information relevant to the marker is displayed. When the distance is long, only one presentation information is displayed.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a technique for displaying information related to an object recognized from an image.

2. Description of the Related Art Conventionally, a technique for projecting and displaying related information on a target focused on by a user is known. For example, Patent Document 1 discloses a technique for capturing an image of an object, searching for related information related to the imaged object, and projecting the image on the object. In this technique, the position and shape of an object are detected, and related information retrieved based on the detected position and shape is projected onto the object.
Patent Document 2 proposes a technique for changing the size and position of an image to be projected according to the distance from a device for projecting information to a projection target when projecting information.

JP-A-8-86615 International Publication No. 2011/105502

  When projecting related information of an object on an object existing in the real world, the amount of information of the related information that is easy for the user to visually recognize varies depending on the projection environment. For example, in a mobile projection apparatus in which the user can hold and carry the projection apparatus, the distance between the user and the object that is the projection plane is variable. In general, when the projection apparatus moves away from the projection plane while the projection apparatus continues to project the same image, the range of the projection plane irradiated with the projection light is expanded. At this time, the characters and images included in the projected image appear to be enlarged on the projection plane. However, when an object on the real world is used as a projection plane, the size of the object does not change regardless of the distance between the user and the projection plane. Therefore, as the user moves away from the projection surface, the appearance of the object becomes relatively small with respect to the enlarged projection image. That is, as the distance between the user and the projection surface increases, the partial area where the related information of the object is to be projected becomes smaller with respect to the entire projection image. Thus, in a situation where the size of the projection range is limited, if the amount of information to be projected is large, it becomes difficult to visually recognize the contents.

  In Patent Document 1, the projection content is generated based on the relative position between the apparatus and the projection target. However, Patent Document 1 does not describe anything about changing the information amount of the related information to be projected in the projection environment. Japanese Patent Application Laid-Open No. 2003-228561 describes the change of the size and position of the projection image according to the distance of the projection surface, but does not describe changing the information amount of the related information to be projected in the projection environment. The present invention has been made in view of the above problems, and the visibility of related information when projecting related information of an object on the object in an environment where the distance between the projection apparatus and the object serving as the projection surface is variable. It aims at improving.

  In order to solve the above-described problem, an information processing apparatus according to the present invention uses an imaging unit that images a target object and a captured image captured by the imaging unit to obtain related information corresponding to the target object. Generation means for generating presentation information for display at a position specified based on the position of the image, image display means for displaying an image including the generated presentation information, the object, and the information processing apparatus And distance acquisition means for acquiring the distance. The generation unit generates the presentation information by changing an amount of information included in the presentation information when the captured image and the distance acquired by the distance acquisition unit are larger than a predetermined value.

  ADVANTAGE OF THE INVENTION According to this invention, the visibility of the relevant information when projecting the relevant information of the said object on an object can be improved in the environment where the distance between the object used as a projection apparatus and a projection surface is variable.

(A) is a hardware block diagram of the information processing apparatus concerning 1st Embodiment, (b) is a functional block diagram showing the function formed in information processing apparatus. (A) is a flowchart which shows the process in 1st Embodiment, (b) is a flowchart which shows the preparation process of presentation information. (A) is an example of related information, (b) and (c) are examples of presentation information to be created, and (d) and (e) are explanatory diagrams of projection results. (A) is a flowchart showing a creation process of presentation information corresponding to a single marker, (b) and (c) are examples of created presentation information, (d) and (e) are explanations of projection results Figure. (A) is a flowchart showing a process of creating presentation information of a plurality of markers, (b) and (c) are examples of created presentation information, (d) and (e) are examples of projection results. Figure. (A) is a flowchart showing a process for creating presentation information corresponding to a plurality of markers, (b) and (c) are examples of created presentation information, and (d) and (e) are explanations of projection results. Figure. Explanatory drawing of a projection image. (A), (b) and (c) is explanatory drawing of an example of the marker group of the object which produces the presentation information in 2nd Embodiment. (A) is a flowchart which shows the determination process of a marker group, (b) is explanatory drawing of the type | formula which calculates | requires the distance between markers. Explanatory drawing of the projection result of the presentation information at the time of grouping based on the distance between markers and the value of an item in 2nd Embodiment. (A)-(c) is explanatory drawing of the projection result of the presentation information at the time of setting the upper limit of the number of presentation information in 2nd Embodiment. (A)-(c) is explanatory drawing of the projection result at the time of projecting preferentially the presentation information of the group of the projection center vicinity in 2nd Embodiment. (A)-(c) is explanatory drawing of the projection result at the time of performing rough grouping in 2nd Embodiment. (A)-(d) is explanatory drawing of the projection result of the presentation information at the time of performing information presentation by the time division in 2nd Embodiment. (A)-(c) is explanatory drawing of the projection result of the presentation information in the case of reducing information amount by not using the character in 2nd Embodiment.

<First Embodiment>
FIG. 1A is a hardware configuration diagram of the information processing apparatus 10 according to the first embodiment of the present invention. The information processing apparatus 10 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, and a RAM (Random Access Memory) 103 as functional components. In addition, the information processing apparatus 10 includes a hard disk drive (HDD) 104, a camera 105 as an optical system imaging unit as a functional component, and a projector 106 as an optical image projection unit. Each functional component can exchange data with each other via the bus 107.

  The CPU 101 reads and executes programs stored in the ROM 102 and the HDD 104. Thereby, the CPU 101 controls each hardware component.

  This program includes, for example, a control program for realizing each function, sequence, and processing described later. In addition to the control program described above, the ROM 102 stores various data that are referred to when the program is executed. The RAM 103 stores a work area used by the CPU 101, a load area for the control program, and the like.

  The HDD 104 is a kind of storage, stores the control program, and is read appropriately by the CPU 101 or the like. The camera 105 captures an image of an object to be imaged and stores the captured image in the RAM 103 or the HDD 104. Projector 106 projects information stored in RAM 103 or HDD 104. The camera 105 and the projector 106 are attached to the information processing apparatus 10 so that the imageable range of the camera 105 and the projectable range of the projector 106 overlap. The bus 107 performs an address signal for instructing a component controlled by the CPU 101, a control signal for controlling each component, transfer of data passed between the component devices, and the like.

The control program may be recorded in advance in the ROM 102 or the HDD 104, or may be stored in the ROM 102 or the HDD 104 from an external device or an external recording medium as necessary. The CPU 101 realizes each function by executing the control program recorded in the ROM 102 or the HDD 104.
In this embodiment, a projector is used as an example of an image display unit. However, the present invention is not limited to this, and any device capable of displaying a live view image such as a head mounted display, a smartphone, or a tablet is used. May be.

  FIG. 1B is a functional block diagram formed in the information processing apparatus 10 when the CPU 101 executes a control program. In the present embodiment, the CPU 101 forms functional modules such as a recognition unit 111, a related information acquisition unit 112, a projection environment estimation unit 113, and a presentation information creation unit 114.

  The recognition unit 111 recognizes the object ID and position of an object having a specific shape included in the captured image captured by the camera 105. In the present embodiment, a marker is used as an object having a specific shape, and the object ID is a marker ID. A marker is a specific pattern in which information such as numerical values and characters are embedded. Typical examples include a one-dimensional barcode and a two-dimensional barcode. The marker ID is information embedded in the marker and is used as an ID for distinguishing markers having different patterns.

  The related information acquisition unit 112 acquires related information associated with the recognized marker ID. The correspondence relationship between the marker ID and the related information is stored in the ROM 102 or the HDD 104 in advance. The related information stored in the server outside the apparatus may be acquired using a communication device. In the present embodiment, each of the marker IDs is associated with one or more items and a set of values classified into one of the items as related information. For example, there is a case where related information of a plurality of items such as “product name”, “size”, and “color” of an article is associated with one marker ID.

  The projection environment estimation unit 113 uses information about environmental factors affecting the projection (hereinafter referred to as projection environment) such as the distance between the information processing apparatus 10 and the projection surface, the color of the projection surface, and ambient light based on the captured image. To estimate. The presentation information creation unit 114 groups related information and creates presentation information based on the grouped related information. The projector 106 receives the presentation information from the presentation information creation unit 114 and projects the presentation information. The presentation information refers to information that presents the content of related information to the user. For example, it is information output as an image to the projector 106 for presentation to the user.

  However, in this embodiment, it is possible to recognize a plurality of markers at a time and project a plurality of presentation information at the same time. Accordingly, there are cases where each of a plurality of partial images included in the image representing one screen output by the projector 106 is referred to as presentation information. The presentation information creation unit 114 changes the number of presentation information and the amount of information included in the presentation information based on the projection environment. For example, even if there are a plurality of recognized marker IDs, depending on the projection environment, control is performed such that the information is aggregated into one piece of presentation information to be projected.

  Next, an operation example of the information processing apparatus 10 configured as described above will be described. FIG. 2A shows a flowchart of processing executed by each functional component constituting the information processing apparatus 10. The CPU 101 executes this process at regular intervals. Unless otherwise specified, the processes described below are executed by the CPU 101 through the recognition unit 111, the related information acquisition unit 112, and the like.

  As shown in FIG. 2A, the recognition unit 111 recognizes a captured image obtained through the camera 105, and determines whether the captured image has been updated since the previous execution (S201). When this sequence is executed for the first time after the information processing apparatus 10 is activated, it is determined that the captured image has been updated. If not updated (S201: N), the CPU 101 ends this sequence. When updated (S201: Y), the recognition unit 111 acquires the marker ID and position of the marker included in the captured image (S202). The marker ID and position can be acquired by image recognition, for example.

Next, the recognition unit 111 determines whether or not one or more markers are included in the captured image (S203). When the marker is not included in the captured image (S203: N), the CPU 101 ends this sequence.
When a marker is included in the captured image (S203: Y), the related information acquisition unit 112 acquires related information corresponding to the marker ID recognized by the recognition unit 111 in S202 (S204). Next, the presentation information creation unit 114 determines whether there are a plurality of marker IDs (S205). For example, the case where the number of markers is plural is assumed to be a case where a projection surface is a surface on which boxes each having one marker printed on a side surface are stacked. By associating the marker ID with information about an object (product) stored in the box, the information processing apparatus 10 can be used to refer to detailed information about the contents of the box as needed without opening the box. It becomes possible.

  When the number of marker IDs is one (S205: N), the presentation information creation unit 114 creates presentation information for a single marker (S206). When the number of marker IDs is plural (S205: Y), the presentation information creating unit 114 creates presentation information for each of the plurality of markers or by collecting information corresponding to the plurality of markers (S207). It is assumed that which process is executed is determined by the content of related information corresponding to the marker and the distance between the marker and the device, as will be described later. Finally, the presentation information creation unit 114 instructs the projector 106 to project the presentation information (S208), and ends the process.

  In S202, the recognition unit 111 acquires the marker ID and position of the marker included in the captured image. In this example, it is assumed that the imaging area matches the projection area of the projector 106, or the imaging area is smaller than the projection area. On the other hand, there may be a case where the imaging area is wider than the projection area. In that case, the marker ID and position of the marker included in the captured image and also included in the projection area may be acquired. An existing method can be used as a method of acquiring the projection area in the captured image. As a simple method, all pixel red color is projected by a projector, and it is imaged by an imaging device. By regarding the range in which the pixel is red in the captured image as the projection region, the projection region in the captured image can be determined.

  FIG. 2B is a flowchart showing details of creation of presentation information of a single marker in S206 of FIG. The projection environment estimation unit 113 acquires the size of the marker included in the captured image (S211). Next, the presentation information creation unit 114 obtains the projection distance between the marker and the device using the size of the marker included in the captured image (S212). Regarding the distance acquisition method, when the actual size of the marker is known in advance, the distance from the marker reflected in the captured image can be obtained from the angle of view of the camera 105 and the number of pixels of the captured image.

For example, when the horizontal angle of view is 90 degrees, the number of pixels of the captured image is 640 pixels wide, and the height is 480 pixels, an object having a length of 1 m at a distance of 1 m appears in 320 pixels in the captured image. . If the marker width is known to be 10 cm in advance, it can be estimated that the distance is 1 m if the marker width in the captured image is 32 cm, and 1.6 m if the marker width is 20 cm. The angle of view and the number of pixels used for estimation are generally determined by the characteristics of a lens or an image sensor that is one of the components that constitute the camera 105.
Next, the presentation information creation unit 114 creates presentation information according to the related information corresponding to the marker, the distance from the marker, and the position of the marker (S213), and ends the process.

FIG. 3A shows an example of the related information that the related information acquisition unit 112 acquires in S204 of FIG. In the present embodiment, the related information is uniquely determined for each marker ID. In this embodiment, the related information includes a plurality of pairs of items and values. For example, for the marker ID S1_LC204_240BK, as shown in FIG. 3A, there is related information composed of a set of five items and values.
In this marker, the value of the item “product name” is “LC204”. In addition, “size”, “color”, “product ID”, and “image” are set as items, and the values are “20 cm”, “black”, “LC204_240BK”, and “202240BK.jpg”, respectively. Yes. Note that the value can be arbitrary information, and can be character information or image information. In the example of FIG. 3A, the value of “image” is image information.

  3B, when the distance between the information processing apparatus 10 and the marker is 2 m, the presentation information creating unit 114 creates based on the related information shown in FIG. 3A in S213 of FIG. 2B. An example of the projected image and the presentation information included therein is shown. In these drawings, a projected image 301 is a projected image projected by the projector 106 onto an object captured by the camera 105. The presentation information 305 indicates the presentation information in the projection image 301 when the distance between the information processing apparatus 10 and the marker is 2 m. A marker region 306 indicated by a square region indicates the position and size of a portion to be superimposed on a marker (marker ID is S1_LC204_240BK) existing in the real world in the projection image.

  The marker area 306 is an area defined for alignment processing when the information processing apparatus 10 outputs a projection image so as to be superimposed on a marker in the real world. It is not represented as an image in the projected image. In FIGS. 3B and 3C, the marker region 306 is indicated by a dotted line in order to distinguish it from text or graphics expressed as an image. In this way, by positioning the marker area in consideration of the position and size of the marker in the real world in the projection image and performing alignment processing, related information can be projected in the vicinity of the object in the real world. It becomes possible. Here, the vicinity of the object means at least a range close to the object and includes a range overlapping the object.

  3C, when the distance between the information processing apparatus 10 and the marker is 1 m, the presentation information creating unit 114 creates based on the related information shown in FIG. 3A in S213 of FIG. 2B. An example of the presented information is shown. In this figure, the presentation information 307 indicates the presentation information in the projection image 301 when the distance between the information processing apparatus 10 and the marker is 1 m. Similarly to FIG. 3B, a marker area 306 indicated by a square area indicates the position and size of the portion of the projected image 301 to which the marker whose marker ID is S1_LC204_240BK should be superimposed.

  In this embodiment, since the presentation information creation unit does not generate a marker image in the projection image 301, the marker image itself is not shown in FIGS. 3 (b) and 3 (c). In these drawings, the marker area 306 is created so that the marker overlaps the marker area 306 when the projection image 301 is projected onto the object imaged by the camera 105.

  As shown in FIGS. 3B and 3C, the presentation information creation unit 114 creates presentation information including more related information when the distance between the information processing apparatus 10 and the marker is short. In the present embodiment, when the distance is 1.5 m or more and less than 2.0 m, the presentation information creation unit 114 includes three values of the related information in one presentation information. When the distance is 2.0 m or more, as shown in FIG. 3B, the presentation information creation unit 114 sets two values of the related information, that is, “LC204” and “24 cm” as one. Include in the presentation information. Although the information amount is counted in correspondence with the number of values of related information, it may be counted by the number of items. When the distance between the information processing apparatus 10 and the marker is less than 1.5 m, as shown in FIG. 3C, the presentation information creation unit 114 displays all of the related information, that is, “LC204”, “24cm”, Presentation information including “black”, “LC204_240BK” and an image is created.

  Note that the position and size of the marker in the projection image are obtained by converting the position and size of the marker region 306 included in the captured image using the correction parameter. In general, the camera 105 and the projector 106 in the information processing apparatus 10 have different mounting positions, projection angles, and focal lengths of lenses.

  Therefore, even if an object is imaged by the camera 105 and the captured image is directly projected by the projector 106, the object reflected in the captured image is not projected at the same position and size as the actual object. In order to project information according to the position and size of the target object included in the captured image, it is necessary to convert the shape of the marker included in the captured image into the shape in the projected image. The correction parameter used for this conversion can be determined based on, for example, a test pattern image projected by the projector 106 captured by the camera 105 and the captured image.

  By performing the conversion in this way, when information is projected onto the marker region 306 in the captured image, the information is superimposed on the marker 303 in the real world shown in FIGS. 3D and 3E below. As described above, the marker 303 is located in the marker region 306, but the marker region 306 itself is not displayed in the projection image. In this example, the presentation information created by the presentation information creation unit 114 from the distance between the information processing apparatus 10 and the marker is displayed with the lower left position of the marker region 306 as the origin. However, the origin is not limited to this position, and the method for setting the origin is not particularly limited.

  FIG. 3D and FIG. 3E show the projection results to the real world by the information processing apparatus 10. Note that the marker ID of the marker 303 shown in FIG. 3D is S1_LC204_240BK. The projectable surface 304 is a range projected by the information processing apparatus 10. In FIG. 3D, since the distance between the marker 303 and the information processing apparatus 10 is 2 m, the information processing apparatus 10 projects the presentation information 305 including two values shown in FIG. In FIG. 3E, since the distance between the marker and the information processing apparatus 10 is 1 m, the information processing apparatus 10 projects the presentation information 307 including all values shown in FIG.

FIG. 4A is a flowchart showing a process different from the process of FIG. 2B regarding the process of creating presentation information of a single marker shown in S206 of FIG. 2A.
The projection environment estimation unit 113 estimates the size of the projectable surface based on the captured image (S401). For example, an area surrounded by a plurality of markers or an area around the markers may be a projectable surface. In particular, an area in the vicinity of the marker where the projected presentation information can be visually recognized can be set as a projectable surface. In this example, the projectable surface is determined using the color gradient of the projection image. The projection environment estimation unit 113 recognizes the color gradient from the captured image, and projects the projection image using a region adjacent to the marker and having a color gradient of a predetermined value or less as a projectable surface. This is because when the color gradient is large, it is difficult to visually recognize the projection image, while when the color gradient is small, the projection image is easily visible. The presentation information creation unit 114 creates presentation information according to the related information corresponding to the marker, the projectable surface around the marker, and the position of the marker (S402), and ends the process.

  FIG. 4B shows a projection image 401 created by the presentation information creation unit 114 based on the related information shown in FIG. In this figure, the presentation information 411 is presentation information when the area of the projectable surface is 60,000 square pixels, and stores three values of three items among the related information, that is, “LC204, 24 cm, black”. Has been. A projectable surface 413 indicates a projectable surface in the projection image 401. FIG. 4C shows a projection image 402 created by the presentation information creation unit 114 based on the related information shown in FIG. FIG. 4C shows a case where the projectable surface is smaller than that in FIG. The presentation information 412 is presentation information when the area of the projectable surface is 25,000 square pixels, and stores only one value of the related information, that is, “LC204”. A projectable surface 414 indicates a projectable surface in the projection image 402.

  Hereinafter, an example of a method for creating presentation information corresponding to a projectable surface will be described. First, the projection environment estimation unit 113 calculates the area of the projectable surface included in the video. In this embodiment, the area of the projectable surface is calculated in units of square pixels. Next, the presentation information creation unit 114 includes the relevant information in an amount that fits in the area of the projectable surface in the presentation information. In the present embodiment, the presentation information creation unit 114 displays the presentation information 411 and 412 with the lower left position of the marker area 306 as the origin.

  FIG. 4D shows a result of projecting the presentation information shown in FIG. Similarly, FIG. 4E illustrates a result of the information processing apparatus 10 projecting the presentation information illustrated in FIG. In these drawings, a projectable surface 413 indicates a region estimated by the projection environment estimation unit 113 as a projectable surface. In FIG. 4D, since the size of the projectable surface 414 is 60,000 square pixels, the information processing apparatus 10 projects the presentation information 411. In FIG. 4E, since the area of the projectable surface is 25,000 square pixels, the information processing apparatus 10 projects the presentation information 412.

  In S212 of FIG. 2B, the distance from the marker is obtained using the size of the marker shown in the captured image, but the distance between the information processing apparatus 10 and the marker can be obtained by an arbitrary method. it can. For example, the distance may be estimated using a distance sensor using ultrasonic waves or infrared rays. By using the distance sensor, the distance to the marker can be estimated even when the size of the marker is not known in advance. Further, as in this embodiment, when an optical system projection unit such as the projector 106 or an optical system imaging unit such as the camera 105 is used, the distance to the marker may be estimated from the focal length. In this case, the user adjusts the focal length and the distance from the marker so that the projector 106 and the camera 105 are focused on the marker. Therefore, it can be assumed that the projector 106 and the camera 105 are in focus with the marker. Accordingly, it is possible to estimate the distance in focus from the current focal length and use the value as the distance. By estimating the distance to the marker from the focal length, the distance to the marker can be estimated even when the size of the marker is not known in advance.

  In S213 of FIG. 2B and S402 of FIG. 4A, the presentation information creation unit 114 may create presentation information according to the aperture value of the projector 106. As an example, when the aperture value is at a minimum, that is, opened to the maximum, the presentation information creation unit 114 may reduce the amount of information included in the related information and display the enlarged information accordingly. Opening the aperture of the projector 106 has the effect of reducing the contrast of the image instead of increasing the brightness of the projector 106. When the user opens the aperture to the maximum, it means that the projected image of the projector 106 is difficult to see as the brightness of the projector 106 needs to be increased to the maximum. For this reason, displaying characters and images large when the aperture is fully released has the effect of improving the visibility of the user.

In the above-described example, the information amount is reduced when the aperture value is minimum. However, the information amount may be decreased when the aperture value is smaller than a predetermined value.
Alternatively, when the aperture value is the minimum or smaller than the predetermined value, the presentation information creation unit 114 may change the display form of the presentation information. In this case, the display form is preferably changed so that the visibility of the presentation information is improved. For example, the display form is changed by enlarging the character, enlarging the space between characters, or displaying the icon instead of displaying the character.

  Further, in S401 of FIG. 4A, the recognition unit 111 may acquire the size of the projectable surface corresponding to the recognized marker ID from the ROM 102, the HDD 104, or a database defined outside the apparatus. When the positions of the information processing apparatus 10 and the marker are known, the position information and the size of the recognizable surface corresponding to each marker are defined in advance by storing the position information of both in the ROM 102 or the like. be able to.

  Presentation information can be created by various methods. You may store in order from the information which can be accommodated in the magnitude | size on the basis of the magnitude | size of a projectable surface. In this case, the presentation information is preferentially stored from information that requires a small display area, such as information with a small number of characters. In addition, a priority may be determined for each value included in the related information, and may be included in the presentation information in order from a value with a higher priority.

  In this case, if values are stored in the presentation information in order of priority, a value with a large required display area (hereinafter referred to as value A) cannot be stored, but a lower priority and a required display area. There may also be situations where a small value can be stored. In this case, the value A is not stored in the presentation information, and it can be determined whether or not the next highest priority value can be stored in the presentation information. This priority may be determined in advance or specified by the user. In the above-described embodiment, the presentation information is projected on a region different from the marker. However, the position and size may be determined so that the presentation information is superimposed on the marker, or the information may be projected in any direction of the marker so as not to be superimposed on the marker. When the color gradient of the marker is strong, visibility can be ensured by not superimposing and projecting on the marker.

  FIG. 5A is a flowchart showing a process of creating presentation information of a plurality of markers shown in S207 of FIG. The projection environment estimation unit 113 calculates the distance between the marker and the information processing apparatus 10 for all markers included in the captured image (S501). As a method for obtaining the distance, there is a method of estimating using the size of the marker shown in the captured image, as in the processing of S211 and S212. Further, the focal length of the distance sensor, projector 106 or camera 105 may be used.

  Next, the presentation information creation unit 114 determines the amount of presentation information according to the distance between the marker and the information processing apparatus 10 (S502). In particular, in this embodiment, as an example of processing for increasing or decreasing the amount of information, the number of pieces of presentation information to be created is determined according to the distance between the marker and the information processing apparatus 10. Specifically, the distance between the information processing apparatus 10 and the marker is obtained, and the number of pieces of presentation information is determined according to the value. Here, the number of presentation information corresponds to the number of images arranged as independent partial areas in the entire projection image output by the projector 106.

  For example, when text information is arranged in a graphic object such as a rectangle or a balloon shape and projection is performed so that the text information is superimposed on a real object, the number of graphic objects can be regarded as the number of presentation information. In this embodiment, if the distance between the marker and the information processing apparatus 10 is equal to or greater than a predetermined value, one presentation information is created for a plurality of markers, and if the distance is less than the predetermined value, the same number of presentation information as the number of recognized markers is created. To create. Here, it is assumed that the amount of information included in one presentation information is substantially constant. Therefore, creating a plurality of pieces of presentation information equal to the number of markers means that the amount of information that can be projected is increased compared to the case where only one piece of presentation information is generated. When the distance between the information processing apparatus 10 and each marker is different, the average value is set as the distance between the information processing apparatus 10 and the marker. Hereinafter, the predetermined value will be described as 1.5 m.

  Next, the presentation information creation unit 114 creates presentation information according to the information amount determined in S502 (S503). In this embodiment, the determined number of presentation information is created. FIG. 5B shows an example of the presentation information created in the projected image 501 by the presentation information creation unit 114 based on the related information shown in FIG. 3A in S503 of FIG. In FIG. 5B, a region 511 indicates the position and size of the marker whose marker ID is S1_LC204_240BK in the projection image 501. An area 512 indicates the position and size in the projection image of the marker whose marker ID is S1_LC204_240WH.

  An area 513 indicates the position and size of the marker whose marker ID is S1_LC204_230BK in the projection image. The presentation information 515 indicates the presentation information created by the presentation information creation unit 114 when the distance between the information processing apparatus 10 and the marker is 2 m. In this example, when the distance is 2 m, the presentation information creation unit 114 determines to create one presentation information in S502.

FIG. 5C shows the presentation information 516, 517, 518, and 519 that the presentation information creation unit 114 creates in the projection image 502 when the distance between the information processing apparatus 10 and the marker is 1 m. Since the distance between the information processing apparatus 10 and the marker is less than 1.5 m, the same number of presentation information as the recognized marker is created. The marker IDs of the created presentation information 516 and 517 are S1_LC204_240BK and S1_LC204_240WH, respectively. Also, the marker IDs of the created presentation information 518 and 519 are S1_LC204_230BK and S1_LC204_230WH, respectively.
When creating a plurality of pieces of presentation information, the presentation information creation unit 114 creates the presentation information so that the contents of the respective pieces of presentation information are different. Specifically, among the related information corresponding to each marker, information having different values for all marker IDs is preferentially included in the presentation information.

  As shown in FIG. 3A, in each marker ID, the product name is LC204 and is common. In addition, the size and the color are common to some markers. For example, in the case of marker IDs S1_LC204_240BK and S1_LC204_230BK, the color is black and they are common. On the other hand, the product ID and the image have different values for all marker IDs. Therefore, for the presentation information 516, 517, 518 and 519, the product ID is preferentially included in the presentation information.

  FIG. 5D and FIG. 5E show the projection results by the information processing apparatus 10. The marker IDs of the markers 521 and 522 are S1_LC204_240BK and S1_LC204_240WH. The marker IDs of the markers 523 and 524 are S1_LC204_230BK and S1_LC204_230WH.

In FIG. 5D, the distance between the markers 521, 522, 523, and 524 and the information processing apparatus 10 is 2 m, which is 1.5 m or more. Therefore, the information processing apparatus 10 projects the presentation information 515.
In FIG. 5E, the distance between the markers 521, 522, 523, and 524 and the information processing apparatus 10 is 1 m, which is less than 1.5 m. Therefore, the information processing apparatus 10 projects the presentation information 516, 517, 518, and 519.

FIG. 6A is a flowchart showing a method different from the process shown in FIG. 5A for the process of creating presentation information of a plurality of markers shown in S207 of FIG. 2A.
The presentation information creation unit 114 estimates a projectable surface including a marker for all markers included in the captured image (S601). As in the process of S401 in FIG. 4A, the projection possible surface may be obtained from the color gradient of the captured image, or the projection possible surface for the recognized marker ID may be inquired to the outside.

  Next, the presentation information creation unit 114 determines the number of presentation information to be created from the size of the projectable surface of each marker (S602). As an example of a method of determining the number of pieces of presentation information, there is a method of determining according to the total value of the area of the projectable surface. In the present embodiment, one presentation information is created when the area of the projectable surface is less than 50,000 square pixels, and the same number of presentation information as the number of recognized markers is created when the area is 50,000 square pixels or more. . Next, the presentation information creation unit 114 creates presentation information by the number determined in S602 (S603).

  6B and 6C show examples of the presentation information created by the presentation information creation unit 114 based on the related information shown in FIG. 3A in S603 of FIG. 6A. In FIG. 6B, the presentation information 611 arranged in the projection image 601 is the presentation information created by the presentation information creation unit 114 when the area of the projectable surface is 25,000 square pixels. Since the area of the projectable surface is less than 50,000 square pixels, only one presentation information 611 is projected in the example of FIG. 6B.

  In FIG. 6C, the presentation information 612, 613, 614, and 615 arranged in the projection image 602 is created by the presentation information creation unit 114 when the area of the projectable surface is 60,000 square pixels. Presented information. Since the area of the projectable surface is 50,000 square pixels or more, four pieces of presentation information equal to the number of markers are created in the example of FIG. 6C. These presentation information 612 and 613 are presentation information corresponding to markers whose marker IDs are S1_LC204_240BK and S1_LC204_240WH. The presentation information 614 and 615 is presentation information corresponding to markers whose marker IDs are S1_LC204_230BK and S1_LC204_230WH.

  FIG. 6D and FIG. 6E show the projection results by the information processing apparatus 10. In FIG. 6D, the total area of the projectable surface 633 including the markers 621, 622, 623 and 624 is 25,000 square pixels. Therefore, the information processing apparatus 10 projects the presentation information 611. In FIG. 6E, since the total area of the projectable surface 634 including the markers 621, 622, 623, and 624 is 60.000 square pixels, the information processing apparatus 10 displays the presentation information 612, 613, 614, and 615. Project.

In the examples of FIGS. 5 and 6A to 6E, in the case where all markers are included in one projectable surface, the area of the projectable surface is less than 50,000 square pixels. Projected one presentation information on one projectable surface.
However, depending on the color of the projection surface, the region including a certain marker and the region including another marker may not be continuous on the projectable surface. As an example, there may be mentioned a case where a plurality of regions having a small color gradient are divided by a region having a large color gradient.

  In such a case, when the presentation information is projected across a plurality of regions within one projectable plane, the visibility may be reduced due to a difference in color gradient or the like. Therefore, in the example illustrated in FIG. 7, the presentation information creation unit 114 divides one projectable surface into divided regions. Hereinafter, the region divided in this way is referred to as a divided surface. And one presentation information is produced with respect to the 1 or several marker contained in each division surface. Details will be described below.

FIG. 7 shows a projected image when the number of pieces of presentation information projected on the projectable surface is determined as “1” in S602 of FIG. 6A and one presentation information includes one value. An example is shown. As shown in FIG. 7, the recognition unit 111 recognizes four markers 621, 622, 623, and 624 in the projection image 701.
In this example, since there is a region having a large color gradient in the projectable surface, the projectable surface is divided into the first divided surface 711 and the second divided surface 712 by the presentation information creation unit 114. Yes. The first dividing surface 711 includes markers 623 and 624. In addition, markers 621 and 622 are included in the second divided surface 712 in a separate area that is not continuous with the first divided surface 711.

As described above, in S602, it is determined that the number of pieces of presentation information created in the projectable plane is one. However, the presentation information creation unit 114 creates one presentation information for each of the first divided surface 711 and the second divided surface 712. As a result, two pieces of presentation information are projected on the projectable surface before division.
As shown in FIG. 7, presentation information 713 is projected on the first divided plane 711, and presentation information 714 is projected on the second divided plane 712. Thereby, it can prevent that presentation information is projected on the area | region where a color gradient is large, and visibility falls.
As described above, by changing the content and number of the presentation information according to the distance between the information processing apparatus 10 and the marker, the area and shape of the projectable surface, the presentation information with an appropriate amount of information according to the projection environment is projected. can do.

  In S502 and S602, the presentation information creation unit 114 may determine the number of pieces of presentation information according to the aperture value of the projector 106. As an example, when the aperture value is at a minimum, that is, when the aperture value is opened to the maximum, the presentation information creation unit 114 may reduce the number of presentation information and display characters and images larger accordingly. When the user opens the aperture to the maximum, it means that the projected image of the projector 106 is difficult to see as the brightness of the projector 106 needs to be increased to the maximum. Therefore, reducing the number of pieces of presentation information when the aperture is fully released and displaying characters and images larger accordingly has the effect of improving the visibility of the user. Further, instead of reducing the number of presentation information, the display form of the presentation information may be changed so that the visibility is high.

In this embodiment, the projector is used as an example of the image display unit. However, the present invention is not limited to this, and any display device that can display a live view image such as a head mounted display, a smartphone, or a tablet is used. Also good. The distance when a projector is used as the image display means means the distance from the information processing apparatus to the projection plane. On the other hand, when a display device is used as the image display unit, the concept of a projection plane does not apply, and thus the distance means a distance from the information processing apparatus to the object.
When a display device is used as the image display means, information can be displayed at a position different from the object. For example, information can be displayed in a space around the object.

  Means different from the present embodiment (for example, displaying information other than on the object) are also effective. However, when a plurality of objects are arranged on the entire screen and there is no gap for displaying information other than on the object, the means is not effective, and the visibility according to the present embodiment is improved. The effect to improve is acquired. Furthermore, although the case where the recognition unit 111 recognizes a marker has been described in the present embodiment, the recognition unit 11 may use something other than the marker for recognition. For example, the recognition unit 11 may recognize an object instead of a marker. In this case, the recognition unit 11 identifies the object ID by recognizing the shape of the object. Therefore, the object itself is recognized as a marker.

  Existing technology is used for object shape recognition. As a simple method, it is effective to prepare an image obtained by capturing the shape of an object from a plurality of angles in advance and perform partial matching with the captured image. It is also effective to prepare an object shape as a 3D model and perform matching using a distance image (in this case, a device capable of acquiring a distance, such as a distance image sensor or a stereo camera).

Second Embodiment
In the first embodiment, when a plurality of markers exist in the captured image, an example in which presentation information is created for each marker and an example in which only one presentation information is created are shown. When presentation information is created for each marker, the presentation information and the marker correspond “one-to-one”, and the relationship between the two is clear. When there is only one presentation information, it is clear that the presentation information is a “1: all” correspondence corresponding to all markers. Therefore, in these examples, the correspondence between the marker and the presentation information is self-evident.

  On the other hand, the number of pieces of presentation information is not limited to this, and may be any number that is, for example, two or more and less than the number of markers. By increasing or decreasing the number of presentation information within this range, the amount of information can be finely adjusted according to the projection environment, and visibility can be improved. However, in this case, there are a plurality of presentation information and markers, and the correspondence between the presentation information and the markers is not obvious.

  In the second embodiment, when the relationship between the marker and the presentation information is not obvious, a method for determining the position of the presentation information so as to clarify the relationship between the two is shown. In this embodiment, a plurality of markers are grouped into N marker groups (groups) each including one or more markers, and one presentation information is associated with each marker group. N is a number that is 2 or more and less than the number of markers.

  Since there may be a plurality of combinations of markers included in the marker group, it is desirable to select a combination having a high visibility among the plurality of combinations. Therefore, in this embodiment, a combination is selected in which markers having high similarity between markers included in the marker group are arranged in the same marker group, and markers having low similarity are arranged in different marker groups. Specifically, the similarity between the markers constituting the marker group was determined, and the combination of markers included in the marker group was determined using each similarity.

  Further, if the markers included in one marker group are separated from each other, the presentation information is displayed at a position separated from at least one marker, regardless of where the presentation information is displayed. Therefore, it is difficult for the user to identify the marker corresponding to the presentation information. Therefore, in this embodiment, the distance between the markers constituting the marker group is obtained, and the combination of markers included in the marker group is determined using the distance obtained for each marker group. As described above, the object itself can be used as a marker. In this case, the distance between the objects is obtained, and the combination of objects included in the object group is determined.

  FIG. 8A shows four markers included in an image captured by the camera 105. As illustrated, in this example, the captured image includes markers 621, 622, 623, and 624. FIG. 8B is an explanatory diagram of IDs of four markers included in the captured image shown in FIG. 8A and positions in the image. FIG. 8C shows all combinations of marker selection methods for the marker group when the number of markers is 4 and two marker groups are generated. However, one marker group must be assigned at least one marker.

As shown in FIG. 8A, in this example, there are four markers, and four markers are assigned to two marker groups.
Further, as shown in FIG. 8B, the position of each marker is designated in the projection image with the lower left of the screen as the origin, the horizontal direction in the drawing as the x axis, and the vertical direction as the y axis. For example, the marker whose marker ID is S1_LC204_230WH is at the position (100, 100), and corresponds to the marker 621 located at the lower left in FIG. Similarly, the marker with marker ID S1_LC204_240BK corresponds to the marker 624 at the position (100, 500), and the marker with marker ID S1_LC204_240WH corresponds to the marker 623 at the position (700, 500). Furthermore, the marker whose marker ID is S1_LC204_230BK corresponds to the marker 622 at the position (700, 100).

  As shown in FIG. 8 (c), there are roughly two types of marker assignment methods. That is, there are a selection method (SEL1 to 4) in which one marker group is assigned one marker and the other marker group is assigned three markers (SEL1 to 4), and a selection method in which two markers are assigned to both marker groups (SEL5 to 7).

  SEL1 to SEL4 indicate the results of selecting one marker from the four markers and assigning it to the marker group 1 and assigning the remaining three markers to the marker group 2. On the other hand, SEL5 to 7 show the results of selecting two markers from the four markers and assigning them to the marker group 1 and assigning the remaining two markers to the marker group 2. As shown in the figure, when one marker group is always assigned one or more markers and four markers are assigned to two marker groups, there are seven combinations.

FIG. 9A is a flowchart showing a process for determining which combination is adopted among the combinations of marker groups shown in FIG.
The presentation information creation unit 114 identifies the marker assigned to each marker group for each combination of marker groups (S901). For example, in SEL1 shown in FIG. 8C, the markers assigned to the marker group 1 and the marker group 2 are identified. As a result, it can be seen that the marker group 1 is assigned a marker having a marker ID of S1_LC204_204BK, and the marker group 2 is assigned another three markers.

Next, the presentation information creation unit 114 calculates a similarity between markers included in each marker group (S902). In the present embodiment, the similarity of related information corresponding to each marker is used as a method for obtaining the similarity. Although the degree of similarity can be determined by an arbitrary method, in this embodiment, the degree of similarity is determined by the number of pieces of related information having the same value for all markers included in the group. It is determined that the similarity is higher as the number of related information having the same value is larger. When the number of markers included in the group is one, the similarity is 0.
Therefore, for example, in SEL1 shown in FIG. 8C, the number of markers included in the marker group 1 is 1, so the similarity is 0. In the marker group 2, as shown in FIG. 3, only the product name value (LC204) of the related information is common to all the markers, and all other values are different. Therefore, the similarity is 1.

The presentation information creation unit 114 determines whether or not the process of S902 has been completed for all marker groups (S903). If not completed, S902 is executed again (S903: N). If it has been completed, it is determined whether or not processing has been completed for all methods of selecting the marker group (S904). In the example of FIG. 8C, it is determined whether or not the processing has been completed for all SEL1 to SEL7. If the process has not been completed (S904: N), S901 is executed again. When the processing is completed (S904: Y), the presentation information creation unit 114 obtains the total similarity of each marker group for each selection method, and whether or not there are a plurality of selection methods with the highest total. Is determined (S905). In the example of FIG. 8C, for SEL1 to SEL7, the sum of the similarities of each marker group is obtained, and it is determined whether there is one method of selecting the highest sum of similarities.
When there is one way of selecting the highest degree of similarity (S905: N), presentation information is created using the method of selection (S908). When there are a plurality of selection methods having the highest similarity, the distance between the markers in the marker group is calculated (S906).

In this embodiment, the distance between markers in the marker group is obtained by the equation shown in FIG. In this equation, K represents the number of markers included in the marker group, X0, X1, Xi, and XK are positions in the X direction in the captured image of each marker, and Y0, Y1, Yi, and YK are It represents the position in the Y direction in the captured image of each marker.
The presentation information creation unit 114 creates presentation information using a method of selecting the group with the highest similarity and using the method of selecting the smallest total distance between the markers in the group (S907), and ends the process.

Hereinafter, a process for determining which of the marker group selection methods shown in FIG. 8C is used to create the presentation information will be specifically described with reference to FIG.
First, the presentation information creation unit 114 executes a process for S902 on the two marker groups included in the selection method SEL1. Since the marker group 1 of SEL1 includes only one marker, the similarity is 0, and the value common to the markers included in the marker group 2 is LC204 of the product name, and the similarity is 1. Therefore, the similarity of SEL1 is 1 as the sum of the similarity of marker group 1 and the similarity of marker group 2. Similarly, the similarities of SEL2, SEL3, and SEL4 are all 1.

On the other hand, for SEL5, the similarity between the marker group 1 and the LC204 in the related information is 23 cm, so the similarity is 2, and the marker group 2 is also similar because the product name and the size are in the related information Becomes 2. Therefore, the total similarity is 4.
Regarding SEL6, although the size of the related information is different in the marker group 1, the colors match, so the similarity is 2. Similarly, the similarity of the marker group 2 is 2, and the total similarity is 4.

In SEL7, since both the marker group 1 and the marker group 2 have related information that matches only the product name, the degree of similarity is 1, respectively. Therefore, the total similarity is 2.
Therefore, the presentation information creation unit 114 determines in S905 that there are two selection methods with the highest degree of similarity, SEL5 and SEL6. Next, in S906, the presentation information creation unit 114 calculates the marker inter-marker distance for each marker group included in SEL5 and SEL6.

Here, in the calculation formula of FIG. 9B, XM is the average value of the X coordinates of the markers in the marker group. In the marker group 1 of SEL5, since the X coordinate of the marker 624 is 100 and the X coordinate of the marker 623 is 700, XM is 400. Regarding the Y coordinate, since the Y coordinate of these markers is 500, YM is also 500.
Next, the inter-marker distance of the marker group 1 of SEL5 is calculated from the marker position shown in FIG. 8B and the calculation formula of FIG. 9B. This value is [√ {(100−400) ² + (500−500) ² } + √ {(700−400) ² + (500−500) ² }], that is, 600.
Further, the distance between markers in the marker group 2 of SEL5 is also determined to be 600 (XM = 400, YM = 100). Therefore, the total distance between markers in the marker group of the marker group included in SEL5 is 1200.

Next, the distance between the markers in the marker group 1 of SEL6 is determined to be about 721 (XM = 400, YM = 300). Further, the distance between markers in the marker group 2 of SEL6 is calculated to be about 721 (XM = 400, YM = 300). Therefore, the total distance between markers in the marker group included in SEL6 is about 1442. Therefore, in S907, the presentation information creation unit 114 determines to create the presentation information with SEL5 in which the total distance between markers in the marker group is small. The presentation information is projected onto the marker group 1 constituted by the markers 623 and 624. And presentation information is projected with respect to the marker group 2 comprised by the marker 621 and the marker 622. FIG.
FIG. 10 shows a state in which the information processing apparatus 10 projects the presentation information created as a result of this processing. In this figure, the projected image 702 shows four markers 621, 622, 623 and 624.

As shown in FIG. 10, by creating presentation information in SEL 5, the projection image 702 is divided into a first divided surface 721 and a second divided surface 722. Then, the presentation information 723 is projected on the two markers 623 and 624. As the value of the presentation information 723, “LC204” and “24 cm” are projected.
Also, presentation information 724 is projected on the two markers 621 and 622. “LC204” and “23 cm” are projected as the values of the presentation information 724. As a result, it is possible to generate a marker group including markers with high similarity and present presentation information to each marker group. Furthermore, the correspondence between the marker and the presentation information is also clarified.

  In this way, by determining how to select a marker group based on the marker related information and the position of the marker included in the captured image, the content is highly related and presented to marker groups that are close to each other. Information can be presented.

8, 9, and 10, the case where a plurality of markers are grouped and information is presented for each group has been described. At that time, it was assumed that all markers were classified into any group. However, the present invention is not limited to this, and a certain marker may not be classified into any group and information may not be presented. Similarly, information may not be presented for a certain group.
For example, in accordance with priorities determined in advance, information on the top M groups (M is the number determined in advance) with higher priorities is presented. When the number of groups is larger than M, information presentation of some groups with low priority is not performed.

  FIGS. 11A to 11C are schematic diagrams illustrating examples of group information presentation in the case of M = 2. In FIG. 11A, reference numerals 1101 to 1112 denote boxes containing products, which are arranged in three vertical rows and four horizontal rows. Assume that the products 110 are stored in the boxes 1101 to 1105, the product C is stored in the boxes 1106, and the products B are stored in the boxes 1107 to 1112. A marker is attached to each product box, but the marker display is omitted for simplicity. FIG. 11B shows the result when the product boxes shown in FIG. 11A are grouped and displayed without limitation on the number of information presentations. In the figure, presentation information 1113 is presentation information for a group of products A, and in this example, information is presented by the letters “product A” and a thick black solid line surrounding the target product.

Similarly, the presentation information 1114 is presentation information about the product B, and the presentation information 1115 is presentation information about the product C. For example, the store focuses on the sales of the products A and B, and the priority is the product A> the product B> the product C, and the information presentation number M = 2. FIG. 11C displays the information presentation result in this case. Since M = 2, only information related to the product A and the product B is presented, so that a customer who uses the information processing apparatus can easily focus on the product A and the product B.
As another form, the priority of a product with a short distance from the projection center may be increased. In this case, the group information is presented in the order from the projection center. As yet another form, the number to be presented may be determined in advance, or the number may be determined according to the size of the projectable area.

FIGS. 12A to 12C illustrate the projection results when information is presented in two groups (M = 2) close to the projection center by increasing the priority of a product having a short distance from the projection center. FIG.
In FIG. 12 (a), as in FIG. 11 (a), 1101 to 1112 represent boxes containing products, the product 110 is in the boxes 1101 to 1105, the product C is in 1106, and 1107 to 1112. Suppose that each product B is contained.
FIG. 12B shows a projection result when the projection center 1201 is in the vicinity of the box 1109 located near the product A among the boxes containing the product B. Since M = 2, information is presented for two groups of products B and A that are close to the projection center 1201 among the three products A to C. Therefore, no information is presented for the group of products C.

  FIG. 12C shows a projection result when the projection center 1202 is in the vicinity of the box 1103 containing the product A. The distance between the product C and the projection center 1202 is shorter than the distance between the product B and the projection center 1202. Accordingly, information on two groups of the product A and the product C is presented, and information on the group of the product B is not presented. As a result, information in the vicinity pointed to by the user by the information processing apparatus is preferentially presented, and the visibility with respect to the merchandise information at the place where the user wants to pay attention is improved. 8 and 10, the case has been described in which information is presented by grouping a plurality of markers for each marker in which at least a part of the related information is common.

However, the present invention is not limited to this, and markers that do not match the related information may exist in the group. Therefore, it is possible to form a group including markers whose related information does not match and present information to the group.
For example, when it is desired to look down on the whole from a long distance, it may be required to present information giving priority to visibility while allowing some information error. In this case, there is an advantage that it is easy for the user to roughly grasp the whole by forming a group including markers whose related information does not match.

  FIGS. 13A to 13C are explanatory diagrams of an example in which a result of a rough grouping that allows a marker having a mismatched item value to be included in a group is displayed. In this example, when the number of products belonging to a certain marker is less than 2, the product belongs to another group without being a single group.

  In FIG. 13 (a), as in FIG. 11 (a), 1101 to 1112 represent boxes in which products are put, products 110 are in boxes 1101 to 1105, products C are in 1106, products 1107 to 1112 are in Each contains product B. FIG. 13B shows the information presentation result in this example. As shown in the figure, the presentation information 1301 is shown for the group of products A, and the presentation information 1114 is shown for the products B. Although the product C is actually arranged in the box 1106, the presentation information regarding the product C is not displayed. As illustrated, the box 1106 is included in the region of the presentation information 1301 for the product A. This is because the product C arranged in the box 1106 has less than 2 products, and thus information is not presented as a single group.

In this example, the product C is included in the group of the product A. However, whether the product C is included in the group of the product A or the group of the product B can be arbitrarily determined. For example, based on the relative positions of the product A and the product B with respect to the product C, it may be determined which group the product C is included in. For example, the distance between the center position of each group of the product A and the product B and the center position of the product C can be obtained, and the product C can be formed by including the product C in the product having the smaller distance.
In addition, the belonging group may be determined so that the number of corners of the frame surrounding the group is reduced. FIG. 13B is consistent with the result of executing this method. In the case of FIG. 13B, the number of corners in the frame surrounding the group of products A is 4, the number of corners in the frame surrounding the group of products B is 4, and the total number of corners is 8.

  On the other hand, an example in which the product C belongs to the product B is shown in FIG. In this example, the frame surrounding the group of products A is the same as that shown in FIG. 12B, and the number of corners is six. On the other hand, the number of corners in the frame surrounding the group of products B is 6, as shown in FIG. 8 and 10, the case has been described in which information on grouped results is presented simultaneously. However, the present invention is not limited to this, and presentation information may be displayed in a time-sharing manner for each group.

  FIG. 14 is a schematic diagram when information is presented for each group in a time-sharing manner. Since FIG. 14A is the same as FIG. 11A, description thereof is omitted. When information is presented, group information presentation is switched in the order of FIG. 14B, FIG. 14C, and FIG. 14D every predetermined time (for example, 3 seconds). Thus, visibility can be improved by performing information presentation by time division. It should be noted that after the information is presented as shown in FIG. 14 (d), the display may be repeated sequentially from FIG. 14 (b) again.

  8 and 10, the case where the number of characters is reduced as a means for reducing the information amount as the distance increases is described. However, the method for reducing the information amount is not limited to this, and information without using characters is used. You may make a presentation. For example, only a frame surrounding the group may be displayed, the group may be displayed in different colors, or an icon, an image, an illustration, or the like representing the type of group may be displayed.

FIGS. 15A to 15C are explanatory diagrams of examples in which information is presented without using characters.
FIG. 15A shows only the frame surrounding the group. In this case, the user cannot know detailed information such as the color, size, and model number of the product. However, the user can easily understand visually that the three types of products are arranged in the illustrated 3 × 4 = 12 boxes and the positions where the various types of products are arranged. Therefore, the user does not need to check all the boxes when checking whether there is a product he / she desires, and only needs to check one box for each of the three types of products.

FIG. 15B shows an example in which colors are grouped for each group. In this example, as in the example of FIG. 15A, the user cannot know detailed information on the product. However, by color-coding as shown in the figure, the user can visually more easily understand the position where each type of product is arranged.
FIG. 15C shows an example in which a shoe picture is displayed as an image instead of characters. In this example, as in the examples of FIGS. 15A and 15B, the positions and the like at which each type of product is arranged can be easily understood visually. Furthermore, by using an image, it is possible to easily visually understand information such as whether the product is a shoe, the color of the shoe, and whether it is a shoelace type or a magic tape type.
As described above, in the examples of FIGS. 15A to 15C, it is possible to reduce the amount of information and improve visibility by deleting characters. Furthermore, by using the product image, the user can easily understand information on the color and appearance of the product.

  As described above, according to the present invention, it is possible to improve the visibility of the user when projecting the presentation information by changing the information amount of the related information using the captured image. Moreover, each embodiment mentioned above is an illustration, and various deformation | transformation are possible. For example, in the above-described embodiment, a two-dimensional barcode is used as a marker. However, as long as identification is possible, not only a barcode but also any figure or the like can be used as a marker. A color barcode or the like may be used.

  The present invention can also be implemented by supplying a storage medium storing a program code (control program) of software or a computer program that implements the functions of the above-described embodiments to a system or apparatus. Further, the present invention can also be implemented by reading and executing computer-readable program code stored in a storage medium by a computer (or CPU or MPU) of the system or apparatus.

In this case, the program code itself read from the storage medium realizes the functions of the above-described embodiments, and the storage medium storing the program code constitutes the present invention.
As a storage medium for supplying the program code, for example, a flexible disk, a hard disk, an optical disk, a magneto-optical disk, a CD-ROM, a CD-R, a magnetic tape, a nonvolatile memory card, a ROM, or the like can be used.
Further, the functions of the above-described embodiments are not only realized by executing a computer program or program code read by a computer. This includes the case where the OS (operating system) running on the computer performs part or all of the actual processing based on the instruction of the program code, and the functions of the above-described embodiments are realized by the processing. .

  Further, the program code read from the storage medium is written in a memory provided in a function expansion board inserted into the computer or a function expansion unit connected to the computer. Thereafter, based on the instruction of the program code, the CPU 101 or the like provided in the function expansion board or function expansion unit performs part or all of the actual processing, and the function of the above-described embodiment is realized by the processing. It is.

Claims (24)

An information processing apparatus,
An imaging means for imaging an object;
Generating means for generating presentation information for displaying related information corresponding to the object at a position specified based on the position of the object, using a captured image captured by the imaging means;
Image display means for displaying an image including the generated presentation information;
A distance acquisition means for acquiring a distance between the object and the information processing apparatus;
The generating unit generates the presentation information by changing an amount of information included in the presentation information when the captured image and the distance acquired by the distance acquisition unit are larger than a predetermined value. ,
Information processing device. The generation unit generates the presentation information by changing a display form of the presentation information when the acquired distance becomes larger than the predetermined value.
The information processing apparatus according to claim 1. The generating means generates the presentation information by reducing the amount of information included in the presentation information when the acquired distance becomes larger than the predetermined value.
The information processing apparatus according to claim 1.   The information processing apparatus according to claim 1, wherein the object is a marker. The size of the marker is known;
The distance obtaining means obtains the size of a marker region that is an area occupied by the marker, and determines the distance using the obtained size of the marker region and the size of the marker,
The information processing apparatus according to claim 4. The size of the object is known;
The distance acquisition means obtains the size of the area occupied by the object, and determines the distance using the obtained area size and the size of the object,
The information processing apparatus according to claim 1. The captured image includes a plurality of the objects,
The generation means generates the presentation information for each object when the distance is less than a predetermined value, and the presentation information that is less than the number of the objects when the distance is greater than or equal to a predetermined value. Is characterized by generating
The information processing apparatus according to claim 1. The imaging means is an optical imaging means,
The distance acquisition unit obtains the distance from a focal length of the optical system imaging unit,
The information processing apparatus according to claim 1. The image display means is an optical image projection means,
The distance obtaining unit obtains the distance from a focal length of the optical system image projecting unit,
The information processing apparatus according to claim 1. The generating means controls the image display means to display the image in an area in the vicinity of the object and in which the displayed presentation information can be visually recognized. And
The information processing apparatus according to claim 1. The generating means controls the image display means to display the image in an area adjacent to the object and having a color gradient equal to or less than a predetermined value.
The information processing apparatus according to claim 1. The captured image includes a plurality of the objects,
The generating means groups a plurality of the objects into groups each including one or more objects, and associates one presentation information with each group,
The information processing apparatus according to claim 1. The generating means performs grouping based on related information corresponding to the object,
The information processing apparatus according to claim 12. The generating means generates related information common to the objects included in the group as presentation information of the group,
The information processing apparatus according to claim 13. The generating means performs grouping based on the position of the object,
The information processing apparatus according to claim 12. The generation unit generates related information common to at least two or more of objects included in the group as presentation information of the group,
The information processing apparatus according to claim 12. The generation means reduces the amount of information by deleting characters included in the presentation information when the acquired distance is greater than a predetermined value.
The information processing apparatus according to claim 12. When there are a plurality of combinations of objects included in the group, the generation unit obtains a similarity between objects constituting the group, and includes the group from the similarity obtained for each group. Determining a combination of objects to be selected,
The information processing apparatus according to claim 12. When there are a plurality of combinations of objects included in the group, the generation unit obtains the distance between objects constituting the group, and the object included in the group from the distance obtained for each group. Determining the combination of objects,
The information processing apparatus according to claim 12. An imaging means for imaging an object;
Generating means for generating presentation information for displaying related information corresponding to the object at a position specified based on the position of the object, using a captured image captured by the imaging means;
Image display means for displaying an image including the generated presentation information,
The captured image includes an object whose size is known,
The generating means obtains the size of an area occupied by the object in the captured image, and determines the amount of information included in the presentation information using the obtained area size and the size of the object. Characterized by the
Information processing device. The generation means generates the presentation information by reducing the amount of information included in the presentation information when the area becomes smaller than a predetermined value.
The information processing apparatus according to claim 20. A method for controlling an information processing apparatus,
Imaging an object with an imaging means;
Obtaining a distance between the object and the information processing device by a distance obtaining unit;
The generation unit generates presentation information for displaying related information corresponding to the target object at a position specified based on the position of the target object, using the captured image captured by the imaging unit, and Using the captured image and the distance acquired by the distance acquisition unit, generating the presentation information by reducing the amount of information included in the presentation information when the distance is greater than a predetermined value;
A step of displaying an image including the generated presentation information by an image display means,
A method for controlling an information processing apparatus. Computer
An imaging means for imaging an object;
Generating means for generating presentation information for displaying related information corresponding to the object at a position specified based on the position of the object, using a captured image captured by the imaging means;
Image display means for displaying an image including the generated presentation information;
A distance acquisition means for acquiring a distance between the object and the computer;
The generation unit generates the presentation information by reducing the amount of information included in the presentation information when the distance becomes larger than a predetermined value using the captured image and the distance acquired by the distance acquisition unit. It is characterized by
A computer program for functioning as an information processing apparatus.   A computer-readable storage medium in which the computer program according to claim 23 is stored.