JP5638789B2 - How to use signs - Google Patents

Description

  The present invention relates to a label and a label using method using the label.

  Due to advances in image recognition technology and moving object recognition technology, images of various scenes in the real world may become the target of image analysis in the future. For example, augmented reality (AR) is used for such image analysis. Augmented reality is a technology for adding and presenting information to a real environment using a computer (see, for example, Patent Document 1).

JP 2009-2111251

  When analyzing an image in the real world, there are cases where information regarding the location and time at which the image was captured is required for comparison or the like. In order to obtain such information, for example, it is conceivable to digitally attach information on location and time to an image by an imaging device. However, it is troublesome to input such information for each image. Further, if such information is taken out from the imaging device one by one, it is not immediate. Further, when an image is obtained by scanning a photograph, it is difficult to identify the place and time because information other than the image cannot be obtained.

  The present invention has been made in view of these problems, and an object thereof is to provide a sign that can provide information on a position and a direction by image processing.

  One embodiment of the present invention relates to a label. This sign is a sign to be installed at a predetermined location, and can be identified from a plurality of different line-of-sight directions. The position indication indicating the position of the sign and the sign from any of the plurality of gaze directions And a direction display indicating whether or not is visually recognized.

  According to this aspect, it is possible to obtain information regarding the position of the sign and the line-of-sight direction in which the sign is visually recognized by analyzing the image including the sign.

  Another aspect of the present invention is a method for using a label. This method includes the steps of imaging a sign, obtaining the position of the sign from the captured sign using an image recognition technique based on the position display, and using the image recognition technique from the captured sign Obtaining a line-of-sight direction in which the sign is imaged based on the display.

  It should be noted that any combination of the above-described constituent elements, or those obtained by replacing the constituent elements and expressions of the present invention with each other between apparatuses, methods, systems, computer programs, recording media storing computer programs, and the like are also included in the present invention. It is effective as an embodiment of

  ADVANTAGE OF THE INVENTION According to this invention, the information regarding the position and direction of the image containing a sign can be provided by the image processing of a sign.

It is a side view which shows the label | marker which concerns on 1st Embodiment. FIG. 2 is a partial development view illustrating an example of a spherical surface in FIG. 1. It is a perspective view which shows the label | marker which concerns on 2nd Embodiment. It is a mimetic diagram showing signs that a plurality of signs are arranged on a street. It is a schematic diagram which shows an example of the scenery seen from a windshield from the user in a vehicle. It is a block diagram which shows the function and structure of the imaging device and image processing apparatus which are mounted in a vehicle. It is a flowchart which shows a part of a series of processes in an imaging device and an image processing device. It is a schematic diagram which shows an example of the scenery seen through the windshield of a head vehicle.

  The present invention will be described below based on preferred embodiments with reference to the drawings. The same or equivalent components and members shown in the drawings are denoted by the same reference numerals, and repeated descriptions are appropriately omitted. In addition, the dimensions of the members in each drawing are appropriately enlarged or reduced for easy understanding. Also, in the drawings, some of the members that are not important for describing the embodiment are omitted.

(First embodiment)
The sign which concerns on 1st Embodiment is installed in the main places of streets, such as an intersection. By performing image processing on the two-dimensional image in which the sign is reflected, information on the position of the sign and the direction of the line of sight in which the two-dimensional image was taken is acquired. The information regarding the position and the line-of-sight direction acquired in this way is effectively used, for example, in augmented reality.

  FIG. 1 is a side view showing a marker 2 according to the first embodiment. The sign 2 includes a shaft portion 4 and a code presentation portion 6 attached to one end of the shaft portion 4. The other end of the shaft portion 4 is fixed to a predetermined location where the sign 2 on the street is to be installed, for example, an intersection. The sign presentation unit 6 is a member that can visually recognize the sign from all directions, in this example, a sphere, and its spherical surface 6a is different from a plurality of position displays that indicate the position of the marker 2 that can be identified from a plurality of different gaze directions. A direction display indicating from which line of sight direction the sign 2 is viewed is arranged. Here, the line-of-sight direction is the direction in which the imaging device faces the marker 2 when the marker 2 is imaged by the imaging device.

  The position display includes information indicating the latitude and longitude of the place where the sign 2 is installed. The direction display includes information indicating at least one of an azimuth angle and an elevation angle in the line-of-sight direction in which the sign 2 is visually recognized. Moreover, when visually recognizing at least two different signs 2 from the same viewpoint, the distance from the sign 2 to the viewpoint may be obtained from the information on the respective line-of-sight directions, for example, by triangulation.

  The plurality of position indications are arranged to serve as direction indications. That is, each position display is displayed on the spherical surface 6a as a position information code including information on the position of the sign 2, and the direction display is determined in a form depending on the visual recognition mode of a plurality of position information codes instead of being displayed as a code. It is done.

  The sign 2 is picked up by an image pickup device such as an in-vehicle camera mounted on a running vehicle, and the picked-up image of the sign 2 is processed by an image processing device connected to the image pickup device. This image processing apparatus acquires the position of the sign 2 imaged based on the position information code, and acquires the line-of-sight direction where the sign 2 was imaged from the way the position information code is reflected.

  FIG. 2 is a partial development view showing an example of the spherical surface 6a of FIG. In FIG. 2, a part of the spherical surface 6a corresponding to the line-of-sight direction is developed by following the Mercator projection. In FIG. 2, a viewing angle outside range 12 surrounded by an alternate long and short dash line indicates a range that does not enter the field of view of the imaging apparatus. The upper and lower parts indicated by broken lines in FIG. 2 indicate parts that cannot be identified because the code presentation unit 6 is a sphere when the code presentation unit 6 is viewed. In the example of FIG. 2, the spherical surface 6a of the code presentation unit 6 is divided into 12 blocks along the latitude direction.

On the spherical surface 6a, a plurality of position information codes 8 which are rows of white (0) black (1) blocks arranged along the latitude direction are continuously arranged in the meridian direction. The position information code 8 is shifted by one block in the counterclockwise direction every time one line is lowered. That is, the plurality of position information codes 8 are arranged along the spiral on the spherical surface 6a.
The position information code 8 has a predetermined length. This predetermined length is set so that the position information code 8 can be entered into the viewing angle when the imaging device visually recognizes the sign 2 without excess or deficiency. The example of FIG. 2 corresponds to the case where the position information code 8 of 6 blocks is input without excess or deficiency at a viewing angle of slightly less than 180 degrees.

  A code obtained by digitizing the position information code 8 with the white block as 0 and the black block as 1 (or vice versa) forms a so-called Hamming code. The image processing apparatus that processes the image of the sign 2 imaged by the image pickup apparatus extracts the hamming code from the image of the sign 2, detects and corrects the error, and acquires position information included therein.

  A viewing angle range 10 that is not hatched in FIG. 2 indicates a range that falls within the field of view of the imaging apparatus. In the state of FIG. 2, the position information code 8 included in the X line 14 from the top can be read without error correction. Further, the position information code 8 included in the X + 1th row 16 can be read with error correction. In addition, the position information code 8 below the X + 2 line 18 cannot be identified.

  Here, it is assumed that the imaging device (viewpoint) moves and the azimuth angle changes, and the viewing angle range 10 moves to the right by one block from the state of FIG. Then, the position information code 8 included in the X line 14 can be read without error correction, and the position information code 8 included in the X + 1 line 16 can be read without error correction. Accordingly, it is possible to identify that the azimuth angle has relatively changed from the initial state by 360 degrees / 12 = 30 degrees.

  Reference information regarding an absolute azimuth, for example, the azimuth angle itself, may be included in the position information code 8 in a cycle shorter than the number of rows (5 rows in the example of FIG. 2) entering the field of view of the imaging device in the meridian direction. In this case, not only the relative azimuth angle but also the absolute azimuth angle can be identified from the relative azimuth angle and the reference information. Moreover, since such information is not included in all the position information codes 8, the amount of information presented on the spherical surface 6a can be kept small.

The elevation angle is determined from information indicating how far the identifiable portion as shown by a solid line in FIG. In other words, using the fact that the sign presentation unit 6 is a sphere, an indistinguishable part is generated in at least one of the upper part and the lower part of the image of the code presentation part 6 imaged by the imaging device. Identify
Alternatively, the position information code 8 may include the corresponding elevation angle information.

  According to the sign 2 according to the present embodiment, by extracting the position information and the line-of-sight direction of the sign 2 from the image where the sign 2 is imaged, the position of the sign 2 and the position / line-of-sight direction where the sign 2 was imaged are increased. It can be obtained with accuracy. Therefore, the base point and identity of other objects in the image in which the sign 2 is reflected can be identified with higher accuracy. The acquired information is preferably used when, for example, an annotation is presented on a building or the like around the sign 2 in augmented reality. Further, as can be seen in the application examples described later, such information is usefully used in various situations.

The sign 2 may further include a display of additional information associated with the place where the sign 2 is to be installed. The additional information is, for example, advertisement information such as the name of a commercial facility in the vicinity of the place where the sign 2 is to be installed and information on events being performed at the commercial facility. In this case, the sign 2 may also serve as a signboard or an advertisement installed on the street. That is, the position display, direction display, and additional information may be embedded in a signboard or an advertisement.
Further, the additional information may be information related to the area such as the traffic accident rate in the area where the sign 2 is installed.

The sign 2 may further include a time display. In this case, the sign 2 is realized by a display device such as a liquid crystal display or an LED. The position display, the direction display, and the time display are displayed on the display device. Thereby, not only position and direction but also time information can be extracted by image analysis of the sign 2. If the time information is known, for example, it is possible to collect images obtained by imaging the marker 2 from different gaze directions at a certain time using the time information as a key.
Even if the marker 2 does not include a time display or does not depend on the time display of the marker 2, images obtained by capturing a certain marker 2 from different line-of-sight directions can be collected.

  Further, a search engine such as Google (registered trademark) may specify the location of the image including the sign 2 from the position display of the sign 2 and create an index based on the specified location. Thus, the sign 2 according to the present embodiment functions as an anchor for the virtual world provided in the real world.

(Second Embodiment)
FIG. 3 is a perspective view showing a sign 24 according to the second embodiment. The sign 24 includes a shaft portion 26 and a code presentation portion 28 attached to one end of the shaft portion 26. The other end of the shaft portion 26 is fixed to a place on the street where the sign 24 is to be installed. The code presentation unit 28 is a spherical member, and a QR code (registered trademark) 30 corresponding to each line-of-sight direction is printed on the spherical surface 28a.

  The QR code 30 is a display including position information related to the position where the sign 24 is installed and direction information related to the line-of-sight direction where the sign 24 is visually recognized. An imaging device such as a camera captures the sign 24, and the image processing device recognizes and decodes the QR code 30 by processing the captured image, and extracts both pieces of information.

  According to the sign 24 according to the second embodiment, the same effects as those of the sign 2 according to the first embodiment can be obtained.

Two application examples using the marker 2 according to the first embodiment will be described below.
(Application 1)
FIG. 4 is a schematic diagram showing a state in which a plurality of signs 2 are arranged on the street. In the subsequent drawings, the position information code 8 on the spherical surface 6a is omitted for the sake of simplicity. A user is driving a vehicle 32 including an imaging device and an image processing device (both not shown in FIG. 4) connected thereto. This imaging device captures the sign 2 on the street, and the image processing device extracts information on the position and line-of-sight direction from the image. The image processing apparatus acquires the names of buildings and commercial facilities around the sign 2 from the database using the extracted position and line-of-sight direction as keys, and presents them as name annotations on the windshield of the vehicle 32. Further, when additional information such as an advertisement associated with the sign 2 is extracted from the sign 2, it is associated with the sign 2 and presented as an advertisement annotation on the windshield.

  FIG. 5 is a schematic diagram illustrating an example of a landscape that can be seen from the user in the vehicle 32 through the windshield 34. Although the windshield 34 is see-through, the windshield 34 is configured to be able to present an annotation at any necessary location. Such a windshield 34 is comprised, for example using the well-known organic EL (Electro-Luminescence) technique.

  As a result of the imaging of the sign 2 by the imaging device mounted on the vehicle 32 and the image processing by the image processing device, a position annotation 36, a name annotation 38, and an advertisement annotation 40 are presented on the windshield 34. The position annotation 36 indicates the position on the map where the sign 2 is installed. The name annotation 38 indicates the name of the building / commercial facility in the visual field space determined by the position / line-of-sight direction acquired from the image of the sign 2. The advertisement annotation 40 indicates the advertisement in association with the position annotation 36 when the sign 2 includes display of advertisement information.

  FIG. 6 is a block diagram illustrating functions and configurations of the imaging device 42 and the image processing device 44 mounted on the vehicle 32. Each block shown here can be realized in hardware by an element such as a CPU (central processing unit) or a mechanical device, and in software by a computer program or the like. Describes functional blocks realized by collaboration. Accordingly, it is understood by those skilled in the art who have touched this specification that these functional blocks can be realized in various forms by a combination of hardware and software.

  The imaging device 42 is a digital camera, for example, and captures an image of the sign 2. The image processing device 44 includes an image acquisition unit 46, an information extraction unit 48, an auxiliary information acquisition unit 50, an auxiliary information DB (Data Base) 52, an advertisement acquisition unit 54, an advertisement DB 56, and an image update unit. 58, an image library 60, and a display control unit 62.

  The image acquisition unit 46 acquires an image captured by the imaging device 42. The information extraction unit 48 extracts the position of the marker 2 from the image of the marker 2 acquired by the image acquisition unit 46 based on the position information code 8 using a known image recognition technique. In addition, the information extraction unit 48 extracts the line-of-sight direction in which the marker 2 is imaged based on the direction display determined in a manner depending on the visual recognition mode of the plurality of position information codes 8 from the image of the marker 2. For example, the method described with reference to FIG. 2 is used for the extraction of the line-of-sight direction. The information extraction unit 48 passes the extracted position information and line-of-sight direction information to the auxiliary information acquisition unit 50, the image update unit 58, and the display control unit 62. The display control unit 62 presents the passed position information as the position annotation 36 on the windshield 34 in association with the sign 2.

  In addition, when the sign 2 includes display of advertisement information, the information extraction unit 48 extracts the information and passes it to the display control unit 62. The display control unit 62 presents the advertisement information as the advertisement annotation 40 on the windshield 34.

  The auxiliary information acquisition unit 50 searches the auxiliary information DB 52 using the passed position information and line-of-sight direction information as keys, and auxiliary information related to the environment in the visual field space determined by the position and the line-of-sight direction, particularly the visual field space. Acquire name assistance information indicating the name of the building / commercial facility. The auxiliary information acquisition unit 50 passes the acquired name auxiliary information to the display control unit 62. The display control unit 62 presents the passed name assistance information as the name annotation 38 on the windshield 34 in association with the target building / commercial facility.

  Regarding the advertisement, the information extraction unit 48 passes the position information and the information on the line-of-sight direction to the advertisement acquisition unit 54, and the advertisement acquisition unit 54 searches the advertisement DB 56 using the passed position information and information on the line-of-sight direction as a key. Further, advertisement auxiliary information related to a commercial facility in the visual field space determined by the line-of-sight direction may be acquired. The advertisement acquisition unit 54 may pass this advertisement auxiliary information to the display control unit 62, and the display control unit 62 may present the advertisement auxiliary information as the advertisement annotation 40 on the windshield 34. In this case, the advertisement can be easily updated by rewriting the data in the advertisement DB 56.

  In addition, the position information and the information on the line-of-sight direction extracted by the information extraction unit 48 are also preferably used for updating the real world image. The image update unit 58 searches the image library 60 using the received position information and line-of-sight direction information as keys, and acquires past images determined by the position and line-of-sight direction. Then, the image update unit 58 compares the past image with the current image acquired by the image acquisition unit 46, and if there is a difference, updates the past image with the current image acquired by the image acquisition unit 46. In this case, the image can be automatically updated even if the landscape changes due to the renovation of the building or the reorganization.

  In addition, when moving image data is sent from the imaging device 42, the information extraction unit 48 extracts an image in which the sign 2 is hidden by a moving object such as a human being and cannot be seen from a series of images constituting the moving image data. You may have the function to do. Thereby, it is possible to cope with the case where the sign 2 is hidden by a moving object such as a human being.

  FIG. 7 is a flowchart showing a part of a series of processes in the imaging device 42 and the image processing device 44. The imaging device 42 images the sign 2 (S202). The information extraction unit 48 acquires the position of the marker 2 from the captured marker 2 using the image recognition method based on the position display (S204). The information extraction unit 48 obtains the line-of-sight direction in which the marker 2 is imaged based on the direction display using the image recognition technique from the imaged marker 2 (S206). The auxiliary information acquisition unit 50 acquires auxiliary information related to the environment in the visual field space determined by the acquired position and line-of-sight direction (S208). The display control unit 62 virtually reflects the acquired auxiliary information in the visual field space and presents it to the user (S210).

  According to this application example, auxiliary information regarding the real environment in the visual field space can be provided to the user more easily and accurately using the marker 2 as an anchor in augmented reality. Additional information such as advertisements can also be presented.

  Note that the image acquisition unit 46 uses image data obtained by scanning a photograph taken by another imaging device using an analog technology, for example, a camera, instead of an image as digital data captured by the imaging device 42. You may get it.

(Application example 2)
In the application example 2, the sign 2 is installed at a predetermined location along the track, for example, a railroad crossing. The leading vehicle of the train running on the track includes an imaging device and an image processing device connected thereto. FIG. 8 is a schematic diagram illustrating an example of a landscape seen through the windshield 64 of the leading vehicle. The windshield 64 is configured in the same manner as the windshield 34 of the first application example.

  Suppose a train reaches railroad crossing 66. A sign 2 is installed at the level crossing 66. The imaging device captures the sign 2 and the image processing device extracts information on the position / gaze direction from the image. The image processing apparatus searches the image DB using the extracted position / gaze direction as a key, and extracts a normal image in a safe state of the level crossing 66. The image processing device compares the extracted normal image with the image of the crossing 66 that is currently imaged by the imaging device. When the image processing apparatus detects a foreign object such as the car 68 on the railroad crossing as a difference, the image processing apparatus displays a danger annotation 70 for calling attention in association with the car 68.

According to this application example, the driver can find out the foreign matter on the railroad crossing 66 at an early stage by the danger annotation 70, which is useful for preventing an accident.
The shaft portion 4 of the sign 2 may be shared with the railroad crossing alarm 72. That is, the sign presentation unit 6 may be provided at the top of the railroad crossing alarm 72.

  Although the present invention has been described based on the embodiments, the embodiments merely show the principle and application of the present invention, and the embodiments are defined in the claims. Needless to say, many modifications and arrangements can be made without departing from the spirit of the present invention.

  In the first and second embodiments, a solar cell may be provided on the spherical surface of the sign display section of the sign, and a position information code may be presented by driving a self-luminous element such as an LED with electric power generated there. Or you may share a code | symbol presentation part with the sign of a bus stop.

  In the first and second embodiments, the case where the sign is installed on the ground has been described. However, the present invention is not limited to this. For example, the sign according to the embodiment may be installed on the sea or in the air.

  In 1st and 2nd embodiment, although the case where the code | symbol presentation part 6 was spherical was demonstrated, it is not restricted to this. For example, the code presentation unit 6 may have a rugby ball shape.

  2 sign, 4 axis part, 6 code presentation part, 8 position information code, 24 sign, 26 axis part, 28 code presentation part, 30 QR code.

Claims (6)

A method executed by an imaging device and an image processing device,
The imaging device is a sign to be installed at a predetermined location, and has a code presentation unit that can visually recognize a code from all directions, and the code presentation unit includes a sign that can be identified from a plurality of different gaze directions. Imaging a sign comprising information in which a position indication indicating a position is displayed as a signal;
The image processing apparatus acquires a position of the sign from the captured sign based on the position display using an image recognition technique;
The sign processing method comprising: the image processing device including a step of acquiring a line-of-sight direction in which the sign is picked up from how the position display is reflected using an image recognition technique from the picked up sign.   In the step of obtaining the line-of-sight direction, the line-of-sight direction in which the sign is visually recognized by the image processing device based on a distance from an upper part or a lower part of the picked-up sign to an identifiable part of the picked-up sign The method of using a sign according to claim 1, wherein the angle of elevation is determined. The image processing apparatus acquires auxiliary information related to the environment in the visual field space determined by the acquired position and line-of-sight direction;
The method for using a sign according to claim 1, further comprising a step in which the image processing apparatus virtually reflects the acquired auxiliary information in the visual field space and presents it to a user.   The said image processing apparatus further includes the step which collects the image which image | photographed the label | marker from a different gaze direction using the acquired position and gaze direction, The method of any one of Claim 1 to 3 characterized by the above-mentioned. How to use the sign.   The image processing apparatus further includes a step of specifying a location of an image including the sign from the acquired position and creating an index in a search engine based on the specified location. The label | marker usage method in any one of. A sign that is to be placed at a predetermined location, and has a sign presentation unit that can visually recognize the sign from all directions, and the sign presentation part indicates the position of the sign that can be identified from a plurality of different gaze directions The imaging device realizes a function of imaging a sign having information displayed as a signal,
A function of acquiring the position of the sign based on the position display from an imaged sign using an image recognition method;
A computer program for causing an image processing apparatus to realize a function of acquiring a line-of-sight direction in which a sign is imaged from a way of displaying the position display using an image recognition technique from the imaged sign.

Images