JP7335726B2 - Information display device - Google Patents

Description

The present invention relates to an information display device that displays information using augmented reality technology.

Augmented reality (AR) is a method of displaying information obtained from an actual photographed image by superimposing additional information associated with the object in advance on the photographed image of a predetermined object photographed by a camera. can be further extended.

When a large amount of additional information is displayed on a captured image, the additional information may be displayed densely in a narrow range so as to overlap each other. In such a case, there is an inconvenience that the additional information becomes difficult to see.

Conventionally, the following methods (1) to (4), for example, have been used to eliminate such inconveniences.
(1) A method to enlarge and display a photographed image by operating a button, touch panel, etc. (2) A method to avoid overlapping of additional information by hiding additional information in the distance (3) Align additional information with 3D space (4) Display POIs (Points of Interest) on the map according to the map enlargement ratio. A method of displaying them all together (diversion of a method not found in augmented reality)
In addition, Patent Literatures 1 and 2 disclose techniques for changing the size and layout of additional information so that pieces of additional information displayed in an AR or map display do not overlap each other.

JP 2015-138428 A JP-A-2006-003958

In the method (1), the operation of enlarging the photographed image must be performed manually. In particular, when using the AR function with a wearable device such as a head-mounted display, it is troublesome to perform the enlargement operation. As described above, the method (1) has a problem of poor operability.

In the method (2), since the additional information in the distance is not displayed, there is a problem that in order to see the additional information, it is necessary to approach the object on which the additional information is to be superimposed and displayed.

In the method (3), since the additional information in the distance is displayed in a small size, in order to see the additional information displayed in a reduced size, the photographed image must be enlarged or the subject must be approached. not. Therefore, there is the same problem as the method (1) or (2). In addition, when the displayed additional information is operated using a touch panel or the like, the displayed additional information is too small for photographing from a distance, making the operation difficult.

In the method (4), since the POIs are grouped together, it is necessary to manually enlarge the map in order to see each piece of additional information.

Also, with the techniques disclosed in Patent Documents 1 and 2, the layout of additional information is automatically changed, so there is a problem that the additional information is displayed outside the position intended by the creator of the AR screen content. there were.

As described above, with the techniques (1) to (4) and the techniques disclosed in Patent Literatures 1 and 2, it is difficult to properly eliminate overlap of additional information without much effort.

An object of one aspect of the present invention is to properly eliminate overlapping of additional information without taking time and effort.

In order to solve the above problems, an information display device according to an aspect of the present invention includes a photographing unit configured to output a photographed image obtained by photographing a specific object associated with a plurality of pieces of additional information; Determining the degree of overlap of a plurality of pieces of additional information present in a specific area set to identify the object of interest and the additional information displayed on the object, and determining the degree of overlap based on the degree of overlap an enlargement unit for enlarging the photographed image at the enlargement ratio; and a coordinate position of the additional information corrected based on the enlargement ratio in the enlarged photographed image. a synthesizing unit for synthesizing the additional information so as to overlap the additional information to generate a synthetic image; and a display unit for displaying the synthetic image.

According to the above configuration, the additional information is superimposed on the coordinates corrected according to the enlargement ratio on the photographed image enlarged by the enlargement ratio according to the degree of overlap of the additional information in the specific area. As a result, it is possible to set an enlargement rate to the extent that the additional information does not overlap according to the degree of overlap.

The information display device may further include an operation section that receives an operation, and a specific area setting section that sets the specific area according to the received operation.

According to the above configuration, a desired specific area can be provided in the captured image by setting the specific area.

In the information display device, each piece of additional information has a certain size, and the enlargement ratio calculation unit determines the degree of overlap based on the number of pieces of the additional information existing in the specific area, and determines the degree of overlap. The magnification may be calculated based on the degree.

According to the above configuration, it is possible to easily determine the degree of overlap.

In the information display device, the magnification calculation unit determines the degree of overlap based on a distance between coordinates of reference points serving as references for displaying the additional information existing in the specific area, and calculates the degree of overlap. You may calculate the said magnification based on.

According to the above configuration, the overlapping degree can be determined more accurately based on the distance between the coordinates of the reference points.

According to one aspect of the present invention, overlapping of additional information can be properly resolved without taking time and effort.

1 is a block diagram showing the hardware configuration of an information display device according to one embodiment of the present invention; FIG. It is a block diagram which shows the system configuration|structure of the said information display apparatus. It is a figure which shows the example of a calculation of the magnification by the magnification calculation part of the said information display apparatus. It is a figure which shows the example of another calculation of the expansion rate by the said expansion rate calculation part. 10 is a captured image captured by the camera of the information display device, in which additional information is densely packed and difficult to see. It is a photographed image in which additional information is displayed according to the 3D space, photographed by the camera of the information display device. FIG. 4 is a diagram showing the information display device displaying a photographed image including a specific area photographed by a camera; 8 is a diagram showing the information display device displaying an enlarged image obtained by enlarging the photographed image shown in FIG. 7; FIG. FIG. 10 is a diagram showing the information display device displaying another photographed image including a specific area photographed by a camera;

[Embodiment]
An embodiment of the present invention is described below with reference to FIGS. 1 to 9. FIG.

First, the hardware configuration of the information display device 100 will be described. FIG. 1 is a block diagram showing the hardware configuration of an information display device 100 according to this embodiment.

The information display device 100 is a communication terminal device on which a dedicated OS (Operating System) is installed. The information display device 100 is implemented by a mobile device such as a portable terminal device such as a tablet terminal, or a wearable device such as smart glasses or a head-mounted display. The information display device 100 also has a display function based on augmented reality (AR) technology for superimposing and displaying additional information on a photographed image of a predetermined object.

As shown in FIG. 1, the information display device 100 includes a CPU (Central Processing Unit) 11, a main memory 12, a ROM (Read Only Memory) 13, an auxiliary memory 14, a camera 15 (shooting unit), an operation A portion 16 and a display portion 17 are provided.

The CPU 11 is a processing device that executes a system program for the information display device 100 . Specifically, when executing the system program, the CPU 11 receives data from the main memory 12, the auxiliary memory 14, the operation unit 16, etc., and outputs the result of calculation or processing to the data to the main memory 12, the auxiliary memory 12, or the like. The data is output to the memory 14, the display unit 17, and the like.

The main memory 12 is a memory that constitutes the main storage device in the information display device 100, and is composed of a DRAM (Dynamic Random Access Memory).

In addition to the OS, the ROM 13 stores programs essential for the operation of the information display device 100, such as a BIOS (Basic Input Output System) that is executed when the information display device 100 is activated or reset.

Auxiliary memory 14 is provided to store various data, application programs, etc. to be held by information display device 100 . The auxiliary memory 14 is composed of flash memory, for example. Incidentally, when the ROM 13 is composed of a large-capacity flash ROM, the auxiliary memory 14 may be composed integrally with the ROM 13 .

The camera 15 is a photographing device that photographs an object to be photographed and outputs a photographed image. The captured image is displayed on the display unit 17 .

The operation unit 16 is an input device for a user to perform an input operation. The operation unit 16 receives an operation by a user and outputs an operation signal according to the operation. As the operation unit 16, a touch panel is used when the information display device 100 is a handy device such as a tablet terminal. If the information display device 100 is a glasses-type wearable device such as smart glasses, the operation unit 16 detects buttons provided on the frame, a touch sensor, a sensor capable of gesture operation, and the line of sight of the user. An eye tracking sensor or the like is used. The user's line of sight may be detected by the camera 15 instead of the eye tracking sensor.

The display unit 17 is provided to display a screen for realizing basic operations of the information display device 100, a screen displayed as a result of execution of an application program, and the like. As the display unit 17, a flat display panel such as a liquid crystal display panel is used when the information display device 100 is a handy device such as a tablet terminal. As the display unit 17, when the information display device 100 is a glasses-type wearable device such as smart glasses, a display (such as a liquid crystal display) or a projector provided in the lens portion is used.

Next, the system configuration of the information display device 100 will be described. FIG. 2 is a block diagram showing the system configuration of the information display device 100. As shown in FIG.

As shown in FIG. 2 , the information display device 100 includes an AR information storage section 1 and an information addition section 2 .

The AR information storage unit 1 stores additional information, coordinate information, and object information. The additional information and the coordinate information are stored in the AR information storage unit 1 in association with each piece of object information. The AR information storage unit 1 is provided in the auxiliary memory 14, for example.

The additional information is associated with a predetermined target object so as to be superimposed on the captured image captured by the camera 15 and displayed. Additional information includes various contents such as character information and image information. The coordinate information is coordinates representing the position where the additional information is displayed on the object. The target object information is information specifying the target object on which additional information is superimposed and displayed.

When additional information is displayed using a marker (marker-type vision-based AR), the coordinate information is obtained by calculation based on information associated with the marker and information associated with the additional information.

The information associated with the marker is positional information of a point (first reference point) that serves as a reference for displaying additional information with respect to the coordinates of the marker detected in the captured image, and is the X direction from the coordinates of the marker and It is represented by the distance in the Y direction. If the coordinates of the marker are (Xm, Ym), the distance in the X direction is Xa, and the distance in the Y direction is Ya, the coordinates of the first reference point are expressed as (Xm+Xa, Ym+Ya).

Information attached to the additional information is positional information of a point (second reference point) that serves as a reference for displaying the additional information with respect to the origin coordinates of the additional information. is represented by the distance of Assuming that the coordinates of the origin are (Xr, Yr), the distance in the X direction is Xb, and the distance in the Y direction is Yb, the coordinates of the second reference point are expressed as (Xr+Xb, Yr+Yb). . The origin coordinates are set at predetermined positions according to the shape of the additional information.

Additional information is displayed at a position where the second reference point coincides with the first reference point. Coordinate information for displaying additional information is expressed as origin coordinates (Xr, Yr)=(Xm+Xa-Xb, Ym+Ya-Yb) using the coordinates of the first and second reference points. This coordinate calculation is performed by a coordinate calculation section (not shown) provided in the information addition section 2 .

Further, when additional information is displayed without using a marker (markerless vision-based AR), the coordinate information is based on information attached to an object image obtained by photographing the object in advance and information attached to the additional information. is obtained by calculating The information attached to the object image is defined as positional information of a point that serves as a reference for displaying additional information with respect to the coordinates of the object image detected in the captured image. Further, the information attached to the additional information is the same as the information attached to the additional information of the marker-type vision-based AR. Therefore, the coordinate information of the markerless vision-based AR is calculated by the above-described coordinate calculation unit and represented as the origin coordinates of the additional information in the same manner as the coordinate information of the marker-type vision-based AR.

In the case of the marker-type vision-based AR, the object information includes a marker for specifying the position where additional information is to be displayed with respect to the object. In the case of the markerless vision-based AR, the object information includes an object image obtained by photographing the object in advance.

The information addition unit 2 performs processing for displaying additional information at an appropriate position on the photographed image of the object displayed on the display unit 17 . Further, when a plurality of pieces of additional information superimposed on a photographed image are densely overlapped, the information addition unit 2 enlarges the photographed image at an enlargement ratio according to the degree of overlap of the additional information, thereby adding additional information to the photographed image. are distributed and displayed.

In order to perform the above processing, the information addition unit 2 includes a camera posture estimation unit 3, a coordinate conversion unit 4, a specific area setting unit 5, a specific area storage unit 6, a monitor unit 7, and an enlargement ratio calculation unit. 8 and an image synthesizing unit 9 .

In the case of the marker-type vision-based AR, when the camera posture estimation unit 3 recognizes a marker stored in the AR information storage unit 1 in the captured image output from the camera 15, the additional information associated with the marker is converted to the corresponding additional information. Additional information to be displayed on the captured image is specified. Further, in the case of markerless vision-based AR, if the camera posture estimation unit 3 determines that the captured image matches the target object image stored in the AR information storage unit 1, the camera posture estimation unit 3 associates the captured image with the target object image. It specifies that the received additional information is the additional information to be displayed on the captured image.

In addition, the camera posture estimation unit 3 calculates (estimates) posture information regarding the posture of the camera 15 photographing the object. In the case of marker-type vision-based AR, when the marker in the captured image is tilted with respect to the normal position of the marker stored in the AR information storage unit 1, the camera posture estimation unit 3 measures the tilt angle. Then, the shooting direction is calculated as posture information based on the angle. Further, in the case of the markerless vision-based AR, the camera orientation estimation unit 3 measures the inclination angle when the photographed image is tilted with respect to the object image, and determines the orientation of the photographing direction based on the tilt angle. Calculate as information.

The coordinate conversion unit 4 converts the coordinate information (coordinates of the additional information) stored in the AR information storage unit 1 into orientation information so that the additional information is displayed in an orientation corresponding to the direction in which the object is photographed by the camera 15 . (imaging direction), and outputs the transformed coordinates (transformed coordinates). In other words, the coordinate conversion unit 4 calculates the position in the captured image to display the additional information based on the estimated shooting direction.

The specific area setting unit 5 sets a specific area in the captured image according to the user's operation on the operation unit 16, and stores it in the specific area storage unit 6 as information of the specific area. The specific region is a region for specifying an object of interest to the user and additional information superimposed on the object in the captured image. For example, the specific region is indicated by broken lines in a rectangular shape, but the line type and shape are not limited.

Note that the specific area may be set in advance by a specific area setting tool provided outside the information display device 100 instead of the specific area setting unit 5 . Such a setting tool may be provided, for example, in an editing tool for editing AR screen content. When editing screen content using such an editing tool, a specific area is also set, and the set specific area is transferred to the information display device 100 via a cable and stored in the specific area storage unit 6. .

Also, the specific area may be set in the camera 15 or the display unit 17 . The camera 15 displays the specific area on the captured image of the object, and outputs the captured image in which the specific area is set by the user's operation to the monitor section 7 . The display unit 17 also displays a specific image on a composite image (an image obtained by combining additional information with a photographed image), which will be described later, and outputs to the monitor unit 7 a specific area set by the user's operation of the touch panel or the like. Here, in a state where the specific area is not set, the enlargement ratio calculation unit 8, which will be described later, does not calculate the enlargement ratio. Therefore, in the above state, the captured image in the synthesized image has the same size as the captured image output from the camera 15 .

When setting the specific area on the display unit 17 as described above, the specific area may be set using the editing tool described above. For example, the specific area is set as described above for the display screen of the display unit 17 displayed within the editing tool, or the display unit 17 is virtually provided on the display screen of the editing tool, and the display screen is On the other hand, a specific area is set as described above.

The specific area may be set in advance by the user, or may be set by the user's operation when the additional information is displayed. Therefore, the specific area setting unit 5 makes it possible to change the specific area by providing the following user interfaces (a) to (d). User interfaces (a) and (b) are suitable for handheld devices such as tablet terminals, and user interfaces (c) and (d) are suitable for wearable devices such as smart glasses. The user interface of (e) is suitable for both handheld and wearable devices.
(a) Display a frame indicating a specific area on the captured image on the display unit 17, and move the displayed frame by a touch operation such as dragging.
(b) Change the size of the specific area by operating the slide bar, specifying two points in the specific area (rectangular area), or performing other touch operations.
(c) Recognizing the hands and fingers captured by the camera 15 and changing the size of the area by a specific gesture operation.
(d) User's line of sight is detected by the camera 15 and an eye tracking sensor provided as the operation unit 16, and the specific area is moved to the position of the line of sight.
(e) Switch between a plurality of specific areas prepared in advance by button operation or gesture operation.

The specific area storage section 6 stores the specific area set by the specific area setting section 5 . The specific area storage unit 6 is provided in the main memory 12 or the auxiliary memory 14, for example.

The monitor unit 7 refers to the specific area information stored in the specific area storage unit 6, monitors the specific area in the captured image, and extracts additional information in the specific area. Specifically, while monitoring the specific region, the monitor unit 7 recognizes additional information indicating that the coordinates of the second reference point exist in the specific region based on the converted coordinates from the coordinate conversion unit 4, and A list in which the coordinate values of the additional information in the captured image and the unique information representing the additional information (name, code, etc. given to the additional information) are associated with each other is output.

Based on the additional information and the coordinates extracted by the monitor unit 7, the magnification calculator 8 determines the degree of overlap between the additional information (degree of overlap), and determines the magnification of the captured image based on the determined degree of overlap. calculate. For example, on the premise that the size of the additional information is constant, the enlargement ratio calculator 8 specifies the degree of overlap based on the number of pieces of additional information (the number of coordinates of the second reference points) in the specific area. Alternatively, the enlargement ratio calculator 8 determines the degree of overlap based on the distance between the coordinates of the second reference point of the additional information.

Note that if the size of the additional information is not constant, the enlargement ratio calculation unit 8 acquires information on the size of each piece of additional information from the AR information storage unit 1, and adds the information on the size to obtain the above 2 Determine the degree of overlap by one of three methods. Therefore, in this case, the AR information storage unit 1 must also store information on the size of each piece of additional information.

Further, the enlargement factor calculator 8 may smoothly change the enlargement factor by filtering the calculated enlargement factor so as to gradually enlarge the photographed image.

Examples of the filtering process include the following processes (A) to (D).
(A) Average value filter When the magnification rate varies irregularly (for example, 1x, 2x, 1x, 2x, etc.) along the time axis, the average position is moved. For example, when the enlargement ratio is calculated every second, the average of a predetermined number (eg, four) of enlargement ratios is calculated while shifting by one second. It is possible to appropriately set the number of enlargement ratios for which the average value is taken. As a result, by averaging the enlargement ratios that have suddenly increased or decreased like noise and the surrounding enlargement ratios that do not change significantly, changes in the enlargement ratio can be smoothed.
(B) Hysteresis Characteristic Filter Different threshold values are set for when the enlargement rate is increasing and when it is decreasing. When the magnification rate rises, the magnification rate changes when the high threshold value is exceeded, and when the magnification rate falls, the magnification rate changes when the low threshold value is exceeded. By providing a plurality of thresholds in this manner, the enlargement ratio can be changed stepwise.
(C) Quantitative Change Filter Using the calculated enlargement ratio as a target value, the enlargement ratio is changed from the current value toward the target value by a constant amount of change.
(D) PID control filter P (proportional) control determines the rate of change of the enlargement rate so as to be proportional to the magnitude of the difference between the initial value and the target value.

I (integral) control integrates and corrects the error when the enlargement rate is stabilized at a value offset from the target value only by P control.

D (differential) control controls the enlargement rate so as to suppress changes in the enlargement rate.

When judging the degree of overlap by the number of pieces of additional information, the photographed image is immediately enlarged at the maximum magnification without performing filtering. You may approach a target value. In this way, it is possible to appropriately combine the method of judging the degree of overlap and the filtering process.

Further, the magnification calculator 8 may clip the magnification. Specifically, the enlargement factor calculator 8 clips the enlargement factors to exclude extremely large enlargement factors and to exclude extremely small enlargement factors (for example, 1.1 times).

The image synthesizing unit 9 enlarges the photographed image from the camera 15 and synthesizes the enlarged photographed image so as to superimpose the additional information. The image synthesis unit 9 has an enlargement processing unit 91 (enlargement unit) and a synthesis processing unit 92 (synthesis unit) in order to perform this processing.

The enlargement processing unit 91 enlarges the captured image centering on the specific area at the enlargement ratio calculated by the enlargement ratio calculation unit 8, and outputs an enlarged image. Further, the enlargement processing unit 91 corrects the transformed coordinates from the coordinate transforming unit 4 according to the enlargement ratio. Specifically, the enlargement processing unit 91 changes the values of the reference point coordinates of each additional information to By multiplying by the magnification of , the corrected coordinates are calculated.

The synthesis processing unit 92 synthesizes additional information acquired from the AR information storage unit 1 with the captured image (enlarged image) enlarged by the enlargement processing unit 91 . The synthesis processing unit 92 arranges the additional information at a position where the reference point coordinates of the additional information match the corrected coordinates in the synthesis.

Next, the operation of the information display device 100 configured as described above will be described.

FIG. 3 is a diagram showing an example of calculation of an enlargement factor by the enlargement factor calculator 8. As shown in FIG. FIG. 4 is a diagram showing another calculation example of the enlargement ratio by the enlargement ratio calculator 8. In FIG.

First, the camera posture estimator 3 calculates the shooting posture of the camera 15 based on the captured image output from the camera 15 and outputs it as posture information. In calculating the shooting posture, the camera posture estimation unit 3 uses the tilt angle and size of the captured image (marker or target object image) with respect to the target object information (marker or target object image) acquired from the AR information storage unit 1 as the target object information. and outputs the tilt angle and the shooting distance representing the shooting direction as posture information.

The coordinate transformation unit 4 transforms the coordinates of the coordinate information acquired from the AR information storage unit 1 based on the shooting direction and shooting distance given as the posture information. The converted coordinates are given to the monitor section 7 and the enlargement processing section 91 as converted coordinates.

On the other hand, the operation unit 16 receives an operation by the user and sets a specific area on the captured image. The operation unit 16 stores the set specific area in the specific area storage unit 6 .

The monitor unit 7 superimposes the specific area read out from the specific area storage unit 6 on the photographed image acquired from the camera 15, and recognizes additional information existing in the specific area based on the converted coordinates from the coordinate conversion unit 4. do. Even if the additional information partially exists outside the specific area, the monitor unit 7 recognizes that the additional information exists in the specific area if the reference point coordinates exist within the specific area. do. The monitor unit 7 outputs the information representing the recognized additional information and the converted coordinates of the additional information in association with each other.

Based on the converted coordinates of the additional information recognized by the monitor 7, the enlargement ratio calculator 8 determines the degree of overlapping of the additional information in the specific area.

For example, as shown in FIG. 3, the enlargement factor calculator 8 sets the enlargement factor M when the number N of pieces of additional information 21 in the specific region 20 is equal to or less than the threshold Th and when the number N exceeds the threshold Th. make different. Specifically, the enlargement factor calculator 8 sets the enlargement factor M to a normal value V when N≦Th, and calculates the enlargement factor M by the following equation when N>Th. Here, K in the following equation represents a magnification constant. The enlargement factor calculator 8 can determine the degree of overlap by (N−Th)*K in the following equation.

M=V+(N-Th)*K
Assuming that the normal value V is 1, the threshold Th is 2, and the magnification constant K is 0.5, as shown in FIG. The rate calculator 8 sets the magnification rate M to 1 (=normal value V). Further, when N is 3, since N>Th, the enlargement factor calculation unit 8 calculates the enlargement factor M (=1+(3-2)*0.5) as 1.5 by the above equation. do. Also, when N is 4, N>Th, so the enlargement factor calculator 8 sets the enlargement factor M (=1+(4-2)*0.5) to 2 by the above equation. The enlargement factor calculator 8 can determine the degree of overlapping of additional information based on the value of the number N as described above.

Note that the threshold Th may be changeable, or may change according to the size of the specific region.

Further, the enlargement factor calculation unit 8 may calculate the enlargement factor based on the distance between the coordinates of the reference points (reference point coordinates) that serve as the reference for displaying the additional information. Here, as shown in FIG. 4, the three pieces of additional information A to C all have the same shape and size, and have a circular main portion with a radius R and a sharp lower end portion. It shall form a shape. As shown on the left side of FIG. 4, three pieces of additional information A to C existing in a specific area (not shown) are such that additional information B overlaps additional information A, and additional information C overlaps additional information A. overlapping. It is also assumed that the initial magnification is 1×.

Based on the reference point coordinates (converted coordinates) of the additional information A, B, and C, the enlargement ratio calculator 8 calculates the distance D _AB between the reference points of the additional information A and B, and the distance D between the reference points of the additional information B and C. _BC and the distance D _AC between the reference points of the additional information A and C are calculated. Also, the example shown in FIG. 4 shows the case where the second reference point is the center point of the additional information.

The enlargement factor calculator 8 specifies the minimum inter-reference point distance D _min among the calculated inter-reference point distances D _AB , D _BC , and _DAC , and calculates 2R/D _min as an enlargement factor. As a result, an enlargement ratio of the photographed image is obtained at which the additional information A to C do not overlap. As described above, the enlargement factor calculator 8 can determine the degree of overlapping of additional information based on the value of the minimum distance _Dmin between reference points.

The enlargement processing unit 91 enlarges the photographed image from the camera 15 at the enlargement ratio obtained as described above, centering on the central coordinates of the specific area, and outputs the enlarged image. Further, the enlargement processing unit 91 corrects the converted coordinates of the additional information based on the above enlargement ratio and outputs the corrected coordinates.

For example, when the center coordinates of the specific area are (X0, Y0) and the reference point coordinates of certain additional information are (X1, Y1), the enlargement processing unit 91 calculates the correction coordinates ( Xc, Yc) are obtained. In the following formula, M represents an enlargement factor. In this calculation, the enlargement processing unit 91 subtracts the central coordinate value of the specific region from the reference point coordinate value of the additional information in order to set the central coordinate of the specific region to the reference coordinate (0, 0), Multiply by the magnification M. Then, the enlargement processing unit 91 adds the central coordinate value of the specific region to the multiplication value of the enlargement factor M in order to return the central coordinate value of the specific region to the original coordinate value.

Xc=M(X1-X0)+X0
Yc=M(Y1-Y0)+Y0
Note that the enlargement processing unit 91 outputs the photographed image input from the camera 15 as it is without enlarging it when the enlargement ratio is not input from the enlargement ratio calculation unit 8 .

The synthesizing unit 92 synthesizes the additional information in the specific area extracted by the monitor unit 7 out of the additional information acquired from the AR information storage unit 1 with the enlarged image from the enlargement processing unit 91 to produce a synthesized image. to create In synthesizing, the synthesizing unit 92 arranges the additional information at a position where the reference point coordinates of the additional information become the reference point coordinates (corrected coordinates) corrected by the enlargement processing unit 91 . Then, the composition processing section 92 outputs the generated composite image to the display section 17 .

The display unit 17 displays the synthesized image from the synthesis processing unit 92 . As a result, in the synthesized image displayed on the display unit 17, the additional information that was in the specific area before enlargement is displayed at dispersed positions without overlapping.

Next, an example in which additional information is displayed at dispersed positions will be described.

FIG. 5 shows a photographed image 200 photographed by the camera 15, in which the additional information is concentrated and difficult to see. FIG. 6 shows a photographed image 300 photographed by the camera 15, in which additional information is displayed according to the 3D space. FIG. 7 is a diagram showing the information display device 100 displaying a photographed image 400 including a specific region 401 photographed by the camera 15. As shown in FIG. FIG. 8 is a diagram showing the information display device 100 displaying an enlarged image 500 obtained by enlarging the photographed image 400 shown in FIG. FIG. 9 is a diagram showing the information display device 100 displaying another photographed image 600 including a specific area 601 photographed by the camera 15. As shown in FIG.

As shown in FIG. 5, in the captured image 200, additional information 201 is superimposed on the lockers placed in the room. In particular, a plurality of additional information 201 are arranged so as to overlap each other in the locker located on the far side. In this state, in some of the additional information 201, the content hidden by the additional information 201 superimposed cannot be sufficiently confirmed.

As shown in FIG. 6, in a 3D photographed image 300, additional information 301 to 305 that become smaller from the front locker to the back locker are arranged according to the 3D space. In the photographed image 300, the plurality of pieces of additional information 305 on the farthest side are positioned furthest from each other and are displayed in the smallest size, so that they do not overlap each other. However, if the additional information 305 is too small, it becomes difficult to confirm its contents.

Therefore, as described above, by enlarging the photographed image 400 at an enlargement factor calculated based on the degree of overlap of additional information within the specific region, the overlapping of additional information is eliminated. Specifically, as shown in FIG. 7, the monitor unit 7 displays the reference point coordinates of each of the plurality of overlapping pieces of additional information 402 in the photographed image 400 displayed on the display unit 17 of the information display device 100. The existing specific area 401 is displayed.

The enlargement ratio calculation unit 8 performs the calculation using the above formula explained with reference to FIG. method to calculate the magnification factor. Alternatively, the enlargement ratio calculation unit 8 calculates the enlargement ratio using the calculation method described above with reference to FIG. to calculate Then, the image synthesizing unit 9 enlarges the photographed image 400 at the enlargement ratio, and synthesizes the additional information 402 so as to be positioned at the corrected coordinates on the enlarged image.

As a result, as shown in FIG. 8, the additional information 402 is displayed in a state of being distributed and arranged on the enlarged image 500 without changing the size of each. Therefore, the contents of the additional information 402 can be easily confirmed.

Further, as shown in FIG. 9, additional information 602 that overlaps with another captured image 600 can be specified in a specific area 601 to be dispersed and displayed on the enlarged image in the same manner as described above.

As described above, the information display device 100 of this embodiment includes the information adding section 2 . The information addition unit 2 determines the degree of overlap of the additional information based on the monitor unit 7 that extracts a plurality of pieces of additional information existing in the specific area and the transformed coordinates thereof, and the extracted transformed coordinates. has an enlargement ratio calculator 8 for calculating the enlargement ratio of the photographed image. Further, the image synthesizing unit 9 in the information adding unit 2 includes an enlargement processing unit 91 for enlarging the photographed image by an enlargement ratio and correcting the converted coordinates of the additional information based on the enlargement ratio. Additional information is synthesized so as to be superimposed on the corrected coordinate position.

According to the above configuration, the photographed image is enlarged at an enlargement ratio corresponding to the degree of overlap of the additional information in the specific area, and the additional information is superimposed on the corrected coordinates corrected according to the enlargement ratio on the photographed image. As a result, it is possible to set an enlargement rate to the extent that the additional information does not overlap according to the degree of overlap. Therefore, when distant additional information is displayed in a small size, there is no need to walk closer to the object or perform an operation to enlarge the captured image. Therefore, overlapping of additional information can be properly eliminated without taking time and effort.

In particular, regarding operability, the captured image is automatically enlarged to an appropriate size and displayed, so the user does not need to adjust the size of the captured image by operating buttons, a touch panel, or the like. As a result, even when the user cannot operate the buttons or the touch panel because the hands are occupied, or when the size of the captured image cannot be adjusted due to restrictions on the operation functions of the operation unit 16 of the wearable device, etc., the user can comfortably AR function can be used for

Conventionally, there has been a method of automatically changing the position and size of balloon-type additional information when the additional information overlaps. However, with this method, there is a possibility that the additional information is displayed at a position not intended by the creator of the AR screen content.

On the other hand, according to the information display device 100, although the captured image is enlarged, the additional information is not enlarged, so the layout of the additional information is not changed. Therefore, the additional information can be displayed as intended by the creator of the screen content.

Further, the information addition section 2 has a specific area setting section 5 that sets a specific area according to an operation accepted by the operation section 16 .

If the specific area is small, there is a possibility that additional information desired to be displayed may not be included. Moreover, if the specific area is large, there is a possibility that other additional information that is not desired to be displayed may be included in the specific area.

On the other hand, according to the above configuration, a desired specific area can be provided in the captured image by setting the specific area. Therefore, by setting the specific area to a desired size according to the object, only the desired additional information can be selected and displayed in a distributed manner.

In particular, when judging overlap based only on the number of pieces of additional information in a specific area, if the specific area is set narrow, the amount of additional information included in the specific area will be reduced, and if the specific area is set wide, the specific area will contain less additional information. The amount of additional information included in is increased. Therefore, by appropriately setting the specific area, it is possible to change the conditions for enlarging the captured image.

Further, the enlargement ratio calculator 8 determines the degree of overlap based on the number of pieces of additional information having a certain size existing in the specific area, and calculates the enlargement ratio based on the degree of overlap.

According to the above configuration, if the size of the additional information is constant, the number of additional information that can exist in the specific area is determined simply from the size of the specific area. Therefore, it is possible to easily determine the degree of overlap from the number of pieces of additional information.

Further, the enlargement ratio calculator 8 determines the degree of overlap based on the distance between the coordinates of the reference points of the additional information existing in the specific area, and calculates the enlargement ratio based on the degree of overlap.

According to the above configuration, if the distance between the reference point coordinates is too short with respect to the size of the additional information, it can be seen that the additional information overlaps. Therefore, the overlapping degree can be determined more accurately based on the distance between the reference point coordinates.

[Example of realization by software]
The control block (especially the information addition unit 2) of the information display device 100 may be implemented by a logic circuit (hardware) formed in an integrated circuit (IC chip) or the like, or may be implemented by software.

In the latter case, the information display device 100 is provided with a computer that executes instructions of a program, which is software that implements each function. This computer includes, for example, one or more processors, and a computer-readable recording medium storing the above program. In the computer, the processor reads the program from the recording medium and executes it, thereby achieving the object of the present invention.

For example, the CPU 11 can be used as the processor. As the recording medium, a "non-temporary tangible medium" such as a ROM 13, a tape, a disk, a card, a semiconductor memory, a programmable logic circuit, or the like can be used. In addition, a RAM (Random Access Memory) for developing the above program may be further provided.

Also, the program may be supplied to the computer via any transmission medium (communication network, broadcast wave, etc.) capable of transmitting the program.

Note that one aspect of the present invention can also be implemented in the form of a data signal embedded in a carrier wave in which the program is embodied by electronic transmission.

[Additional notes]
The present invention is not limited to the above-described embodiments, but can be modified in various ways within the scope of the claims, and can be obtained by appropriately combining technical means disclosed in different embodiments. is also included in the technical scope of the present invention.

5 specific area setting unit 8 magnification calculation unit 15 camera (shooting unit)
16 operation unit 17 display unit 91 enlargement processing unit (enlargement unit)
92 Synthesis processing unit (synthesis unit)
100 information display device 401, 601 specific area

Claims (3)

a photographing unit that outputs a photographed image obtained by photographing a specific object associated with a plurality of pieces of additional information;
determining the degree of overlap of a plurality of pieces of additional information present in a specific area set to identify the object of interest in the captured image and the additional information displayed on the object; an enlargement ratio calculator that calculates an enlargement ratio based on the degree of overlap;
an enlarging unit for enlarging the photographed image at the enlargement ratio;
a synthesizing unit for synthesizing the enlarged photographed image with the additional information so as to overlap the position of the coordinates of the additional information corrected based on the enlargement ratio to generate a synthesized image;
A display unit that displays the synthesized image ,
Each piece of additional information has a certain size,
The enlargement ratio calculation unit sets the enlargement ratio to a predetermined reference enlargement ratio when the number of pieces of the additional information existing in the specific area is equal to or less than a threshold, An information display device , wherein the enlargement ratio is calculated by multiplying a value obtained by subtracting a threshold value by a predetermined constant and adding the reference enlargement ratio. 2. The information display device according to claim 1, wherein the magnification calculator sets the threshold according to the size of the specific area. an operation unit that receives an operation;
3. The information display device according to claim 1, further comprising a specific area setting unit that sets the specific area according to an accepted operation.

Images