JP2018066823A - Information display device and method for controlling processing thereof - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an information display device that reduces a processing time for displaying extended information to reduce delay in display, and a method for controlling processing thereof.SOLUTION: An information display device comprises: a storage part that stores display information related to an object; an imaging part that outputs a picked-up image; a display part that displays the display information; and a control part. The control part executes object detection processing of detecting the object included in an imaging range of the imaging part, and information display processing including processing of performing image recognition on the object in the output picked-up image and processing of displaying the display information related to the recognized object on the display part. The control part switches an operation mode of the information display processing according to the positional relationship between the detected object and imaging part.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an information display device and a processing control method thereof.

  In maintenance work of equipment such as a server, for example, an operator refers to a manual of a document, and checks the equipment while comparing the actual equipment and its parts with the explanations and drawings in the manual. However, when specifying a device to be inspected from a plurality of devices having the same shape or specifying a portion to be inspected from a plurality of portions having the same shape, an operator mistakenly selects another device or The site may be specified.

  Therefore, it is expected to implement augmented reality (AR) technology on devices such as tablet computers and wearable computers used by workers. For example, a head-mounted display that can be viewed through an actual field of view displays information such as characters and images (also referred to as “extended information”) superimposed on the actual field of view. Thereby, the worker can specify the work target and the work target position intuitively and accurately.

  For example, Patent Document 1 discloses a head-mounted display device that includes a display unit that can visually permeate the outside world. In Patent Document 1, in order to eliminate the positional deviation between the object and the extended information, “at least when the acquired view movement information and the acquired object movement information do not substantially match, Based on the object movement information, the relative position of the object with respect to the display unit after a predetermined time is estimated, and the display position of the predetermined image is determined based on the estimated relative position. Yes.

Japanese Patent Laid-Open No. 2006-082462

  A display device equipped with the AR technology specifies a work target and a work target position from a captured image of an actual field of view by image recognition processing, and displays extended information related to the specified work target and the work target position. However, since the image recognition process is a computational process with a high load, a display delay of the extended information occurs. In the technique described in Patent Document 1, the relative position may be erroneously estimated depending on the movement of the work target, and the display delay cannot be fundamentally reduced.

  An object of the present invention is to shorten the processing time for displaying extended information and reduce display delay.

  The present application includes a plurality of means for solving at least a part of the above-described problems. Examples of such means are as follows.

  One aspect of the present invention is an information display device, and includes a storage unit that stores display information related to a target, an imaging unit that outputs a captured image, a display unit that displays the display information, and a control unit. Including. The control unit includes a target detection process for detecting the target included in an imaging range of the imaging unit, a process for recognizing the target in the output captured image, and the related to the recognized target. And an information display process including a process of displaying display information on the display unit. The control unit switches an operation mode of the information display process according to a positional relationship between the detected object and the imaging unit.

  According to one embodiment of the present invention, the processing time for displaying extended information can be shortened, and display delay can be reduced.

  Problems, configurations, and effects other than those described above will be clarified by the following description of embodiments.

It is a use form figure of a system concerning an embodiment of the present invention. It is a block diagram which shows the structural example of the information display apparatus based on embodiment of this invention. It is explanatory drawing which shows the example of a data structure of the display information database based on embodiment of this invention. It is a block diagram which shows the hardware structural example of the information display apparatus based on embodiment of this invention. It is a flowchart which shows an example of the process which the information display apparatus which concerns on embodiment of this invention performs. It is explanatory drawing which shows the 1st example of an operator's visual field based on embodiment of this invention. It is explanatory drawing which shows the 2nd example of an operator's visual field based on embodiment of this invention. It is explanatory drawing which shows the 3rd example of an operator's visual field based on embodiment of this invention. It is explanatory drawing which shows the 4th example of an operator's visual field based on embodiment of this invention. It is a flowchart which shows an example of the process which the information display apparatus performs based on the 1st modification of embodiment of this invention. It is explanatory drawing which shows the example of a worker's visual field based on the 2nd modification of embodiment of this invention.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a system usage diagram according to an embodiment of the present invention. In the present embodiment, for the purpose of facilitating understanding, an example will be described in which the worker U maintains a plurality of servers T installed in the server room.

  The server T is a target of maintenance work by the worker U. The worker U wears an information display device 1 on which augmented reality (AR) technology is mounted. A label L may be attached to the server T. The label L can be used as a marker for the information display device 1 to recognize the server T from the captured image. Each label L describes an identifier of each server T, for example, an identifier such as a character, a code, or a graphic.

  The information display device 1 is a terminal called a so-called head mounted display that can be attached to and detached from the head of the worker U. The information display device 1 of the present embodiment implements AR technology. That is, the information display device 1 displays a real-world view and display information (also referred to as “extended information”) superimposed on the view. The information display device 1 of the present embodiment images the real world, recognizes the label L (and the identifier of the server T) from the captured image, and displays display information related to the recognized server T.

  For example, the worker U can see the display information displayed on the information display device 1, confirm the position of the server T as a work target, and perform maintenance such as inspection, component replacement, and setting.

  FIG. 2 is a block diagram illustrating a configuration example of the information display device according to the embodiment of the present invention. The information display device 1 includes a control unit 10, a storage unit 20, an imaging unit 30, a display unit 40, a communication unit 50, and an input unit 60.

  The imaging unit 30 images a real space and outputs a captured image. The imaging unit 30 of the present embodiment outputs video including captured images that are continuous in time series.

  The display unit 40 displays display information related to the work target that has been image-recognized.

  The communication unit 50 is connected to a network such as the Internet and communicates with a device such as an external server.

  The input unit 60 receives an input operation of the worker U.

  The control unit 10 controls the information display device 1 in an integrated manner. The control unit 10 includes a target detection unit 11, an image acquisition unit 12, an image recognition unit 13, a display processing unit 14, and a processing control unit 15.

  The target detection unit 11 detects a target included in the imaging range of the imaging unit 30. Moreover, the object detection unit 11 detects the positional relationship between the information display device 1 and each detected object. In the present embodiment, the distance from the imaging unit 30 to each target is used as the positional relationship. Note that the distance is not limited to a linear distance on a two-dimensional plane, and various defined distances such as a linear distance in a three-dimensional space in consideration of height can be used.

  Specifically, the target detection unit 11 acquires position information prepared in advance for the position of the server T. For example, the position coordinates of the server T can be acquired from an external position database via the communication unit 50 or from a position database stored in the storage unit 20. In addition, the target detection unit 11 acquires the position coordinates and orientation of the information display device 1 using one or more positioning techniques. Examples of the positioning technique include GPS (Global Positioning System), PDR (Pedestrian Dead Reckoning), and PMM (Pedestrian Map Matching).

  The target detection unit 11 can use the acquired position coordinates and orientation of the information display device 1 as the position coordinates and orientation of the imaging unit 30. The target detection unit 11 specifies the imaging range (for example, determined by the imaging direction and angle of view) of the imaging unit 30 based on the acquired position coordinates and orientation of the imaging unit 30 and a predetermined angle of view of the imaging unit 30. Then, the server T existing in the imaging range may be specified with reference to the position database. The target detection unit 11 calculates the distance from the imaging unit 30 to each server T based on the position coordinates of the imaging unit 30 and the specified position coordinates of the server T. Note that the imaging range of the imaging unit 30 is not limited to being defined by the imaging direction and the angle of view, and may be defined by the imaging direction, the angle of view, and the distance from the imaging unit 30, for example.

  The image acquisition unit 12 acquires a captured image from the imaging unit 30 and outputs the acquired image to the image recognition unit 13.

  The image recognition unit 13 performs image recognition processing on the captured image output from the image acquisition unit 12, and recognizes a target included in the captured image. Various techniques can be used as the image recognition process. For example, a pattern matching technique can be used in which features such as contours are extracted from a captured image and matched with a target feature prepared in advance. The image recognition unit 13 of this embodiment recognizes the label L as a target. The image recognition unit 13 recognizes the server T by recognizing the identifier (target ID) of the server T described in the label L.

  The display processing unit 14 acquires display information related to the target recognized in the captured image from the display information database 21. Further, the display processing unit 14 determines the display position of the acquired display information, and outputs it to the display unit 40 so as to be displayed in association with the recognized object.

  The process control unit 15 is required for the information display process by switching operation modes of processes (hereinafter referred to as “information display process”) executed by at least the image acquisition unit 12, the image recognition unit 13, and the display processing unit 14. Change the processing time. “Switching the operation mode” means changing the processing contents so that the processing procedure, processing target data, and the like are different before and after the switching. Further, “switching the operation mode” may include changing a setting of a predetermined control parameter in the program or data used by the program. For example, the process control unit 15 can switch the operation mode of the information display process by changing the setting of the control parameter of the information display process.

  The process control unit 15 of the present embodiment selects one target among the targets detected by the target detection unit 11. Then, the processing control unit 15 changes the resolution of the captured image in accordance with the distance from the imaging unit 30 to the selected target (obtained from the target detection unit 11). For example, the processing control unit 15 instructs the imaging unit 30 that can switch the resolution to change the resolution. Note that the resolution in the present embodiment is a resolution based on the number of pixels constituting the captured image, and the lower the resolution, the smaller the number of pixels.

  The storage unit 20 stores data used by the control unit 10. The storage unit 20 stores a display information database 21, for example. FIG. 3 is an explanatory diagram showing a data configuration example of the display information database according to the embodiment of the present invention.

  The display information database 21 stores one or more records including the target ID 211 and the display information 212. The target ID 211 is a work target identifier. The display information 212 is extended information definition information displayed in association with a work target. In the example of FIG. 3, the display information 212 includes a character string and a color of the character string. Of course, the display information 212 is not limited to character strings and colors, and may include information such as graphics, images, moving images, display sizes, display positions, and the like.

  The display processing unit 14 can acquire the display information 212 related to the target from the display information database 21 using the target identifier (target ID) in the captured image recognized by the image recognition unit 13.

  FIG. 4 is a block diagram illustrating a hardware configuration example of the information display device according to the embodiment of the present invention. The information display device 1 includes an arithmetic device 100, a main storage device 110, an auxiliary storage device 120, a communication device 130, a sensor group 140, an input device 150, an imaging device 160, and a display device 170.

  The arithmetic device 100 is an arithmetic unit such as a CPU (Central Processing Unit). The main storage device 110 is a storage device such as a RAM (Random Access Memory). The auxiliary storage device 120 is a storage device such as a hard disk, an SSD (Solid State Drive), or a flash ROM (Read Only Memory). The communication device 130 is a device that transmits and receives information, including a communication device that performs wired communication via a cable and a communication device that performs wireless communication via an antenna. The auxiliary storage device 120 may be built in the information display device 1, may be detachable from the information display device 1, or may be realized by an external storage that can communicate via the communication device 130.

  The sensor group 140 is one or more sensors including, for example, a GPS sensor, an acceleration sensor, an angular velocity sensor, an orientation sensor, a geomagnetic sensor, and the like. The positioning technique described above can use the output of the sensor group 140. The input device 150 is a device that receives input information including a key, a touch sensor, a microphone, and the like.

  The imaging device 160 is, for example, a digital still camera or a digital video camera equipped with an image sensor. When the information display device 1 is realized by a head mounted display, the imaging device 160 is provided in the information display device 1 at a position and a direction in which an image of a field of view viewed by a user wearing the information display device 1 can be captured. A plurality of imaging devices 160 may be provided.

  The display device 170 is a device that outputs an image including display information. When the information display device 1 is realized as a head-mounted display, the output image is projected onto, for example, a reflection plate provided in the information display device 1, reflected by the reflection plate, and guided to the user's eyes. . The output image may be directly projected on the eye. It is desirable that the front side of the user's eye of the information display device 1 is made of a transparent member so that the user can see the real field of view together with the display information. Note that the display device 170 may cover the front side of the user's eyes and display display information together with an image of an actual field of view.

  The control unit 10 can be realized by the arithmetic device 100, for example. That is, the function and processing of the control unit 10 can be realized, for example, when the arithmetic device 100 executes a predetermined application program. For example, the application program is stored in the auxiliary storage device 120, loaded onto the main storage device 110 for execution, and executed by the arithmetic device 100. This application program may be installed in the information display device 1 before the information display device 1 is shipped, or may be downloaded from a server on the network and installed. The application program is stored in a recording medium such as an optical medium such as a CD (Compact Disk), a DVD (Digital Versatile Disk), or a Blu-ray Disc (registered trademark), or a semiconductor memory. It may be read and installed in the information display device 1.

  The storage unit 20 can be realized by the main storage device 110 or the auxiliary storage device 120, for example. The imaging unit 30 can be realized by the imaging device 160, for example. The display unit 40 can be realized by the display device 170, for example. The communication unit 50 can be realized by the communication device 130, for example. The input unit 60 can be realized by the input device 150, for example.

  FIG. 5 is a flowchart showing an example of processing executed by the information display apparatus according to the embodiment of the present invention. This flowchart shows the processing of the control unit 10. This flowchart can be understood as being divided into three parts: an object detection process (steps S1 to S2), an operation mode switching process (steps S3 to S7), and an information display process (steps S8 to S13). .

  When the control unit 10 receives a start instruction from, for example, an external work instruction server or the operator U's operation terminal via the communication unit 50 or from the worker U via the input unit 60, the control unit 10 Start processing. The control part 10 complete | finishes the process of this flowchart, when the completion instruction | indication is received from the operation instruction server or the operation terminal of the operator U, or from the operator U, for example.

  First, the target detection unit 11 detects a target (step S1). For example, the target detection unit 11 acquires the position coordinates and orientation of the information display device 1 using a positioning technique, and specifies the imaging range of the imaging unit 30. Further, the target detection unit 11 refers to the position database and identifies the position coordinates of the target existing in the imaging range. In addition, the target detection unit 11 calculates the distance from the imaging unit 30 to each target.

  Then, the target detection unit 11 determines whether one or more targets are included in the imaging range of the imaging unit 30 based on the processing result of step S1 (step S2). If it is determined that the target is not included in the imaging range (NO in step S2), the target detection unit 11 returns the process to step S1.

  When it is determined that one or more targets are included in the imaging range (YES in step S2), the process control unit 15 includes a plurality of targets in the imaging range based on the processing result in step S1. It is determined whether or not (step S3).

  When it is determined that a plurality of targets are included in the imaging range (YES in step S3), the process control unit 15 selects one from the plurality of targets (step S4). For example, the process control unit 15 selects a target having a maximum (which may be the minimum or intermediate between the maximum and minimum) based on the distance from the imaging unit 30 acquired in step S1 to each target. To do. The distance condition to be used (maximum, minimum, or intermediate) is received in advance from an external work instruction server or the operator U's operation terminal via the communication unit 50 or from the worker U via the input unit 60. That's fine. When there are a plurality of objects having an intermediate distance, one object may be selected according to a predetermined condition (for example, an intermediate object having a maximum distance or a minimum distance, or an object having a median value, etc.) . When it is determined that a plurality of targets are not included in the imaging range (only one is included) (NO in step S3), the process control unit 15 selects a target included in the imaging range.

  After step S3 or step S4, the process control unit 15 determines whether the distance from the imaging unit 30 to the selected target is equal to or less than a predetermined threshold value (step S5). The imaging unit 30 of the present embodiment can be set to either low resolution or high resolution. For example, a maximum distance that enables image recognition of the label L from a low-resolution captured image is set in advance as the predetermined threshold.

  If it is determined that the distance to the target is equal to or less than the threshold (YES in step S5), the processing control unit 15 sets the resolution of the imaging unit 30 to a low resolution. The case where the distance to the target is equal to or smaller than the threshold is a case where the distance from the imaging unit 30 to the selected target is close and it is estimated that image recognition is possible even from a low-resolution image.

  If it is determined that the distance to the target exceeds the threshold (NO in step S5), the processing control unit 15 sets the resolution of the imaging unit 30 to a high resolution. The case where the distance to the target exceeds the threshold is a case where the distance from the imaging unit 30 to the selected target is long and it is estimated that image recognition is impossible from a low-resolution image.

  After step S6 or step S7, the image acquisition unit 12 instructs the imaging unit 30 to image the real space and acquires a captured image (step S8). In step S8, an image having the resolution set in step S6 or step S7 can be obtained.

  Then, the image recognition unit 13 performs image recognition processing on the captured image acquired in step S8, and recognizes one unrecognized target from the targets included in the image (step S9). For example, the image recognition unit 13 recognizes the label L. Further, the image recognition unit 13 recognizes the target ID of the server T described on the label L.

  Then, the image recognition unit 13 determines whether or not the object is recognized in step S9 (step S10). When it is determined that the object has not been recognized, that is, when the recognition of all objects has been completed (NO in step S10), the image recognition unit 13 returns the process to step S1.

  When it is determined that the target is recognized (YES in step S10), the display processing unit 14 acquires the display information 212 from the display information database 21 using the target ID of the target recognized in step S9 as a key (step S11). ).

  Then, the display processing unit 14 determines the display position of the display information (Step S12). For example, the display processing unit 14 determines the position on the target image recognized in step S9 as the display position. In the head mounted display, the position of the imaging unit 30 and the position of the user's eye are separated. Therefore, even if the display position determined on the captured image is used, the position of the projected display information may not be a preferable position. Therefore, the display processing unit 14 may correct the display position using the position correction value determined in advance.

  Then, the display processing unit 14 sets the display information acquired in step S11 to the display position determined in step S12, outputs it to the display unit 40, and displays it (step S13). Then, the display processing unit 14 returns the process to step S9.

  With reference to FIGS. 6 to 10, an example of an actual field of view (including a work target) that can be viewed by the operator U via the information display device 1 and display information superimposed thereon will be described. Recognizing the label L will be described as synonymous with recognizing the server T.

  FIG. 6 is an explanatory diagram illustrating a first example of a worker's field of view according to the embodiment of the present invention. FIG. 6 shows a case where the server T1 is recognized from the captured image, and display information A1 describing the server T1 is displayed in association with the actual server T1. The worker U can simultaneously confirm the server T1 that is the actual work target and the display information A1 related thereto.

  FIG. 7 is an explanatory diagram illustrating a second example of the worker's field of view according to the embodiment of the present invention. FIG. 7 shows a case where five servers T1 to T5 are recognized from the captured image. Corresponding display information A1 to display information A5 are displayed in association with each of the actual servers T1 to T5. ing. In FIG. 7, a server T <b> 1 (the same applies to the servers T <b> 2 to T <b> 4) and the server T <b> 5 are arranged in the order from the imaging unit 30.

  FIG. 7 illustrates a case where the server T5 farthest from the imaging unit 30 is selected from the plurality of servers T1 to T5 specified in the imaging range in step S4 of the flowchart of FIG. In this case, since the resolution is set based on the server T5 (that is, at least a resolution capable of recognizing the server T5 is set), the servers T1 to T4 in front of the server T5 can easily recognize the image. As a result, all the servers T1 to T5 are recognized from the captured image, and display information A1 to display information A5 corresponding to each of them is displayed.

  As described above, when a target farthest from the imaging unit 30 is selected, a target closer to the target is easily recognized, and display information can be displayed for a larger number of targets.

  FIG. 8 is an explanatory diagram illustrating a third example of the worker's field of view according to the embodiment of the present invention. FIG. 8 shows a case where one server T1 is recognized from the captured image, and corresponding display information A1 is displayed in association with the actual server T1. In FIG. 8, the servers T <b> 1 and five servers Tx are arranged in the order from the imaging unit 30.

  FIG. 8 shows a case where the server T1 closest to the imaging unit 30 is selected from the server T1 specified in the imaging range and the five servers Tx in step S4 of the flowchart of FIG. In this case, since the resolution is set on the basis of the server T1 (that is, at least a resolution capable of recognizing the image of the server T1 is set), it is difficult for the five servers Tx far from the server T1 to recognize the image. As a result, only the server T1 is recognized from the captured image, and the corresponding display information A1 is displayed.

  As described above, when the closest target is selected from the imaging unit 30, it is difficult to recognize a target farther than that, and display information can be displayed for a smaller number of targets. Compared with the case of FIG. 7, the recognition target and the display information are reduced.

  FIG. 9 is an explanatory diagram illustrating a fourth example of the worker's field of view according to the embodiment of the present invention. FIG. 9 shows a case where two servers T1 to T2 are recognized from the captured image, and corresponding display information A1 to display information A2 is displayed in association with each of the actual servers T1 to T2. ing. In FIG. 9, a server T1, a server T2, and a server Tx are arranged in order from the imaging unit 30.

  Further, FIG. 9 shows that the server T1, the server T2, and one server Tx specified in the imaging range in step S4 in the flowchart of FIG. 5 are intermediate from the imaging unit 30 (between the farthest and the nearest). The case where the server T2 located in is selected is shown. If there are a plurality of servers T located in the middle, one of them may be selected according to a predetermined condition. In this case, since the resolution is set on the basis of the server T2 (that is, at least a resolution capable of recognizing the server T2 is set), it is difficult for the server Tx farther than the server T2 to recognize the image. On the other hand, the server T1 in front of the server T2 can easily recognize the image. As a result, only the server T1 and the server T2 are recognized from the captured image, and display information A1 and display information A2 corresponding to each of them are displayed.

  As described above, when selecting an object at an intermediate position from the image capturing unit 30, an intermediate number of objects can be recognized and display information can be displayed as compared with the case of FIGS.

  The embodiment of the present invention has been described above. According to this embodiment, the processing time for displaying the extended information can be shortened, and the display delay can be reduced.

  For example, the information display device 1 according to the present embodiment switches the operation mode of the information display process by changing the resolution of the captured image according to the positional relationship (distance) between the imaging unit 30 and the target. That is, the operation mode for processing a high-resolution image can be switched to the operation mode for processing a low-resolution image. Thereby, the processing time of information display processing can be shortened, and display delay of display information can be reduced. Further, by processing a low-resolution image, it is possible to reduce the power consumption of the arithmetic device of the information display device 1.

  In addition, for example, the information display device 1 can switch the resolution to a low resolution when the distance to the target is short (below a predetermined threshold). In particular, when the work target is near the worker U, the worker U often performs manual work without moving from the spot, and the work of the worker U is required to be fast. By reducing the display delay, the waiting time for displaying the display information is reduced, and work productivity is improved.

  For example, when a plurality of targets are detected, the information display device 1 selects one of them and changes the resolution according to the distance to the selected target. Thereby, even when a plurality of objects are present in the captured image, the resolution can be changed.

  For example, the information display apparatus 1 skips the information display process when the target is not detected within the imaging range in the target detection process. Thereby, power consumption of the arithmetic unit etc. of the information display apparatus 1 can be reduced.

  The present invention is not limited to the above-described embodiments, and various modifications can be made within the scope of the gist of the present invention, and those aspects are included. Two or more selected from the above-described embodiments and modifications can be combined as appropriate. Hereinafter, a plurality of modified examples will be described, but the description of the same configuration as that of the above embodiment will be omitted, and a configuration different from the above embodiment will be mainly described.

[First Modification]
The control unit 10 may acquire the positional relationship between the imaging unit 30 and the target using image recognition processing instead of the positioning technique or the position database.

  Specifically, for example, the target detection unit 11 performs image recognition processing on the captured image output from the image acquisition unit 12, and recognizes a target included in the captured image. In addition, the target detection unit 11 calculates the distance from the imaging unit 30 to each target based on features such as the position, size, and shape of the recognized target image. Various techniques can be used for the process of calculating the distance (depth) of the object on the image. For example, it can be estimated by comparison with a target model prepared in advance. The image recognition unit 13 may use the result of the image recognition process of the target detection unit 11.

  FIG. 10 is a flowchart illustrating an example of processing executed by the information display device according to the first modification of the embodiment of the present invention. This flowchart shows the processing of the control unit 10. This flowchart can be understood as being divided into three parts: object detection processing (steps S8A to S2A), operation mode switching processing (steps S3 to S7), and information display processing (steps S9A to S13A). .

  First, the image acquisition unit 12 instructs the imaging unit 30 to image a real space and acquires a captured image (step S8A). In step S8A, an image having an initial resolution or a resolution set in step S6 or step S7 executed last time can be obtained.

  Then, the target detection unit 11 detects a target (step S1A). For example, the target detection unit 11 performs image recognition processing on the captured image acquired in step S8A, and recognizes one or more targets included in the image. Further, the object detection unit 11 calculates the distance to each object recognized from the imaging unit 30 by image processing.

  Then, the target detection unit 11 determines whether one or more targets are included in the imaging range (captured image) of the imaging unit 30 based on the processing result of step S1A (step S2A). If it is determined that the target is not included in the imaging range (NO in step S2A), the target detection unit 11 returns the process to step S8A.

  When it is determined that one or more targets are included in the imaging range (YES in step S2A), the process control unit 15 includes a plurality of targets in the imaging range based on the processing result in step S1A. It is determined whether or not (step S3). The processing from step S3 to step S7 is the same as step S3 to step S7 in FIG.

  After step S6 or step S7, the image recognition unit 13 selects one unselected target from the targets included in the image using the result of the image recognition process executed in step S1A (step S9A). ).

  Then, the image recognition unit 13 determines whether or not an object has been selected in step S9A (step S10A). If it is determined that no object has been selected, that is, if all objects have been selected (NO in step S10A), the image recognition unit 13 returns the process to step S8A.

  When it is determined that the target has been selected (YES in step S10A), the display processing unit 14 acquires the display information 212 from the display information database 21 using the target target ID selected in step S9A as a key (step S11A). ).

  Then, the display processing unit 14 determines the display position of the display information (Step S12A). For example, the display processing unit 14 determines the position on the target image recognized in step S1A as the display position.

  Then, the display processing unit 14 sets the display information acquired in step S11A to the display position determined in step S12A, outputs it to the display unit 40, and displays it (step S13A). Then, the display processing unit 14 returns the process to step S9A.

  Also according to the first modification, it is possible to obtain the same effect as that of the above embodiment. Moreover, since the object detection process and the information display process can use a common image recognition process, the process can be made more efficient by executing the object detection process and the information display process in parallel. In FIG. 10, the operation mode switching process is executed between the target detection process and the information display process. However, the operation mode switching process is not limited to this place. For example, after the information display process (NO in step S10A) May be executed.

[Second Modification]
The control unit 10 may acquire the positional relationship with the imaging unit 30 and display the display information only for the target selected by the work instruction.

  Specifically, for example, the position database stores the identifiers (target IDs) of one or more servers T selected as work targets together with the position coordinates of each server T. The work instruction server that instructs the worker U to work can select the work target server T and store the identifier in the position database.

  In step S1 of the flowchart of FIG. 5, the target detection unit 11 refers to the position database and identifies one or more servers T selected as work targets from among the servers T existing in the imaging range of the imaging unit 30. To do. Further, the target detection unit 11 calculates the distance from the imaging unit 30 to each identified server T. In addition, the display processing unit 14 matches the identifier of the recognized server T with the identifier of the server T selected as the work target (obtained in step S1) between step S10 and step S11 in the flowchart of FIG. Step S11 to Step S13 are skipped if they do not match, and Steps S11 to S13 may be executed if they match.

  FIG. 11 is an explanatory diagram illustrating an example of an operator's field of view according to a second modification of the embodiment of the present invention. FIG. 11 shows a case where only the server T2 is selected as the work target among the servers T1 to T5 specified in the imaging range in step S1 of the flowchart of FIG. As a result, the display information B2 and the display information C2 are displayed only for the server T2 among the servers T1 to T5 recognized from the captured image. In FIG. 11, the display information B2 presents that the server T2 is a work target, and the display information C2 presents the work content. Such display information may be set in advance in the display information database 21 by a work instruction server, for example. The display information C2 may indicate a work position.

  According to the second modification, it is possible to omit the process of specifying the positional relationship regarding the object that is not selected by the work instruction. Moreover, the display process of the display information regarding the object not selected by the work instruction can be omitted. Note that the target detection unit 11 may accept selection of a work target from the worker U via the input unit 60 or from the work instruction server or the operation terminal of the worker U via the communication unit 50.

  Also in the first modified example, the target detection unit 11 may specify one or more servers T selected as work targets from the targets detected from the captured image. The target detection unit 11 may accept the selection of the work target from the worker U via the input unit 60 or from the work instruction server or the operation terminal of the worker U via the communication unit 50.

[Other variations]
The control unit 10 of the above embodiment or the modification switches the operation mode of the information display process by changing the resolution of the captured image, but the operation mode of the information display process may be switched by another method.

  For example, the information display device 1 may be equipped with the arithmetic device 100 capable of switching the processing speed, and the processing control unit 15 may switch the processing speed of the arithmetic device 100 according to the resolution of the captured image. For example, it switches to high speed for high resolution and low speed for low resolution. In this way, for example, in the case of high resolution, it is possible to prevent an increase in processing time. Three or more resolutions may be prepared, and the processing speed corresponding to each resolution (for example, the higher the resolution, the higher the speed) may be switched.

  For example, the information display device 1 includes the arithmetic device 100 that can switch the processing speed, and the processing control unit 15 sets the processing speed of the arithmetic device 100 according to the number of objects specified in the imaging range of the imaging unit 30. You may switch. For example, when the number of objects is plural, the speed is switched to high speed, and when the number of objects is one, the speed is switched to low speed. In this way, for example, when the number of objects is plural, it is possible to prevent the processing time from becoming long. The number of objects may be divided into three or more groups, and the processing speed corresponding to each group (for example, the higher the number of objects, the higher the speed) may be switched.

  For example, when the target is not detected within the imaging range (NO in step S2 in FIG. 5 or NO in step S2A in FIG. 11), the process control unit 15 may stop the information display process. Moreover, the process control part 15 may start an information display process, when a target is detected within the imaging range (YES in step S2 in FIG. 5 or YES in step S2A in FIG. 11). In this way, while the target is not detected, programs and data developed on a memory such as a RAM can be reduced, and power consumption can be reduced.

  For example, the information display device 1 may be equipped with the imaging unit 30 that can switch and set the operation mode. The operation mode includes, for example, a power-on state, a sleep state, and a power-off state in descending order of power consumption. When the target is not detected within the imaging range (NO in step S2 in FIG. 5 or NO in step S2A in FIG. 11), the processing control unit 15 sets the imaging unit 30 to the sleep state or the power-off state. Also good. Further, when the target is detected within the imaging range (YES in step S2 in FIG. 5 or YES in step S2A in FIG. 11), the processing control unit 15 may set the imaging unit 30 to the power-on state. . In this way, it is possible to reduce power consumption while no target is detected.

  For example, the information display device 1 includes the imaging unit 30 that can adjust the focus, and the processing control unit 15 controls the imaging unit 30 to focus the selected target based on the distance to the identified target. You may combine them. In this way, it is possible to improve accuracy while reducing the processing time for image recognition.

  For example, when the target detection unit 11 detects an obstacle between the detected target and the imaging unit 30, the target detection unit 11 may exclude the target from the targets within the imaging range. The obstacle can be detected using, for example, the position information of the obstacle or using an image recognition process. The image recognition unit 13 does not execute the image recognition process on the region including the excluded target in the captured image. In this way, the processing load of the information display process can be reduced.

  The control unit 10 of the above-described embodiment or modification performs the image recognition process using the label L, but other methods may be used as long as the target can be recognized. For example, the work target itself such as the server T may be recognized as an object. Also in this case, the identifier of the recognized object can be obtained from a database prepared in advance including the characteristics of the object and the identifier of the object.

  The control unit 10 of the above embodiment or the modification acquires the target position using the position database, but other methods may be used as long as the target position can be acquired. For example, an electronic tag such as an RFID (Radio Frequency Identifier) may be attached to the object, and detection may be performed by wireless communication using an electromagnetic field or a radio wave using a tag reader mounted on the information display device 1. Further, for example, a distance image (depth image) sensor such as a TOF (Time of Flight) method may be mounted on the information display device 1 and the target may be specified from the captured distance image.

  The control unit 10 according to the above-described embodiment or modification changes the resolution of the imaging unit 30, but other methods may be used as long as the resolution can be changed. For example, the processing control unit 15 instructs the image acquisition unit 12 to change the resolution setting, and the image acquisition unit 12 converts the resolution of the image output from the imaging unit 30 to the set resolution by image processing. do it.

  The processing control unit 15 of the above embodiment or the modification selects one from two resolutions using one threshold, but is not limited to such a configuration.

  For example, the processing control unit 15 may select one from three or more resolutions using two or more thresholds (for example, select a lower resolution as the distance is closer). In this way, the processing time of the information display process can be accurately controlled step by step.

  For example, the process control unit 15 may hold an individual threshold value for each target and execute the distance determination using the threshold value corresponding to the selected target. In this way, the resolution can be set so that the object can be recognized with high accuracy according to the selected object.

  The control unit 10 of the above embodiment or the modification determines the resolution using the distance as the positional relationship between the imaging unit 30 and the selected target, but is not limited to such a configuration.

  For example, the target detection unit 11 specifies the angle of the target with respect to the imaging unit 30 (for example, the angle at which the imaging direction of the imaging unit 30 intersects the target surface to which the label L is attached), and uses the specified angle. The resolution may be determined. The process control unit 15 sets the resolution to a high resolution when the angle of the target surface exceeds a threshold (angle), and sets the resolution to a low resolution when the angle of the target surface is equal to or less than the threshold (angle). . Note that the target angle can be specified using a positioning technique and a position database, or using an image recognition process. In this way, for example, when the angle of the target surface to which the label L is affixed is large (the accuracy of image recognition decreases), a high-resolution image can be obtained to improve the accuracy of image recognition. it can. When the angle of the target surface is small, the processing load of the information display process can be reduced while obtaining a low-resolution image and maintaining the accuracy of image recognition.

  For example, the target detection unit 11 detects a target within the imaging range by image recognition processing, specifies the area occupied by the target in the captured image (or approximate area), and determines the resolution using the specified area. May be. The process control unit 15 sets the resolution to a low resolution when the area of the target exceeds the threshold (area), and sets the resolution to a high resolution when the area of the target is equal to or less than the threshold (area).

  Note that the target detection unit 11 may use a combination of two or more of the above-described distance, angle, and area.

  The above embodiment uses the positioning technique and the position database for the detection of the object, and the first modification uses the image recognition technique for the detection of the object, but both of them may be used. Good.

  The target detection unit 11 of the first modification described above tracks the target among a plurality of time-series captured images, and uses the distance to the target calculated in the previous image, and uses the target in the subsequent image. The distance up to may be estimated.

  The processing control unit 15 of the above embodiment or the modification selects one target based on the distance from the imaging unit 30 to each target when a plurality of targets are detected, but selects one target. If possible, it is not limited to this method.

  For example, the angle of the target with respect to the imaging unit 30 as described above may be used, or the area in the captured image may be used. Further, for example, the processing control unit 15 may accept the selection of an object from an external work instruction server or the operation terminal of the worker U via the communication unit 50 or from the worker U via the input unit 60. .

  The information display device of the above embodiment or modification employs a head mounted display. However, the information display device of the present invention is not limited to a head mounted display as long as it has a configuration that achieves the object of the present invention. That is, the information display apparatus of the present invention can include a wearable device that can be attached to and detached from a part of the user's body, a device that the user carries. In addition, at least some of the units such as the imaging unit and the display unit do not need to be provided in one device, and may be provided separately. Some of the functions and processing of the information display apparatus may be executed on other devices such as a server on the network.

  In the above embodiment or modification, the server maintenance work has been described as an example. Of course, the present invention can be utilized in many other situations. For example, it is a case where extended information on exhibits is shown to a visitor in a museum, or extended information on products is shown to an operator during sorting work in a warehouse.

  The present invention provides not only an information display apparatus, but also a process control method for the information display apparatus, a program for the information display apparatus, a computer-readable recording medium storing the program, an information display system including the information display apparatus and a work instruction server, It can be provided in various modes.

  The present invention is not limited to the above-described embodiment, and includes various modifications. For example, each of the above-described embodiments has been described in detail for easy understanding of the present invention, and the present invention is not necessarily limited to the one provided with all the constituent elements described. In addition, a part of the configuration of a certain embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of a certain embodiment. In addition, it is possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

  Each of the above-described configurations, functions, processing units, processing means, and the like may be realized by hardware by designing a part or all of them with, for example, an integrated circuit. Each of the above-described configurations, functions, and the like may be realized by software by the processor interpreting and executing a program that realizes each function. Information such as programs, tables, and files that realize each function can be stored in a storage device such as a memory, a hard disk, and an SSD, or a recording medium such as an IC card, an SD card, and a DVD. Further, the control lines and information lines indicate what is considered necessary for the explanation, and not all the control lines and information lines on the product are necessarily shown. Actually, it may be considered that almost all the components are connected to each other.

DESCRIPTION OF SYMBOLS 1 ... Information display apparatus, 10 ... Control part, 11 ... Object detection part, 12 ... Image acquisition part, 13 ... Image recognition part, 14 ... Display processing part, 15 ... Processing control part, 20 ... Memory | storage part, 21 ... Display information Database, 30 ... Imaging unit, 40 ... Display unit, 50 ... Communication unit, 60 ... Input unit, 100 ... Arithmetic device, 110 ... Main storage device, 120 ... Auxiliary storage device, 130 ... Communication device, 140 ... Sensor group, 150 ... Input device, 160 ... Imaging device, 170 ... Display device, 211 ... Target ID, 212 ... Display information, A ... Display information, B ... Display information, C ... Display information, L ... Label, T ... Server, U ... Work Person

Claims (12)

A storage unit for storing display information related to the object;
An imaging unit that outputs a captured image;
A display unit for displaying the display information;
The object detection processing for detecting the target included in the imaging range of the imaging unit, the processing for recognizing the target in the output captured image, and the display information related to the recognized target are displayed. A control unit that executes an information display process including a process to be displayed on the unit, and
The control unit switches an operation mode of the information display process according to a positional relationship between the detected object and the imaging unit. The information display device according to claim 1,
The control unit switches the operation mode by changing the resolution of the imaging unit or the resolution of the captured image output from the imaging unit. The information display device according to claim 2,
The control unit sets the resolution to a first resolution when the distance from the imaging unit to the target is equal to or less than a predetermined threshold, and sets the distance from the imaging unit to the target to the predetermined threshold. In the case of exceeding the resolution, the resolution is set to a second resolution higher than the first resolution. The information display device according to claim 1,
The information display device, wherein the control unit switches the operation mode according to a distance from the imaging unit to the target. The information display device according to claim 1,
The control unit switches the operation mode according to the area or angle of the target in the captured image. The information display device according to claim 1,
When a plurality of the targets are detected, the control unit switches an operation mode of the information display process according to a positional relationship between the one target selected from the plurality of targets and the imaging unit. An information display device. The information display device according to claim 6,
The control unit selects the target located farthest, nearest, or intermediate to the imaging unit from the plurality of targets, and according to a distance from the imaging unit to the selected target An information display device that switches the operation mode. The information display device according to claim 1,
The controller is
The target detection process is executed before or in parallel with the information display process, and the target included in the imaging range of the imaging unit is detected by recognizing the target in the output captured image. ,
When the target is included in the imaging range, the information display process is executed to execute the next target detection process. When the target is not included in the imaging range, the information display process is skipped. An information display device that executes the next object detection process. The information display device according to claim 1,
The controller is
The target detection process is executed before the information display process, and the target included in the imaging range of the imaging unit is detected using the positional information of the imaging unit and the positional information of the target,
When the target is included in the imaging range, the information display process is executed to execute the next target detection process. When the target is not included in the imaging range, the information display process is skipped. An information display device that executes the next object detection process. The information display device according to claim 1,
The controller is
The target detection process is executed before the information display process, and the target included in the imaging range of the imaging unit is detected using the positional information of the imaging unit and the positional information of the target,
When the target is included in the imaging range, the imaging unit is set to a power-on state, the information display process is executed, the next target detection process is executed, and the target is included in the imaging range. If not, the information display device sets the imaging unit to a sleep state or a power-off state, skips the information display process, and executes the next object detection process. The information display device according to claim 1,
Including an input unit or a communication unit,
The controller is
Among the one or more objects included in the imaging range of the imaging unit, the target selected via the input unit or the communication unit is detected, and the display information related to the selected target is An information display device characterized by being displayed on a display unit. A process control method for an information display device for displaying display information related to a target,
The information display device includes:
A target detection process for detecting the target included in the imaging range of the imaging unit; a process for recognizing the target in the captured image output from the imaging unit; and the display information related to the recognized target. Including a control unit that executes information display processing including processing to be displayed on the display unit,
The process control method, wherein the control unit switches an operation mode of the information display process according to a positional relationship between the detected object and the imaging unit.

Images