JP6743080B2 - Display control system and display control method - Google Patents

Description

The present invention relates to a display control device, a display control system, a display control method, and a display control program.

Conventionally, as a technique for allowing an operator and an instructor at a remote location to share work information, there is known a technique in which an image captured by a worker at a work site is transmitted to an instructor's terminal device. Moreover, in such a technique, video transmission is performed via the Internet.

Meiden Corporation, "Remote site real-time technical support software EduNet (creative design)", [online], [February 27, 2018 search], Internet (http://www.meidenshoji.co.jp/business/ ict_07/prod_03/prod_03_01/index.html) Panasonic, “Video Sharing Support Solution”, [online], [Search on February 27, 2018], Internet (https://www.panasonic.com/jp/business/its/field.html) Nippon Steel & Sumikin Solutions, "Smart Glass Field Usage Introduction Verification Service", [online], [Search on February 27, 2018], Internet (https://www.nssol.nssmc.com/ss/production/smart_glass.html) )

However, the conventional technique has a problem that it may be difficult to share information with a remote place in an environment where the communication speed is low. For example, when a low speed network such as a mobile network is used, it is considered that it is difficult to share the information between the instructor and the worker in the conventional technique because the image cannot be transmitted in real time. ..

The display control device according to the embodiment includes a receiving unit, a specifying unit, and a display control unit. The receiving unit receives position information indicating the position of the terminal device in a predetermined space. The specifying unit specifies the position corresponding to the position information in the model representing the space each time the receiving unit receives the position information. The display control unit causes the display unit to display an image representing the model and the position specified by the specifying unit every time the position is specified by the specifying unit.

FIG. 1 is a diagram for explaining the outline of the display control system according to the first embodiment. FIG. 2 is a diagram illustrating an example of the configuration of the terminal device according to the first embodiment. FIG. 3 is a diagram illustrating an example of the configuration of the display control device according to the first embodiment. FIG. 4 is a flowchart showing a processing flow of the terminal device according to the first embodiment. FIG. 5 is a flowchart showing a processing flow of the display control device according to the first embodiment.

Embodiments of a display control device, a display control system, a display control method, and a display control program according to the present application will be described below in detail with reference to the drawings. The present invention is not limited to the embodiments described below.

(First embodiment)
First, the outline of the display control system of the present embodiment will be described with reference to FIG. FIG. 1 is a diagram for explaining the outline of the display control system according to the first embodiment. As shown in FIG. 1, the display control system 1 includes a terminal device 10 and a display control device 20. The terminal device 10 and the display control device 20 are connected via the network 2. For example, the network 2 is the Internet.

The terminal device 10 is carried by the worker 10a in the space 30. For example, the terminal device 10 is a tablet terminal, a smartphone, or the like. Further, the worker 10 a works in the space 30. The space 30 may be indoors or outdoors. For example, the space 30 is an office, a plant, or the like. Further, for example, the worker 10a performs installation, repair, maintenance, etc. of the device.

The display control device 20 is used by the instructor 20a. For example, the display control device 20 is a personal computer, a tablet terminal, a smartphone, or the like. Further, the instructor 20a remotely monitors the work status and the like of the worker 10a via the display control device 20 and gives a work instruction to the worker 10a.

Further, the display control device 20 is provided in a place apart from the space 30. For example, the display control device 20 may be provided in a different room in the same building as the space 30, or may be provided in a country different from the country in which the space 30 exists.

Here, the display control system 1 supports information sharing between the worker 10a and the instructor 20a. For example, the terminal device 10 transmits information about the position of the worker 10a to the display control device 20. Then, the display control device 20 presents the position of the worker 10a to the instructor 20a based on the information received from the terminal device 10. Thereby, the instructor 20a can give an instruction after grasping the position of the worker 10a.

Further, the display control system 1 can present information for intuitively grasping the position of the worker 10a while suppressing the amount of data transmitted and received between the terminal device 10 and the display control device 20. ..

As shown in FIG. 1, the terminal device 10 identifies the current position with the position of the marker 31 provided in the space 30 as the origin. For example, the marker 31 is a QR code (registered trademark). For example, the terminal device 10 can specify the relative position from the marker 31 at each time point by recording the movement route from the time point when the marker 31 is read by a sensor or the like. The terminal device 10 transmits the position information indicating the specified position to the display control device 20.

The number of markers is not limited to that shown in FIG. For example, a plurality of different markers may be provided at fixed positions in the space 30 defined respectively. In that case, the display control device 20 can correct the error in the position identification each time the marker is read.

Here, the display control device 20 can display an image of the model of the space 30 on the display unit 13. For example, the display control device 20 displays an image using 3D CAD (Computer Aided Design) data of the space 30.

Further, when the display control device 20 receives the position information from the terminal device 10, the display control device 20 specifies the position in the model corresponding to the position information, and displays the specified position on the image. In the example of FIG. 1, the display control device 20 displays an icon 232 indicating a person at the specified position. Further, the position information received by the display control device 20 from the terminal device 10 may include information capable of specifying the orientation. In this case, the display control device 20 displays the arrow icon 233 in the specified direction.

Further, the display control device 20 can display the coordinates indicating the position and the vector indicating the direction. In the example of FIG. 1, the display control device 20 displays (2.0, 4.0, 0.5) as the coordinate indicating the position and (0.43, 0.75, −0) as the vector indicating the direction. . 5) is displayed. Here, the display control device 20 can represent coordinates indicating the position of the terminal device 10 with the position corresponding to the marker 31 in the space 30 as the origin 231.

Further, the display control device 20 can accept a work instruction from the instructor 20a. At this time, the display control device 20 transmits instruction information indicating the input instruction to the terminal device 10. Then, the display unit 13 of the terminal device 10 can display the icon 133 indicating the content of the instruction as an AR (Augmented Reality) object at a predetermined position of the image captured by the camera or the like at the specified position. In the example of FIG. 1, the display unit 13 of the terminal device 10 displays, as an AR object, an icon 133 in which a text message “Confirm here” is described. Further, the terminal device 10 may output the content of the instruction by voice.

Further, at this time, the display control device 20 displays the icon 234 at the corresponding position in the model. The display control device 20 can further accept the designation of the display position of the AR object by the instructor 20a. In this case, the icon 133 and the icon 234 are displayed at the designated position at the position designated by the instructor 20a.

The display unit 13 may be included in the display control device 20 or may be a device different from the display control device 20. In this case, the display control device 20 controls the display unit 13 to display each image and the like.

(Configuration of the terminal device according to the first embodiment)
The configuration of the terminal device 10 will be described with reference to FIG. FIG. 2 is a diagram illustrating an example of the configuration of the terminal device according to the first embodiment. As shown in FIG. 2, the terminal device 10 includes a communication unit 11, an input unit 12, a display unit 13, an imaging unit 14, a sensor 15, a storage unit 16 and a control unit 17.

The communication unit 11 performs wired or wireless data communication with another device via a network. For example, the communication unit 11 is a NIC (Network Interface Card). The input unit 12 receives input of data from the user. The input unit 12 is, for example, an input device such as a mouse or a keyboard. The display unit 13 is a display device such as a display that displays a screen. The input unit 12 and the display unit 13 may be touch panel displays. The image capturing unit 14 captures an image. For example, the imaging unit 14 is a camera. The sensor 15 is a depth sensor, an infrared sensor, an IMU (Inertial Measurement Unit), or the like. The display unit 13 is an example of a second display unit.

The storage unit 16 is a storage device such as an HDD (Hard Disk Drive), an SSD (Solid State Drive), an optical disk, a RAM (Random Access Memory), a flash memory, and an NVSRAM (Non Volatile Static Random Access Memory). The storage unit 16 stores an OS (Operating System) and programs executed by the terminal device 10. Further, the storage unit 16 stores various kinds of information used for executing the program.

The control unit 17 controls the terminal device 10. The control unit 17 is a CPU (Central Processing Unit), an MPU (Micro Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like. For example, the control unit 17 functions as the reading unit 171, the specifying unit 172, the transmitting unit 173, the receiving unit 174, and the display control unit 175.

The reading unit 171 reads the marker 31 installed in the space 30. The marker 31 may be a QR code or a bar code, or may be an image displaying a predetermined specific shape, character string, or the like.

The identifying unit 172 identifies the position of the terminal device 10 in the space 30. The identifying unit 172 identifies a position by using, for example, infrared rays, a beacon signal of a wireless LAN (Local Area Network), GPS (Global Positioning System), or the like. A method of identifying the position of the terminal device 10 by the identifying unit 172 may be any known method. The identifying unit 172 is an example of a second identifying unit.

The specifying unit 172 can also specify the relative position of the terminal device 10 when the position of the marker 31 read by the reading unit 171 is the origin. In this case, the identifying unit 172 identifies the movement route of the terminal device 10 from the time when the reading unit 171 reads the marker 31 based on the information acquired from the sensor 15 that is the depth sensor. By using the depth sensor, the identifying unit 172 can stereoscopically identify the travel route from the surrounding environment information. Further, the identifying unit 172 can identify the change in the orientation of the terminal device 10 using the depth sensor.

The transmitting unit 173 transmits the position information indicating the position specified by the specifying unit 172. The position information may be latitude and longitude, or coordinates when a predetermined position in the space 30 is the origin. The transmission unit 173 is an example of a second transmission unit.

The transmitting unit 173 transmits the position information indicating the relative position specified by the specifying unit 172. Specifically, the transmission unit 173 transmits, as position information, coordinates indicating a relative position when the position of the marker 31 is the origin.

The receiving unit 174 receives the instruction information from the display control device 20. For example, the instruction information is a text message indicating the content of the instruction. Further, the instruction information may be voice data. The receiving unit 174 is an example of a second receiving unit.

The display control unit 175 causes the display unit 13 to display a predetermined object indicating the content of the instruction information in a superimposed manner at a predetermined position in the image captured by the image capturing unit 14. For example, the display control unit 175 causes the display unit 13 to display the image as displayed on the display unit 13 in FIG. 1. The display control unit 175 is an example of a second display control unit.

(Configuration of Display Control Device of First Embodiment)
The configuration of the display control device 20 will be described with reference to FIG. FIG. 3 is a diagram illustrating an example of the configuration of the display control device according to the first embodiment. As shown in FIG. 3, the display control device 20 includes a communication unit 21, an input unit 22, a display unit 23, a storage unit 24, and a control unit 25.

The communication unit 21 performs wired or wireless data communication with other devices via a network. For example, the communication unit 21 is a NIC. The input unit 22 receives data input from the user. The input unit 22 is, for example, an input device such as a mouse or a keyboard. The display unit 23 is a display device such as a display that displays a screen. The input unit 22 and the display unit 23 may be touch panel displays. The display unit 23 is an example of a first display unit.

The storage unit 24 is a storage device such as an HDD, SSD, optical disk, RAM, flash memory, NVSRAM, or the like. The storage unit 24 stores the OS and programs executed by the display control device 20. Further, the storage unit 24 stores various information used for executing the program. The storage unit 24 also stores model information 241.

The model information 241 is data regarding the model of the space 30. For example, the model information 241 is 3D CAD data representing the space 30. In addition, the model information 241 includes information on the corresponding position of the marker 31 in the model.

The control unit 25 controls the display control device 20. The control unit 25 is a CPU, MPU, ASIC, FPGA, or the like. For example, the control unit 25 functions as the specifying unit 251, the transmitting unit 252, the receiving unit 253, and the display control unit 254.

The identifying unit 251 identifies the position corresponding to the position information in the model representing the space 30 each time the receiving unit 253 receives the position information. The identifying unit 251 can identify the position corresponding to the position information based on the position of the origin in the model representing the space 30 specified based on the position of the marker 31 acquired in advance. The identifying unit 251 is an example of a first identifying unit.

The transmission unit 252 transmits instruction information indicating an instruction input by the user. The transmission unit 252 is an example of a first transmission unit. Further, for example, the instruction information indicates the instruction input by the instructor 20a.

The receiving unit 253 receives the position information from the terminal device 10. When the terminal device 10 transmits the position information including the orientation, the receiving unit 253 receives the position information indicating the position and orientation of the terminal device 10. The receiving unit 253 is an example of a first receiving unit.

The display control unit 254 causes the display unit 23 to display an image representing the model and the position specified by the specifying unit 251 every time the position is specified by the specifying unit 251. Further, the display control unit 254 causes the display unit 23 to display an image indicating the orientation included in the position information together with the position specified by the specifying unit 251. The display control unit 254 is an example of a first display control unit.

(Processing of the first embodiment)
The processing of the terminal device 10 will be described with reference to FIG. FIG. 4 is a flowchart showing a processing flow of the terminal device according to the first embodiment. First, as shown in FIG. 4, the terminal device 10 reads the marker 31 (step S101).

Next, the terminal device 10 specifies the position of the terminal device 10 (step S102). At this time, the terminal device 10 can specify the relative position from the position of the read marker 31. Then, the terminal device 10 transmits the position information indicating the specified position to the display control device 20 (step S103).

Here, when the terminal device 10 receives the instruction information (step S104, Yes), the terminal device 10 displays the instruction (step S105), and returns to step S102 to repeat the process. On the other hand, when the terminal device 10 has not received the instruction information (No in step S104), the process returns to step S102 and repeats the processing.

The processing of the display control device 20 will be described with reference to FIG. FIG. 5 is a flowchart showing a processing flow of the display control device according to the first embodiment. First, as shown in FIG. 5, the display control device 20 receives position information from the terminal device 10 (step S201). Next, the display control device 20 identifies the position of the terminal device 10 from the position information (step S202).

Then, the display control device 20 displays the model of the space 30 and the position of the terminal device 10 (step S203). Here, when the user inputs an instruction (Yes in step S204), the display control device 20 transmits the instruction information to the terminal device 10 (step S205), and returns to step S201 to repeat the process. On the other hand, if the user has not input an instruction (step S204, No), the display control device 20 returns to step S201 and repeats the processing.

As described above, the terminal device 10 repeatedly executes the process of step S102. On the other hand, the display control device 20 can update the image to be displayed each time the terminal device 10 executes step S102 and transmits the position information. Therefore, the faster the repetitive processing by the terminal device 10, the more the number of frames of the image displayed by the display control device 20 can be increased, and the real-time property of information sharing can be improved.

Here, for example, when the terminal device 10 transmits the spatial coordinates composed of three numerical values as the positional information, if each numerical value is represented by 32 bits, the size of the positional information transmitted at one time is about 100 bits. Therefore, according to the display control system 1, it is possible to share information in real time even when the communication speed is low, as compared with the case where video is directly transmitted and received.

In the above case, in order for the display control device 20 to display an image of 30 frames per second, a communication speed of about 3 kbps is sufficient. On the other hand, it is considered that a communication speed of 1 Mbps or higher may be required in order to transmit and receive images of 30 frames per second.

(Effect of the first embodiment)
In the embodiment, the receiving unit 253 of the display control device 20 receives the position information from the terminal device 10. Further, the specifying unit 251 of the display control device 20 specifies the position corresponding to the position information in the model representing the space 30 each time the receiving unit 253 receives the position information. In addition, the display control unit 254 of the display control device 20 causes the display unit 23 to display an image representing the model and the position specified by the specifying unit 251 every time the position is specified by the specifying unit 251.

In this way, the display control device 20 can display the image representing the position of the terminal device 10 in the space 30 by receiving the position information without receiving the image, and thus communication with the remote place is possible. The amount can be reduced. Therefore, according to the first embodiment, it is possible to share information with a remote place even in an environment where the communication speed is low.

The reception unit 253 receives position information indicating the position and orientation of the terminal device 10. Further, the display control unit 254 causes the display unit 23 to display an image indicating the orientation included in the position information together with the position specified by the specifying unit 251.

In this way, by transmitting not only the position but also the orientation information, the instructor 20a can grasp the situation of the worker 10a more accurately.

The identifying unit 172 of the terminal device 10 identifies the position of the terminal device 10 in the space 30. Further, the transmission unit 173 of the terminal device 10 transmits the position information indicating the position specified by the specifying unit 172. Further, the receiving unit 253 of the display control device 20 receives the position information from the terminal device 10. Further, the specifying unit 251 of the display control device 20 specifies the position corresponding to the position information in the model representing the space 30 each time the receiving unit 253 receives the position information. Further, the display control unit 254 of the display control device 20 causes the display unit 23 to display an image representing the model and the position specified by the specifying unit 251 every time the position is specified by the specifying unit 251.

In this way, since the terminal device 10 transmits the position information without transmitting the image, the display control device 20 can display the image representing the position of the terminal device 10 in the space 30, and thus the remote location. It is possible to reduce the amount of communication between and. Therefore, according to the first embodiment, it is possible to share information with a remote place even in an environment where the communication speed is low.

The reading unit 171 of the terminal device 10 reads the marker 31 installed in the space 30. Further, the specifying unit 172 of the terminal device 10 specifies the relative position of the terminal device 10 when the position of the marker 31 read by the reading unit 171 is the origin. In addition, the transmission unit 173 of the terminal device 10 transmits the position information indicating the relative position specified by the specifying unit 172. Further, the specifying unit 251 of the display control device 20 specifies the position corresponding to the position information based on the position of the origin in the model representing the space 30 specified based on the position of the marker 31 acquired in advance. Thereby, the display control device 20 can easily and accurately specify the position of the terminal device 10 based on the position information received from the terminal device 10.

The transmission unit 252 of the display control device 20 transmits instruction information indicating an instruction input by the user. Further, the receiving unit 174 of the terminal device 10 receives the instruction information from the display control device 20. In addition, the display control unit 175 of the terminal device 10 causes the display unit 13 to display a predetermined object indicating the content of the instruction of the instruction information at a predetermined position in the image captured by the image capturing unit 14 in a superimposed manner. Therefore, the terminal device 10 can intuitively display the instruction using the AR, and can facilitate information sharing.

The programs executed by the terminal device 10 and the display control device 20 of the present embodiment are provided by being pre-installed in a ROM or the like. The program executed by the terminal device 10 and the display control device 20 of the present embodiment may be configured to be recorded in a computer-readable recording medium in a file of an installable format or an executable format and provided. Good.

Further, the program executed by the terminal device 10 and the display control device 20 of the present embodiment may be stored in a computer connected to a network such as the Internet and provided by being downloaded via the network. Good. Further, the program executed by the terminal device 10 and the display control device 20 of the present embodiment may be provided or distributed via a network such as the Internet.

The programs executed by the terminal device 10 and the display control device 20 according to the present embodiment include the above-described units (reading unit 171, identifying unit 172, transmitting unit 173, receiving unit 174, display control unit 175, identifying unit 251, and transmitting unit). 252, a receiving unit 253, and a display control unit 254), and as actual hardware, the CPU reads a program from the ROM and executes the program to load the above units onto the main storage device. The reading unit 171, the identification unit 172, the transmission unit 173, the reception unit 174, the display control unit 175, the identification unit 251, the transmission unit 252, the reception unit 253, and the display control unit 254 are generated in the main storage device. There is.

Although the embodiments of the present invention have been described, the embodiments are presented as examples and are not intended to limit the scope of the invention. The embodiment can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the spirit of the invention. The embodiments and the modifications thereof are included in the invention described in the claims and equivalents thereof as well as included in the scope and the gist of the invention.

DESCRIPTION OF SYMBOLS 1 display control system 10 terminal device 11, 21 communication part 12, 22 input part 13, 23 display part 14 imaging part 15 sensor 16, 24 storage part 17, 25 control part 20 display control device 171 reading part 172, 251 specific part 173 , 252 Transmitter 174, 253 Receiver 175, 254 Display controller 241 Model information

Claims (6)

A display control system having a display control device and a terminal device,
The terminal device,
A first specifying unit that specifies the position of the terminal device in a predetermined space;
A first transmitting unit that transmits only position information indicating the position specified by the first specifying unit;
Have
The display control device,
A first receiving unit that receives only the position information from the terminal device;
A second specifying unit that specifies a position corresponding to the position information in the model representing the space each time the position information is received by the first receiving unit;
A first display control unit that causes the first display unit to display an image representing the model and the position specified by the second specifying unit every time the position is specified by the second specifying unit;
A display control system comprising: A display control system having a display control device and a terminal device,
The terminal device,
A first specifying unit that specifies the position of the terminal device in a predetermined space;
A first transmitting unit that transmits position information indicating the position specified by the first specifying unit;
Have
The display control device,
A storage unit for storing 3D CAD (Computer Aided Design) data,
A first receiving unit that receives the position information from the terminal device;
A second specifying unit that specifies a position corresponding to the position information in the model representing the space each time the position information is received by the first receiving unit;
Every time the position is specified by the second specifying unit, an image representing the position specified by the model and the second specifying unit is used as a first image by using the 3D CAD data stored in the storage unit . A first display control unit for displaying on the display unit;
A display control system comprising: The receiving unit receives position information indicating the position and orientation of the terminal device,
Wherein the display control unit, together with the position specified by the specifying section, the display control system according to claim 1 or 2, characterized in that to display an image representing the direction contained in the position information on the display unit. The terminal device,
Further comprising a reading unit for reading a marker installed in the space,
The first specifying unit specifies a relative position of the terminal device when the position of the marker read by the reading unit is an origin,
The first transmitting unit transmits position information indicating a relative position specified by the first specifying unit,
The second specifying unit specifies a position corresponding to the position information based on a position of the origin in a model representing the space specified based on a position of the marker acquired in advance. Item 5. The display control system according to Item 3. The display control device,
A second transmitting unit that transmits only instruction information indicating an instruction input by the user,
The terminal device,
An image capturing unit for capturing an image,
A second receiving unit that receives only the instruction information from the display control device;
A second display control unit configured to superimpose a predetermined object indicating the content of the instruction of the instruction information on a second display unit at a predetermined position in the image captured by the image capturing unit;
The display control system according to claim 3, further comprising: A display control method executed by a display control system having a display control device and a terminal device ,
A first identifying step in which the terminal device identifies a position of the terminal device in a predetermined space;
A transmitting step in which the terminal device transmits only position information indicating the position specified in the first specifying step;
A receiving step in which the display control device receives position information indicating a position of the terminal device in a predetermined space;
A second specifying step of specifying a position corresponding to the position information in a model representing the space, each time the display control device receives the position information in the receiving step;
The display control device, every time the position is specified in the specifying step, a display control step of displaying an image representing the model and the position specified in the specifying step on a display unit,
A display control method comprising:

Images