WO2022009872A1

WO2022009872A1 - Electronic device, control method and program

Info

Publication number: WO2022009872A1
Application number: PCT/JP2021/025440
Authority: WO
Inventors: 智志河内; 亮小山; 櫻子溝口; 泰孝金田; 聖村井
Original assignee: 京セラ株式会社
Priority date: 2020-07-08
Filing date: 2021-07-06
Publication date: 2022-01-13
Also published as: JP7450039B2; JPWO2022009872A1

Abstract

An electronic device 100 is provided with: a camera 130; a display 112 that displays a video being captured by the camera 130; and a controller 180 that causes the display 112 to display a virtual object indicating information concerning electromagnetic waves in an imaging range of the camera 130 such that the virtual object overlaps with the video being captured. The controller 180 controls a display mode of the virtual object on the basis of the intensity of the electromagnetic waves in the imaging range.

Description

Electronics, control methods, and programs

The present invention relates to electronic devices, control methods, and programs.

Conventionally, a method of calculating the strength of electromagnetic waves and the loudness of sound for each position in a real environment by simulation in a virtual environment and displaying the simulation results is known (see, for example, Patent Document 1). According to such a method, the user can determine the position where the electromagnetic wave or sound source is arranged or rearranged based on the simulation result without actually measuring the strength or loudness of the electromagnetic wave.

International Publication No. 2019/176747

The electronic device according to the first aspect is described by superimposing a camera, a display displaying an image being imaged by the camera, and a virtual object representing information on electromagnetic waves within the imaging range of the camera on the image being imaged. It is equipped with a controller to be displayed on the display. The controller controls the display mode of the virtual object based on the intensity of the electromagnetic wave within the imaging range.

The control method according to the second aspect is a method of controlling an electronic device. The control method is to display an image being imaged by a camera on a display, and to superimpose a virtual object representing information on electromagnetic waves within the imaging range of the camera on the image being imaged and display it on the display. It includes controlling the display mode of the virtual object based on the intensity of the electromagnetic wave within the imaging range.

The program according to the third aspect superimposes a process of displaying an image being imaged by a camera on a display on an electronic device and a virtual object representing information on electromagnetic waves within the image pickup range of the camera on the image being imaged. The process of displaying on the display and the process of controlling the display mode of the virtual object based on the intensity of the electromagnetic wave within the imaging range are executed.

It is a figure which shows the system including the electronic device which concerns on one Embodiment. It is a figure which shows the functional block composition of the electronic device which concerns on one Embodiment. It is a figure which shows the operation example of the electronic device which concerns on one Embodiment. It is a figure for demonstrating the screen display example which concerns on one Embodiment. It is a figure for demonstrating the screen display example which concerns on one Embodiment. It is a figure which shows the display example of the 2nd virtual object which concerns on change example 1. It is a figure which shows the operation which concerns on change example 1. It is a figure which shows the other operation which concerns on the modification 1. It is a figure which shows the display example of the planar virtual object which is the 1st virtual object which concerns on modification 2. FIG. It is a figure which shows the display example of the 1st virtual object which concerns on modification 3. It is a figure which shows the other screen display example which concerns on the modification example 3. It is a figure which shows the other screen display example which concerns on the modification example 3. It is a figure which shows the other screen display example which concerns on the modification example 3. It is a figure which shows the screen display example which concerns on modification 4. It is a figure which shows the method of extracting the contour of the object which concerns on change example 4.

Since electromagnetic waves are invisible, there is a problem that it is difficult for the user to grasp the strength of electromagnetic waves calculated by simulation in relation to the actual environment.

Therefore, the purpose of this disclosure is to make it easier for the user to understand the strength of electromagnetic waves by associating them with the actual environment.

The electronic device according to the embodiment will be described with reference to the drawings. In the description of the drawings, the same or similar parts are designated by the same or similar reference numerals.

The electronic device according to the embodiment can be a communication terminal such as a smartphone terminal or a tablet terminal. However, the electronic device is not limited to such a communication terminal, and may be, for example, a personal computer or a wearable terminal.

(Configuration of electronic devices)
First, the configuration of the electronic device according to the embodiment will be described. FIG. 1 is a diagram showing a system including an electronic device 100 according to an embodiment.

As shown in FIG. 1, the electronic device 100 connects to the communication network NW by wireless communication and communicates with the server 200 via the communication network NW. The electronic device 100 may be connectable to the external device 300 by wire or wirelessly. The external device 300 is an earphone, a headset, an external memory, an external camera, or the like that can be connected to the electronic device 100 by wire or wirelessly.

The electronic device 100 includes a touch panel display 110, a physical button 113, a microphone 121a, a speaker 122a, and a camera 130.

The touch panel display 110 is provided with its display surface exposed from the housing 101 of the electronic device 100. The touch panel display 110 has a touch panel 111 and a display 112. The touch panel 111 receives an operation input (touch input) to the electronic device 100. As a method for detecting touch, for example, there are a resistance film method and a capacitance method, but any method may be used.

The display 112 outputs video. The display 112 displays an image on the screen. The display 112 is, for example, a liquid crystal display or an organic EL (Electroluminescence) display. In the touch panel display 110, the display 112 is provided so as to overlap the touch panel 111, and the display area of the display 112 overlaps with the touch panel 111.

The physical button 113 accepts an operation input (press) to the electronic device 100. The physical button 113 includes, for example, a power button, a home button, and the like.

The microphone 121a accepts voice input to the electronic device 100. Further, the microphone 121a collects surrounding sounds. The speaker 122a outputs audio. Further, the speaker 122a outputs the voice of a telephone, information of various programs, and the like by voice.

The camera 130 electronically captures an image using an image sensor such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Sensor). The camera 130 is an out-camera that captures an object facing the opposite surface of the touch panel display 110. The camera 130 may be an in-camera that captures an object facing the touch panel display 110.

FIG. 2 is a diagram showing a functional block configuration of the electronic device 100 according to the embodiment.

As shown in FIG. 2, the electronic device 100 includes a touch panel 111, a display 112, a physical button 113, a voice input unit 121, a voice output unit 122, a camera 130, a sensor 140, and a storage unit 150. It has a communication interface 161, a connection unit 162, a battery 170, and a controller 180.

The touch panel 111 inputs a signal corresponding to the touch operation to the controller 180. The display 112 displays an image on the screen based on the signal input from the controller 180.

The voice input unit 121 inputs a signal corresponding to the voice received to the controller 180 to the controller 180. The voice input unit 121 may be the microphone 121a shown in FIG. 1 or may be an input interface to which an external microphone can be connected.

The voice output unit 122 outputs voice based on the signal input from the controller 180. The audio output unit 122 may be the speaker 122a shown in FIG. 1 or may be an output interface to which an external speaker can be connected.

The camera 130 converts the captured image into an electronic signal and outputs the image signal to the controller 180.

The sensor 140 detects various physical quantities and data, and outputs the detection result to the controller 180. The sensor 140 includes at least one of a position sensor 141, an acceleration sensor 142, and a geomagnetic sensor 143. At least one of these sensors may be provided in the external device 300.

The position sensor 141 detects the position of the own device and outputs the position data to the controller 180. The position sensor 141 may include a GNSS (Global Navigation Satellite System) receiver. The GNSS receiver performs positioning based on the GNSS satellite signal, and outputs GNSS position data indicating the geographical position (latitude / longitude) of the own device to the controller 180.

The acceleration sensor 142 detects the acceleration applied to the own device and outputs the acceleration data to the controller 180. The acceleration sensor 142 may be a multi-axis acceleration sensor including a plurality of acceleration sensors.

The geomagnetic sensor 143 detects the geomagnetism and outputs the direction data based on the detected geomagnetism to the controller 180.

The storage unit 150 includes at least one memory for storing programs and data. The storage unit 150 is also used as a work area for temporarily storing the processing result of the controller 180. The storage unit 150 may include any non-transitory storage medium such as a semiconductor storage medium and a magnetic storage medium. The storage unit 150 may include a plurality of types of storage media. The storage unit 150 may include a combination of a portable storage medium such as a memory card, an optical disk, or a magneto-optical disk, and a reading device for the storage medium. The storage unit 150 may include a storage device used as a temporary storage area such as a RAM (Random Access Memory).

The communication interface 161 communicates wirelessly. The wireless communication standards supported by the communication interface 161 include, for example, cellular communication standards such as 2G, 3G, and 4G, short-range wireless communication standards, and the like. Examples of short-range wireless communication standards include IEEE802.11, Bluetooth (registered trademark), IrDA (Infrared Data Association), NFC (Near Field Communication), and / or WPAN (Wiress Personal Area) et. WPAN communication standards include, for example, ZigBee®. The communication interface 161 may be provided in the external device 300.

The connection unit 162 is an interface that is electrically connected to the external device 300. The connection unit 162 may be any interface as long as it is electrically connected to the external device 300, and is, for example, a USB (Universal Serial Bus) interface.

The battery 170 stores electric power for driving its own device. The battery 170 is a secondary battery such as, for example, a lithium ion battery.

The controller 180 is an arithmetic processing unit. The arithmetic processing unit includes, but is not limited to, for example, a CPU (Central Processing Unit), a SOC (System-on-Chip), an MCU (Micro Control Unit), an FPGA (Field-Programmable Gate Array), and a coprocessor. .. Further, the controller 180 includes a GPU (Graphics Processing Unit), a VRAM (Video RAM), and the like, and causes the display 112 to execute drawing. The controller 180 comprehensively controls the operation of the electronic device 100 to realize various functions. The controller 180 executes various controls based on the operation input detected by the touch panel 111 and / or the physical button 113.

(Simulator)
Next, the simulator according to the embodiment will be described. As shown in FIG. 1, the server 200 has a simulator 201 that simulates electromagnetic waves. In one embodiment, an example in which the server 200 has the simulator 201 will be described, but the electronic device 100 may have the simulator 201.

Note that electromagnetic waves are waves that propagate through changes in electric and magnetic fields. For example, radio waves and light are included in electromagnetic waves. In the following, an example in which the simulator 201 performs a radio wave simulation, particularly a simulation on the strength of radio waves (radio wave strength) will be described. However, the simulator 201 may perform an optical simulation, particularly a simulation relating to the intensity of light.

The radio wave intensity may be RSSI (Received Signal Strength Indicator) or RSRP (Reference Signal Received Power). The radio wave intensity may be an index in which interference and noise are taken into consideration, for example, SINR (Signal to Interference and Noise power Ratio) or RSRQ (Reference Signal Received Quality).

The simulator 201 performs a radio wave simulation using a virtual environment simulating a real environment, and transmits information based on the simulation result to the electronic device 100. The real environment refers to the environment in the real space where the source of radio waves (for example, a radio base station) is placed. For example, the actual environment is a factory, an office, or the like.

A virtual environment is built in advance in the simulator 201. When the actual environment is a factory, layout information including the structure of the factory and the arrangement of industrial equipment is preliminarily incorporated in the virtual environment. When the real environment is an office, layout information including the floor plan of the office and the arrangement of desks is preliminarily incorporated in the virtual environment. The virtual environment may be modified or constructed by image recognition processing from the image obtained by the camera 130 of the electronic device 100.

The radio wave to be the target of the radio wave simulation may be a radio wave in a high frequency band such as a millimeter wave band or a sub 6 GHz band. Since the high frequency band has high straightness and is less likely to cause diffraction, it is difficult to design a communication area including the arrangement of radio base stations, and it is difficult to utilize the high frequency band.

The electronic device 100 (controller 180) displays information based on the simulation result of the simulator 201 on the display 112. As a result, the user can appropriately determine the position where the wireless base station is arranged or rearranged in the factory, office, or the like based on the simulation result, even if the user does not have advanced knowledge about the communication area design.

(AR display control)
Next, AR (Augmented Reality) display control in the electronic device 100 according to the embodiment will be described.

The electronic device 100 according to the embodiment includes a camera 130 and a display 112 that displays an image being imaged by the camera 130. Specifically, the controller 180 controls the display 112 so as to display the image being captured by the camera 130 in real time.

The controller 180 superimposes a virtual object (hereinafter, referred to as “first virtual object”) representing the radio wave intensity within the imaging range of the camera 130 on the image being imaged and displays it on the display 112. By superimposing such a first virtual object on the image being imaged and displaying it on the display 112, it becomes easy for the user to associate the radio wave intensity with the actual environment (that is, the image being imaged) and grasp it.

The controller 180 controls the display mode of the first virtual object based on the radio wave intensity within the imaging range of the camera 130. As a result, the user can easily recognize the radio field strength depending on the display mode of the first virtual object.

The controller 180 controls the display mode of the first virtual object based on the result of the radio wave simulation for calculating the radio wave intensity within the imaging range of the camera 130. Here, the simulator 201 of the server 200 executes the radio wave simulation, and the controller 180 acquires information based on the result of the radio wave simulation from the server 200 via the communication interface 161. However, the controller 180 may execute the radio wave simulation and acquire the simulation result by itself.

When the image pickup range of the camera 130 is changed according to the change of the image pickup condition of the camera 130, the controller 180 is a first virtual object so as to represent the radio wave intensity within the changed image pickup range based on the result of the radio wave simulation. The display mode of is changed. The imaging condition of the camera 130 includes, for example, at least one of the position, orientation, and angle of view of the camera 130.

For example, in a factory or office, the user points the camera 130 at a position where he / she wants to check the radio field strength while carrying the electronic device 100. At this time, the user can grasp the radio wave intensity in relation to the actual environment by checking the image being captured and the first virtual object displayed on the display 112. Then, when the camera 130 is pointed at another position, the image being imaged displayed on the display 112 is changed, and the display mode of the first virtual object is changed so as to represent the radio wave intensity at the other position. .. As a result, the user can smoothly confirm the radio wave strength at each position.

The radio wave simulation is executed in a virtual environment. Therefore, when executing the radio wave simulation, the server 200 (simulator 201) needs to specify the imaging range in the virtual environment corresponding to the imaging range in the real environment and acquire the simulation result in the specified imaging range. There are three such methods, for example, a position-based acquisition method, a video-based acquisition method using a marker, and a video-based acquisition method using no marker. The radio wave simulation may be performed by combining two or more of these three methods.

In the case of the position-based acquisition method, the controller 180 transmits, for example, the detection results of the position sensor 141, the acceleration sensor 142, and the geomagnetic sensor 143 to the server 200 via the communication interface 161. The controller 180 may further transmit camera parameters such as the angle of view of the camera 130 to the server 200. The server 200 (simulator 201) is a camera based on the position indicated by the detection result of the position sensor 141, the inclination (elevation angle, depression angle) indicated by the detection result of the acceleration sensor 142, and the direction indicated by the detection result of the geomagnetic sensor 143. The imaging range of 130 is specified. The server 200 (simulator 201) may further specify the imaging range of the camera 130 based on the camera parameters. The server 200 (simulator 201) transmits the simulation result in the specified imaging range to the electronic device 100. The server 200 (simulator 201) may transmit drawing information for drawing the first virtual object to the electronic device 100.

In the case of the video-based acquisition method using a marker, the marker is placed in the actual environment. A marker is a figure having a specific pattern to be recognized. When the controller 180 detects a marker included in the image captured by the camera 130, the controller 180 transmits marker information about the detected marker to the server 200 via the communication interface 161. The server 200 (simulator 201) specifies the imaging range of the camera 130 based on the marker information. The server 200 (simulator 201) may further specify the imaging range of the camera 130 based on the camera parameters. The server 200 (simulator 201) transmits the simulation result in the specified imaging range to the electronic device 100. The server 200 (simulator 201) may transmit drawing information for drawing the first virtual object to the electronic device 100.

In the case of the image-based acquisition method that does not use a marker, the object in the real environment is recognized by the image recognition process for the captured image. The controller 180 transmits the video data from the camera 130 to the server 200 via the communication interface 161. The server 200 (simulator 201) recognizes an object in a real environment by image recognition processing for video data and specifies an imaging range of the camera 130. The server 200 (simulator 201) may specify the position of the recognized object and specify the imaging range of the camera 130 by the matching process between the recognized object and the object in the virtual environment. The server 200 (simulator 201) transmits the simulation result in the specified imaging range to the electronic device 100. The server 200 (simulator 201) may transmit drawing information for drawing the first virtual object to the electronic device 100.

(Operation example of electronic device)
Next, an operation example of the electronic device 100 according to the embodiment will be described. FIG. 3 is a diagram showing an operation example of the electronic device 100 according to the embodiment.

As shown in FIG. 3, in step S101, the controller 180 starts AR display control. For example, the controller 180 starts the AR display control in response to the reception of the user operation for activating the AR display control application by the touch panel 111.

In step S102, the controller 180 starts control to display the image being captured by the camera 130 on the display 112 in real time. The display 112 displays the image being captured by the camera 130.

In step S103, the controller 180 acquires the simulation result of the radio field intensity within the imaging range of the camera 130.

In step S104, the controller 180 superimposes a virtual object (first virtual object) representing the radio wave intensity within the imaging range of the camera 130 on the image being imaged by the camera 130 and displays it on the display 112.

In step S105, the controller 180 determines whether or not the imaging range of the camera 130 has been changed. The controller 180 may make a determination in step S105 based on the detection result of at least one of the position sensor 141, the acceleration sensor 142, and the geomagnetic sensor 143, or the determination in step S105 based on the video data from the camera 130. May be done.

When the imaging range of the camera 130 is changed (step S105: YES), in step S103, the controller 180 acquires the simulation result of the radio wave intensity within the changed imaging range. Then, the controller 180 changes the display mode of the first virtual object so as to represent the acquired simulation result.

On the other hand, when the imaging range of the camera 130 has not been changed (step S105: NO), in step S106, the controller 180 determines whether or not to end the AR display control. When the AR display control is terminated (step S106: YES), this flow is terminated. If the AR display control is not terminated (step S106: NO), the controller 180 returns the process to step S105.

(Screen display example)
Next, a screen display example according to the embodiment will be described. 4 and 5 are diagrams for explaining a screen display example according to an embodiment.

As shown in FIG. 4, the simulator 201 calculates the radio field intensity by radio wave simulation for each cube obtained by dividing the virtual environment into cube groups having grid widths at equal intervals. In the example shown in FIG. 4, an example in which a cube group is composed of a total of 27 cubes having three lengths, three widths, and three heights is shown. The simulation result of the radio field strength is assigned to each cube.

As shown in FIG. 5, the controller 180 superimposes a first virtual object including an individual object individually assigned to each cube on the image being imaged and displays it on the display 112. FIG. 5 shows an example in which the image being imaged is an office image. Further, FIG. 5 shows an example in which each individual object is a translucent sphere.

Each individual object represents the signal strength in the corresponding cube. For example, an individual object expresses the radio field intensity in the corresponding cube by the color of the individual object or the size of the individual object. When the radio field strength is expressed by the color of an individual object, "red" may be assigned to the radio wave strength "high", "yellow" may be assigned to the radio wave strength "medium", and "blue" may be assigned to the radio wave strength "low". When expressing the radio field strength by the size of an individual object, the "large" size is assigned to the radio field strength "high", the "medium" size is assigned to the radio field strength "medium", and the "small" size is assigned to the radio field strength "low". May be good.

According to such a screen display example, the user can grasp the radio field strength in the actual environment (office) three-dimensionally. As a result, for example, the user can specify a position where the radio wave strength is weak based on the screen display, change the installation position of the radio base station, or install a radio wave reflector.

(Summary of embodiments)
In the electronic device 100 according to the embodiment, the camera 130, the display 112 displaying the image being imaged by the camera 130, and the first virtual object representing the radio wave intensity within the imaging range of the camera 130 are superimposed on the image being imaged. It has a controller 180 to be displayed on the display 112. This makes it easy for the user to understand the radio field strength in relation to the actual environment.

(Change example 1)
Next, modification 1 of the embodiment will be described.

As mentioned above, the radio wave simulation is executed in the virtual environment, but the virtual environment does not always exactly match the real environment. If the radio wave simulation is performed using a virtual environment that does not match the real environment, the correct radio wave strength cannot be calculated.

In addition, when trying to determine the installation position of a wireless base station before the start of operation of a factory or office, we would like to grasp the radio field strength in relation to the actual environment while considering the arrangement of objects such as industrial equipment or desks. There is a need. Furthermore, when changing the layout of a factory or office, there is also a need to grasp the state of radio field strength in the actual environment after the layout change.

Therefore, the controller 180 according to the first modification superimposes a virtual object (hereinafter, referred to as “second virtual object”) representing an object within the imaging range of the camera 130 on the image being imaged in the virtual environment used for the radio wave simulation. Is displayed on the display 112. In other words, the controller 180 causes the display 112 to display an object constituting the simulation condition of the radio wave simulation as a second virtual object.

This allows the user to confirm whether the simulation conditions (virtual environment) match the actual environment, so that it is possible to determine whether or not the simulation results are appropriate.

The controller 180 may display the second virtual object on the display 112 at the same time as the first virtual object. The controller 180 may display the first virtual object on top of the second virtual object when the first virtual object and the second virtual object overlap at least partially.

The controller 180 may display the second virtual object on the display 112 at a timing different from the display timing of the first virtual object. For example, the controller 180 may switch between the display of the first virtual object and the display of the second virtual object according to the user operation.

FIG. 6 is a diagram showing a display example of the second virtual object according to the modification example 1.

As shown in FIG. 6, the controller 180 superimposes a second virtual object representing an object within the imaging range of the camera 130 on the image being imaged and displays it on the display 112 in the virtual environment used for the radio wave simulation. FIG. 6 shows an example in which the object represented by the second virtual object is a sofa, and the sofa is not installed in the actual environment.

For example, the user performs a user operation on the electronic device 100 (touch panel 111) to add a second virtual object representing the sofa before newly installing the sofa in the real environment. In this case, the simulator 201 performs re-simulation using a virtual environment reflecting the addition of the sofa, and changes the display mode of the first virtual object representing the radio field intensity based on the result of the re-simulation. This allows the user to understand the effect of adding a sofa on the signal strength.

Alternatively, a sofa may be installed in the actual environment. In this case, the image being captured by the camera 130 includes the sofa. Under such circumstances, the controller 180 superimposes a second virtual object representing an object (here, a sofa) within the imaging range of the camera 130 on the image being imaged on the display 112 in the virtual environment used for the radio wave simulation. Display.

Here, when the position of the sofa included in the image being captured and the position of the second virtual object are different, the user electronically moves the second virtual object to the position of the sofa included in the image being captured. This is performed for the device 100 (touch panel 111). As a result, the simulation conditions (virtual environment) can be matched with the real environment, so that appropriate simulation results can be obtained. In this case, the simulator 201 performs re-simulation using a virtual environment in which the movement of the second virtual object is reflected, and changes the display mode of the first virtual object representing the radio wave intensity based on the result of the re-simulation.

FIG. 7 is a diagram showing the operation according to the modification example 1. Here, an example of displaying the second virtual object on the display 112 at the same time as the first virtual object will be described.

As shown in FIG. 7, in step S201, the controller 180 starts AR display control. For example, the controller 180 starts the AR display control in response to the reception of the user operation for activating the AR display control application by the touch panel 111.

In step S202, the controller 180 starts control to display the image being captured by the camera 130 on the display 112 in real time. The display 112 displays the image being captured by the camera 130.

In step S203, the controller 180 acquires the simulation result of the radio wave intensity in the image pickup range of the camera 130 and the information about the object in the image pickup range of the camera 130 in the virtual environment used for the radio wave simulation from the simulator 201.

In step S204, the controller 180 represents a virtual object (first virtual object) representing the radio field intensity within the image pickup range of the camera 130 and a virtual object (first virtual object) representing an object within the image pickup range of the camera 130 in the virtual environment used for radio wave simulation. The second virtual object) is superimposed on the image being captured by the camera 130 and displayed on the display 112.

In step S205, the controller 180 determines whether or not the imaging range of the camera 130 has been changed. The controller 180 may make a determination in step S205 based on the detection result of at least one of the position sensor 141, the acceleration sensor 142, and the geomagnetic sensor 143, or the determination in step S205 based on the video data from the camera 130. May be done.

When the image pickup range of the camera 130 is changed (step S205: YES), in step S203, the controller 180 is within the image pickup range of the camera 130 in the virtual environment and the simulation result of the radio wave intensity in the changed image pickup range. Information about the object is acquired from the simulator 201, and the display of the first virtual object and the second virtual object is updated.

On the other hand, when the imaging range of the camera 130 has not been changed (step S205: NO), in step S206, the controller 180 determines whether or not to end the AR display control. When the AR display control is terminated (step S206: YES), this flow is terminated. If the AR display control is not terminated (step S206: NO), the controller 180 returns the process to step S205.

FIG. 8 is a diagram showing another operation according to the modification example 1. The operation shown in FIG. 8 is started, for example, by a user operation on the touch panel 111 as a trigger.

As shown in FIG. 8, in step S301, the controller 180 determines whether or not there has been a user operation representing the movement, addition, or deletion of the second virtual object.

When there is a user operation representing the movement, addition, or deletion of the second virtual object (step S301: YES), in step S302, the controller 180 is the result of re-simulation using the virtual environment in which the user operation is reflected. Is obtained from the simulator 201. In step S303, the controller 180 changes the display mode of the first virtual object based on the result of the re-simulation.

For example, when there is a user operation representing the movement of the second virtual object, the controller 180 changes the display mode of the first virtual object based on the result of the re-simulation using the virtual environment reflecting the movement. ..

Alternatively, when there is a user operation indicating the addition of the second virtual object, the controller 180 changes the display mode of the first virtual object based on the result of the re-simulation using the virtual environment reflecting the addition. ..

Alternatively, when there is a user operation indicating the deletion of the second virtual object, the controller 180 changes the display mode of the first virtual object based on the result of the re-simulation using the virtual environment in which the deletion is reflected. ..

(Change example 2)
Next, modification 2 of the embodiment will be described.

In the case of the screen display example as shown in FIG. 5, it is difficult to understand the context of each individual object (each sphere in FIG. 5) included in the first virtual object representing the radio field strength, and the user can grasp the sense of depth. There are difficult concerns. Further, in the case of the screen display example as shown in FIG. 5, since the direction in which the radio wave travels (radio wave traveling direction) cannot be expressed, the user cannot grasp the radio wave traveling direction.

Therefore, the controller 180 according to the second modification causes the display 112 to display a planar virtual object parallel to the floor surface of the space included in the image being captured as the first virtual object representing the radio wave intensity. The planar virtual object is a first virtual object that displays information in the horizontal direction. By displaying the planar virtual object on the display 112 as the first virtual object representing the radio wave strength, the user can easily grasp the sense of depth.

FIG. 9 is a diagram showing a display example of a planar virtual object which is the first virtual object according to the modification example 2. In FIG. 9, a room is assumed as a real environment. The floor surface, wall # 1, wall # 2, and pillars shown in FIG. 9 are assumed to be objects existing in the real environment. It is assumed that the information of the floor surface, the wall # 1, the wall # 2, and the pillar is also incorporated in the virtual environment used for the radio wave simulation.

As shown in FIG. 9, the planar virtual object is the first virtual object parallel to the floor surface of the space included in the image being captured. In FIG. 9, the planar virtual object is a translucent virtual object. The controller 180 causes the display 112 to display the planar virtual object that expresses the radio wave intensity by the shade of color or the type of color for each region (area A to C) included in the planar virtual object.

In FIG. 9, it is assumed that the radio base station, which is the source of radio waves, is installed on the wall # 1 side. Therefore, the radio field intensity in the region A near the wall # 1 is the highest. The region B separated from the wall # 1 has a lower radio field strength than the region A. When the radio field strength is expressed by the type of color, "red" may be assigned to the region A having the radio wave strength "high", and "yellow" may be assigned to the region B having the radio wave strength "low". When the radio field intensity is expressed by the shade of color, "dark red" may be assigned to the region A having the radio wave strength "high", and "light red" may be assigned to the region B having the radio wave strength "low". According to such a screen display example, the user can easily grasp the depth of the distribution of the radio field intensity.

Area C is an area where radio waves do not pass because there is a pillar between it and the source of radio waves. The controller 180 causes the display 112 to display the region C through which radio waves do not pass in the planar virtual object in a predetermined color (for example, black or white). As a result, the user can grasp that the area C is an area through which radio waves do not pass.

Further, the controller 180 causes the display 112 to display a planar virtual object including a symbol or a figure indicating the direction in which the radio wave travels (radio wave traveling direction). Although FIG. 9 shows an example of displaying an arrow as a symbol indicating the radio wave traveling direction, the present invention is not limited to the arrow, and any symbol or figure indicating the radio wave traveling direction can be used. As a result, the user can easily grasp the radio wave traveling direction.

Note that the controller 180 may display the average value of the radio wave intensity in the height direction in the planar virtual object. When the height of the planar virtual object is specified by the user operation, the controller 180 may display the radio field intensity at the specified height. The controller 180 may continuously change the display so as to display the radio field intensity at the corresponding height in response to the user operation of moving the planar virtual object in the height direction.

In FIG. 9, it is assumed that there is only one source of radio waves, but it is also possible to assume that there are multiple sources of radio waves. When there are a plurality of radio wave sources, the controller 180 may display the planar virtual objects individually assigned to each radio wave source on the display 112 at different heights and different colors.

For example, when radio waves arrive from each of the radio base station A and the radio base station B in one room, the controller 180 is assigned to the planar virtual object A and the radio base station B assigned to the radio base station A. The planar virtual object B is displayed on the display 112. Here, the controller 180 causes the planar virtual object A and the planar virtual object B to be displayed on the display 112 at different heights and different colors. This makes it easier for the user to distinguish between the two planar virtual objects. Further, by displaying the planar virtual object A and the planar virtual object B at the same time, the user can grasp the existence of the dead region that neither the radio base station A nor the radio base station B covers.

When there are a plurality of radio wave sources, the controller 180 may display the planar virtual objects individually assigned to each radio wave source on the display 112 at different timings. For example, the controller 180 may switch between the display of the planar virtual object A and the display of the planar virtual object B according to the user operation.

(Change example 3)
Next, modification 3 of the embodiment will be described. Modification 3 is an example of improving the screen display of the first virtual object representing the radio field intensity, as in the modification 2.

The controller 180 according to the third modification causes the display 112 to display the first virtual object associated with the object included in the image being captured by the camera 130. Here, the object refers to an object existing in the real environment, but may be an object existing in the virtual environment.

The first virtual object includes at least one of a numerical value, a symbol, a figure, and a character representing an object or a radio field intensity in the vicinity of the object. In the following, an example of expressing the radio wave strength numerically will be described, but in the first virtual object, the radio wave strength may be represented by, for example, a symbol or a figure of the number of antennas, and the radio wave strength is “high”, “medium”, and so on. It may be represented by characters such as "low" and "zero".

By displaying the first virtual object in association with the object in this way, it becomes easier for the user to grasp which object the radio field intensity is for. Further, by displaying the first virtual object only on the object, it is possible to prevent the first virtual object from occupying most of the display area of the display 112, so that the image being captured by the camera 130 can be prevented from being occupied. Visibility can be improved.

FIG. 10 is a diagram showing a display example of the first virtual object according to the modification example 3. In FIG. 10, an office is assumed as a real environment. The desk and chair shown in FIG. 10 are assumed to be objects existing in the real environment. It is assumed that the desk and chair information is also incorporated in the virtual environment used for the radio wave simulation.

Here, an example in which the object associated with the first virtual object is a desk will be described. When the actual environment is a factory, the object may be equipment such as industrial equipment.

As shown in FIG. 10, the controller 180 superimposes the first virtual object on at least a part of the object and displays it on the display 112. In FIG. 10, four first virtual objects (first virtual object # 1 to first virtual object # 4) displayed on top of each of the four desks are illustrated. Each first virtual object is a numerical value representing the radio field intensity at the corresponding desk.

The controller 180 may display the first virtual object on the display 112 with the object designated by the user operation as the object among the plurality of objects included in the image being captured by the camera 130. That is, the user may be able to specify the object to which the first virtual object is assigned. Thereby, the object for displaying the first virtual object can be limited.

The controller 180 may omit the display of the first virtual object for the object whose entire image is not included in the image captured by the camera 130 among the plurality of objects included in the image captured by the camera 130. For example, the first virtual object is not displayed for an object that includes only a part at the edge of the display area of the display 112. Thereby, the object for displaying the first virtual object can be limited.

When the plurality of first virtual objects corresponding to the plurality of objects included in the image being captured by the camera 130 overlap at least partially, the controller 180 is the target located in front of the plurality of first virtual objects. Only the first virtual object corresponding to the object may be displayed on the display 112. That is, the controller 180 does not display the first virtual object on the display 112 for the object located behind the object located in front. This makes it possible to improve the visibility of the first virtual object.

The controller 180 may display a numerical value indicating the radio wave intensity on the display 112 for each region of a predetermined range in the object, and may display a boundary line representing the region of the predetermined range on the display 112. FIG. 11 is a diagram showing another screen display example according to the modification example 3.

As shown in FIG. 11A, when displaying one first virtual object for one object, if a large object or a large wall surface is the object, the numerical value changes for each part of the object. I can't express it. The upper left portion shown in FIG. 11A has a radio field intensity of “30”, while the lower right portion shown in FIG. 11A has a radio wave strength of “100”. In the example of FIG. 11A, "60" is displayed as a numerical value of the average radio field intensity.

Therefore, as shown in FIG. 11B, the controller 180 displays a numerical value (representative value such as an average value) representing the radio wave intensity on the display 112 for each region of a predetermined range in the object, and also displays the predetermined value. A border representing the area of the range is displayed on the display 112. Here, the region in a predetermined range means a region in which the value range of the numerical value of the radio wave intensity is in a specific range. In the example shown in FIG. 11B, the representative value is “10”, the representative value is “50”, the representative value is “70”, and the representative value is “90”. It is divided into a total of four areas with a certain area by a boundary line. As a result, even when a large object or a large wall surface is the object, the change in the numerical value for each part of the object can be expressed.

After displaying a numerical value indicating the radio wave intensity on the display 112, the controller 180 displays a new numerical value indicating the radio wave intensity at the position of the movement destination by the user operation on the display 112 when there is a user operation to move the numerical value. You may let me. FIG. 12 is a diagram showing another screen display example according to the modification example 3.

As shown in FIG. 12A, when displaying one first virtual object for one object, if a large object or a large wall surface is the object, the numerical value changes for each part of the object. I can't express it.

Therefore, as shown in FIG. 12B, when there is a user operation to move the numerical value on the displayed object, the controller 180 is a new numerical value representing the radio wave intensity at the position of the movement destination by the user operation. Is displayed on the display 112. FIG. 12B shows an example in which the numerical value changes in the order of “30”, “80”, and “50” according to the movement. As a result, even when a large object or a large wall surface is the object, the change in the numerical value for each part of the object can be expressed.

The controller 180 may be displayed on the display 112 in such a manner that the first virtual object is attached to the object. FIG. 13 is a diagram showing another screen display example according to the modification example 3. FIG. 13 shows an example of numerically displaying the radio field strength of the floor surface and the wall surface of the corridor in the room. The controller 180 displays the first virtual object (numerical value) at an angle corresponding to the depth. This makes it easier for the user to grasp the position of the object associated with the first virtual object (numerical value).

FIG. 13A shows an example of displaying the first virtual object (numerical value) in a manner of being attached to the floor surface in front, and the displayed numerical value is not angled. FIG. 13B shows an example of displaying the first virtual object (numerical value) in a manner of being attached to the wall surface, and since the numerical value to be displayed is angled, the numerical value is displayed toward the back side. Is shrinking. FIG. 13C shows an example in which the first virtual object (numerical value) is displayed in a manner of being attached to a distant floor surface, and since the numerical values to be displayed are angled, the numerical values are numerical values in the vertical direction. The display is compressed.

(Change example 4)
Next, modification 4 of the embodiment will be described. Modification 4 is an embodiment for improving the screen display. As described above, when the first virtual object is displayed on the image being imaged by the camera 130, there is a problem that the visibility of the image being imaged by the camera 130 is lowered.

When the object included in the image being captured by the camera 130 overlaps with the first virtual object, the controller 180 according to the fourth modification superimposes a line representing the outline of the object on the first virtual object and displays it on the display 112. .. This makes it possible to improve the visibility of the object hidden behind the first virtual object. The object included in the image being captured by the camera 130 means an object existing in the real environment, but may be an object existing in the virtual environment.

FIG. 14 is a diagram showing a screen display example according to the modification example 4.

As shown in FIG. 14A, the controller 180 causes the display 112 to display the image being captured by the camera 130. FIG. 14A shows an example in which the objects included in the image being captured by the camera 130 are a plurality of microwave ovens and shelves.

As shown in FIG. 14 (b), the controller 180 causes the display 112 to display a first virtual object that expresses the radio wave intensity by a shade of color or a type of color over the entire image being captured by the camera 130. FIG. 14B shows an example of expressing the radio wave intensity by the shade of color. Here, the image being captured by the camera 130 is hidden by the first virtual object, and the visibility of the image is reduced. In FIG. 14B, it is assumed that the radio field intensity on the left side where the color is light is high and the radio wave intensity on the right side where the color is dark is low.

As shown in FIG. 14 (c), the controller 180 superimposes the first virtual object on the image being imaged by the camera 130 and displays it on the display 112, and at the same time, displays the outline of the object included in the image being imaged by the camera 130. The line to be represented is superimposed on the first virtual object and displayed on the display 112. As a result, the user can grasp the outline of the object included in the image being captured by the camera 130, so that the object hidden by the first virtual object can be roughly grasped. Further, the user can grasp that the radio wave intensity is lowered from the position of the second object (microwave oven) from the front based on the screen display as shown in FIG. 14 (c).

FIG. 15 is a diagram showing a method of extracting the contour of the object according to the modified example 4.

As shown in FIG. 15A, the controller 180 identifies a part where the depth of the object is cut off when viewed from a viewpoint (that is, the position of the camera 130), extracts the specified part, and borders the object. On the other hand, as shown in FIG. 15A, the controller 180 does not border the portion where the depth of the object is continuous when viewed from the viewpoint (that is, the position of the camera 130). That is, the controller 180 does not border the image in a range where the distance from the position imaged by the camera 130 to the position of the object included in the image is continuous, and the object included in the image from the position imaged by the camera 130. The image is bordered for the range where the distance to the position of is not continuous.

(Other embodiments)
The above-described embodiment and each modification can be implemented in any combination. For example, the modification 1 and the modification 2 to 4 may be combined, or the modification 4 and the modification 2 or 3 may be combined.

In the above-described embodiment, an example in which the controller 180 displays the first virtual object representing the radio wave intensity calculated by the radio wave simulation on the display 112 has been described. However, the controller 180 may display a first virtual object representing the actually measured radio field intensity on the display 112. For example, the controller 180 may periodically measure the radio field intensity using the communication interface 161 and display a first virtual object representing each measurement result on the display 112.

In the above-described embodiment, an example in which the simulator 201 performs a radio wave simulation has been described, but the simulator 201 may perform a simulation related to sound (sound wave). In this case, the "radio wave intensity" in the above-described embodiment may be read as "loudness". Specifically, the controller 180 causes the display 112 to display a first virtual object representing the loudness calculated by the sound wave simulation.

A program may be provided that causes a computer to execute each process performed by the electronic device 100. The program may be recorded on a computer-readable medium. Computer-readable media can be used to install programs on a computer. Here, the computer-readable medium on which the program is recorded may be a non-transient recording medium. The non-transient recording medium is not particularly limited, but may be, for example, a recording medium such as a CD-ROM or a DVD-ROM.

Although one embodiment has been described in detail with reference to the drawings above, the specific configuration is not limited to the above, and various design changes and the like can be made within a range that does not deviate from the gist. ..

This application claims the priority of Japanese Patent Application No. 2020-117751 (filed on July 8, 2020), the entire contents of which are incorporated in the specification of the present application.

Claims

With the camera
A display that displays the image being captured by the camera, and
A controller that superimposes a virtual object representing information on electromagnetic waves within the imaging range of the camera on the image being captured and displays it on the display.
The controller is an electronic device that controls a display mode of the virtual object based on the intensity of the electromagnetic wave within the imaging range.
The electronic device according to claim 1, wherein the controller controls a display mode of the virtual object based on the result of a simulation for calculating the intensity of the electromagnetic wave in the imaging range.
When the imaging range is changed in response to a change in the imaging conditions of the camera, the controller virtualizes the electromagnetic wave intensity within the changed imaging range based on the result of the simulation. The electronic device according to claim 2, wherein the display mode of the object is changed.
The electronic device according to any one of claims 1 to 3, wherein the controller displays, as the virtual object, a planar virtual object parallel to the floor surface of the space included in the image being captured on the display. device.
The electronic device according to claim 4, wherein the controller displays the planar virtual object on the display, which expresses the intensity of the electromagnetic wave by a shade of color or a type of color for each region included in the planar virtual object. ..
The electronic device according to claim 5, wherein the controller displays a region of the planar virtual object through which electromagnetic waves do not pass on the display in a predetermined color.
The electronic device according to any one of claims 4 to 6, wherein the controller displays the planar virtual object including a symbol or a figure indicating a traveling direction of the electromagnetic wave on the display.
When the controller has a plurality of sources of electromagnetic waves, the controller displays the planar virtual objects individually assigned to each source on the display at different heights and different colors. The electronic device according to any one of 7.
The controller according to any one of claims 4 to 7, wherein when there are a plurality of sources of the electromagnetic wave, the planar virtual objects individually assigned to each source are displayed on the display at different timings. The listed electronic device.
The controller displays the virtual object associated with the object included in the image being captured on the display.
The electronic device according to any one of claims 1 to 9, wherein the virtual object includes at least one of a numerical value, a symbol, and a character representing the intensity of the electromagnetic wave in the object or in the vicinity of the object.
The electronic device according to claim 10, wherein the controller superimposes the virtual object on at least a part of the object and displays it on the display.
The electronic device according to claim 10 or 11, wherein the controller displays the virtual object on the display with an object designated by a user operation among a plurality of objects included in the image being captured as the object.
The electronic device according to claim 10 or 11, wherein the controller omits the display of the virtual object for an object whose entire image is not included in the image being imaged, among a plurality of objects included in the image being imaged. device.
When a plurality of virtual objects corresponding to a plurality of objects included in the image being captured overlap at least partially, the controller corresponds to the virtual object located in front of the plurality of virtual objects. The electronic device according to claim 10 or 11, wherein only the object is displayed on the display.
15. 11. The electronic device according to 11.
The controller displays a numerical value indicating the strength of the electromagnetic wave on the display, and then when there is a user operation to move the numerical value, the controller newly indicates the strength of the electromagnetic wave at the position of the destination by the user operation. The electronic device according to claim 10 or 11, wherein the numerical value is displayed on the display.
The electronic device according to claim 10 or 11, wherein the controller displays the virtual object on the display in a manner of attaching the virtual object to the object.
One of claims 1 to 17, when the controller overlaps an object included in the image being captured with the virtual object, the controller superimposes a line representing the outline of the object on the virtual object and displays it on the display. The electronic device described in the section.
The controller displays the virtual object, which expresses the intensity of the electromagnetic wave by a shade of color or a type of color, on the display over the entire image being imaged, and displays a line representing the outline of the object on the virtual object. The electronic device according to claim 18, which is superimposed on the display and displayed on the display.
It is a control method for controlling electronic devices.
Displaying the image being captured by the camera on the display and
A virtual object representing information on electromagnetic waves within the imaging range of the camera is superimposed on the image being captured and displayed on the display.
A control method comprising controlling a display mode of the virtual object based on the intensity of the electromagnetic wave within the imaging range.
For electronic devices
The process of displaying the image being captured by the camera on the display,
A process of superimposing a virtual object representing information on electromagnetic waves within the imaging range of the camera on the image being captured and displaying it on the display.
A program for executing a process of controlling the display mode of the virtual object based on the intensity of the electromagnetic wave in the imaging range.