JP2023172851A - Program, information processing apparatus, method, and system - Google Patents

Abstract

To provide a program, a method and a system for allowing a target device to be operated from a user terminal without advance preparation.SOLUTION: A program for operating a terminal device, such as a smartphone comprising a processor, a memory, a camera and a three-dimensional scanner causes a processor to execute the steps of: acquiring image data obtained by a camera imaging a real space; acquiring, through a three-dimensional scanner, three-dimensional position data of a surface of an object existing in the real space including at least an imaging range of the camera; causing a display to display the image data; associating the image data with the three-dimensional position data; accepting designation of specific image data out of image data displayed on the display, to accept designation of a specific spot on the surface of the object; and acquiring three-dimensional position data of the specific spot on the basis of position information of the camera, an imaging direction of the camera, the designation of the specific spot and the three-dimensional position data.SELECTED DRAWING: Figure 3

Description

The present disclosure relates to a program, an information processing device, a method, and a system.

There is a technique in which a predetermined position on an image acquired by a photographing unit mounted on a user terminal is specified on the screen of the user terminal, and a flying object is moved to the specified position (Patent Document 1).

JP 2021-162572 Publication

In the technology according to Patent Document 1, it is necessary to obtain three-dimensional coordinate data in virtual space in advance using a sensor such as LiDAR (Laser Imaging Detection and Ranging). Therefore, if three-dimensional coordinate data in the virtual space cannot be acquired in advance, the technique described in Patent Document 1 may not be able to operate the flying object.

Therefore, the present disclosure has been made to solve the above problems, and its purpose is to enable a user terminal to operate an instruction target device without prior preparation.

This is a program for operating a computer including a processor, a memory, an imaging device, and a three-dimensional position acquisition device. The program causes the processor to acquire image data obtained by imaging a real space with an imaging device, and to obtain three-dimensional position data of the surface of an object existing in the real space including at least an imaging range of the imaging device. Acquisition by an acquisition device; displaying image data on a display device; correlating the image data with three-dimensional position data; and accepting designation of specific image data among the image data displayed on the display device. This step includes accepting the designation of a specific location on the surface of the object, and obtaining 3D position data of the specific location based on the position information of the imaging device, the imaging direction of the imaging device, the designation of the specific location, and the 3D location data. Execute the steps to obtain.

According to the present disclosure, an instruction target device can be operated from a user terminal without prior preparation.

FIG. 2 is a diagram for explaining an overview of the operation of the system. FIG. 1 is a diagram showing the overall configuration of the system. FIG. 2 is a diagram showing a functional configuration of a terminal device. FIG. 3 is a diagram showing the data structure of a database. FIG. 3 is a diagram showing the data structure of a database. 3 is a flowchart showing an example of a processing flow in the system. 3 is a flowchart showing an example of a processing flow in the system. FIG. 2 is a schematic diagram showing an example of a screen displayed on a terminal device. FIG. 3 is a schematic diagram showing another example of a screen displayed on a terminal device. FIG. 7 is a schematic diagram showing yet another example of a screen displayed on a terminal device.

Embodiments of the present disclosure will be described below with reference to the drawings. In all the figures explaining the embodiments, common components are given the same reference numerals and repeated explanations will be omitted. Note that the following embodiments do not unduly limit the content of the present disclosure described in the claims. Furthermore, not all components shown in the embodiments are essential components of the present disclosure. Furthermore, each figure is a schematic diagram and is not necessarily strictly illustrated.

Furthermore, in the following description, a "processor" refers to one or more processors. The at least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be another type of processor such as a GPU (Graphics Processing Unit). At least one processor may be single-core or multi-core.

Furthermore, at least one processor may be a broadly defined processor such as a hardware circuit (for example, an FPGA (Field-Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit)) that performs part or all of the processing.

In addition, in the following explanation, information such as "xxx table" may be used to explain information that provides an output in response to an input, but this information may be data of any structure, and A learning model such as a generated neural network may also be used. Therefore, the "xxx table" can be called "xxx information."

In addition, in the following explanation, the configuration of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table. good.

In addition, in the following description, processing may be explained using the subject "program", but a program is executed by a processor to carry out a prescribed process, and to use the storage unit and/or interface unit as appropriate. Since the processing is performed while using the processor, the subject of the processing may be a processor (or a device such as a controller having the processor).

The program may be installed on a device such as a computer, or may be located on, for example, a program distribution server or a computer-readable (eg, non-transitory) recording medium. Furthermore, in the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

Furthermore, in the following description, identification numbers are used as identification information for various objects, but other types of identification information than identification numbers (for example, identifiers containing alphabetic characters or codes) may be employed.

In addition, in the following explanation, when the same type of elements are explained without distinguishing them, reference numerals (or common numerals among the reference numerals) are used, and when the same kind of elements are explained separately, the element An identification number (or reference number) may be used.

In addition, in the following description, control lines and information lines are shown to be necessary for the explanation, and not all control lines and information lines are necessarily shown in the product. All configurations may be interconnected.

<0 System overview>
FIG. 1 is a diagram showing an overview of the operation of a system according to the present disclosure. In the system of the present disclosure, an imaging device such as a camera included in a terminal device such as a smartphone captures an image of a three-dimensional space, which is a real space in front of the terminal device, to obtain image data. Further, three-dimensional position data of the surface of an object existing in a three-dimensional space including the imaging range by this imaging device is similarly acquired by a three-dimensional position acquisition device such as LiDAR provided in the terminal device. Next, these image data and three-dimensional position data are associated with each other. Then, the image data is displayed on a display device such as a display of the terminal device, and a specific part of the image data (image data and three-dimensional position data (This specific location is also associated with a specific location on the surface of the object.) By specifying this specific location, it is possible to calculate the three-dimensional position data of the specific location.

As shown in FIG. 1, for example, the owner of the terminal device 1 is in a flat parking lot 2 marked with a lot line 2a, and is trying to park a car 3 in a parking area 2b separated by the lot line 2a. shall be. In this case, it is assumed that the automobile 3 has an automatic parking function that is an example of advanced driver-assistance systems (ADAS).

The automatic parking function uses various devices included in the car 3, such as an in-vehicle camera that can capture a 360° image around the car 3, a proximity sensor such as an ultrasonic sensor that can detect obstacles, and a GPS (Global Positioning System). Using GNSS (Global Navigation Satellite System) devices, such as devices, the car 3 autonomously grasps the surrounding situation, and the car 3 autonomously moves toward a three-dimensional position specified by the owner of the terminal device. and parking function. Systems with similar functions have already been realized (Reference: “Clarion | Develops advanced automatic parking technology “Park by Memory” that remembers the surrounding environment of parking lots such as homes”, [online], 2018) January 9, 2020, Clarion Co., Ltd., [Retrieved May 13, 2020], Internet <URL: https://www.clarion.com/jp/ja/corp/information/news-release/2018/ 0109-1/index.html>).

The terminal device 1 is equipped with a camera (not shown in FIG. 1) that is an example of an imaging device, and a LiDAR (also not shown in FIG. 1) that is an example of a three-dimensional position acquisition device. The three-dimensional position acquisition range of the object by LiDAR includes at least the imaging range by the camera.

The owner of the terminal device 1 who wishes to park the automobile 3 in the parking area 2b first images an area including the parking area 2b where he/she wishes to park the automobile 3 using the camera of the terminal device 1. The camera images an imaging range specified by the owner of the terminal device 1 and obtains image data. Image data acquired by the camera is displayed on the screen 1a of the terminal device 1.

LiDAR of the terminal device 1 images an imaging range with a camera, and then acquires three-dimensional position data of the surface of an object existing in a three-dimensional space that includes at least this imaging range. At this time, if the 3D position data is acquired by LiDAR immediately after the image data is acquired by the camera (in other words, without much time), the imaging range by the camera and the acquisition range of 3D position data by LiDAR can be expanded. It never shifts.

At this time, there is no need to clearly indicate on the screen 1a of the terminal device 1 that the three-dimensional position data acquisition operation using LiDAR is being performed. In other words, if three-dimensional positioning is performed using LiDAR without displaying anything on the screen 1a from the imaging operation by the camera, the owner of the terminal device 1 will see that the imaging operation by the camera is taking a certain amount of time. During that time, if you move the terminal device 1 significantly, the image will be blurred, so you can be guided to keep the terminal device 1 as still as possible, so the accuracy of the 3D position data obtained by LiDAR (data is not blurred), and the matching rate between the imaging range by the camera and the acquisition range of three-dimensional position data by LiDAR may be further increased.

After this, the image data acquired by the camera and the three-dimensional position data acquired by LiDAR are correlated. The details of the correspondence will be described later, but while image data is data in pixel units and is two-dimensional data, three-dimensional position data (at this point, it is not absolute coordinates but relative coordinates with the location of the terminal device 1 as the origin). However, since the data is literally three-dimensional data, for example, the correspondence may be performed by projecting the three-dimensional position data onto a two-dimensional plane defined by the screen 1a of the terminal device 1.

Next, the owner of the terminal device 1 specifies a location 1b on the screen 1a that corresponds to the target position 4 where he or she wants to park the car 3, such as by touching the screen 1a. When the owner specifies a location 1c, the terminal device 1 specifies the three-dimensional position data of the target position 4 corresponding to this location 1c. Preferably, geographical position information (latitude, longitude, altitude) of the terminal device 1 and three-dimensional coordinates with the terminal device 1 as the origin are obtained by an acceleration sensor and a GPS sensor (not shown in FIG. 1) installed in the terminal device 1. The inclinations around the X, Y, and Z axes that constitute the terminal device 1 are obtained, and thereby the position information (uniquely determined on the earth) of the terminal device 1 and the imaging direction of the camera are obtained. As a result, the three-dimensional position data by LiDAR can be converted into geographical position data, and as a result, it is converted into the geographical position data uniquely determined on the earth at the target position 4 specified by the owner of the terminal device 1. can do.

After this, the owner of the terminal device 1 can transmit the geographical position data of the target position 4 to the car 3 and instruct it to perform a parking operation toward this geographical position data (target position 4). .

<1 Configuration diagram of the entire system>
FIG. 2 is a block diagram showing an example of the overall configuration of the system S. The system S shown in FIG. 2 includes, for example, a terminal device 10 and an instruction target device 20. The terminal device 10 and the instruction target device 20 are communicably connected via a network 80, for example.

Although FIG. 2 shows an example in which the system S includes one terminal device 10, the number of terminal devices 10 included in the system S is not limited to one. The terminal device 10 is a terminal owned by a person who specifies a target position when instructing the instruction target device 20 to perform a predetermined operation with respect to the target position.

The terminal device 10 shown in FIG. 1 may be, for example, a mobile terminal such as a smartphone or a tablet, a stationary PC (Personal Computer), or a laptop PC. Furthermore, it may be a wearable terminal such as an HMD (Head Mount Display) or a wristwatch type terminal.

The terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage 16, and a processor 19.

The communication IF 12 is an interface for inputting and outputting signals so that the terminal device 10 communicates with a device in the system S, such as the instruction target device 20, for example.

The input device 13 is a device (for example, a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.) for receiving input operations from a user.

The output device 14 is a device (display, speaker, etc.) for presenting information to the user.

The memory 15 is for temporarily storing programs and data processed by the programs, and is a volatile memory such as DRAM (Dynamic Random Access Memory).

The storage 16 is for storing data, and is, for example, a flash memory or an HDD (Hard Disc Drive).

The processor 19 is hardware for executing a set of instructions written in a program, and is composed of an arithmetic unit, registers, peripheral circuits, and the like.

The instruction target device (mobile object) 20 is a device that is instructed by the owner of the terminal device 10 to perform some action toward a target position provided by the terminal device 10 . An example of the instruction target device 20 is the automobile shown in FIG. In the case of a car, the target position provided from the terminal device 10 is, for example, a parking target position of the car. The specific content of the target position provided by the terminal device 10 differs depending on the type of the pointing device 20, and how the pointing device 20 moves with respect to the target position depending on the type of the pointing device 20. They also differ in how they perform their actions.

The instruction target device 20 has a function of performing some kind of operation toward a target position provided from an external device including the terminal device 10. If the instruction target device 20 is a car, it has an automatic parking function as shown in the example of FIG. Further, if the instruction target device 20 is a cleaning robot, it has a function of recognizing a room presented as a target position, or furthermore, a specific area of the room, and performing a cleaning operation by self-propelling in this room.

The instruction target device 20 is not particularly limited, including the function of the instruction target device 20 itself and the size of the device, as long as it has the above-mentioned functions.

<1.1 Functional configuration of terminal device>
FIG. 3 is a block diagram showing an example of the functional configuration of the terminal device 10 shown in FIG. 2. As shown in FIG. The terminal device 10 shown in FIG. 3 is realized by, for example, a PC, a mobile terminal, or a wearable terminal. As shown in FIG. 3, the terminal device 10 includes a communication section 120, an input device 13, an output device 14, an audio processing section 17, a microphone 171, a speaker 172, a camera 160, and a position information sensor 150. , an acceleration sensor 155, a three-dimensional scanner 165, a storage section 180, and a control section 190. Each block included in the terminal device 10 is electrically connected by, for example, a bus or the like.

The communication unit 120 performs processing such as modulation/demodulation processing for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on the signal generated by the control unit 190, and transmits the signal to the outside (for example, the instruction target device 20). The communication unit 120 performs reception processing on a signal received from the outside, and outputs the signal to the control unit 190.

The input device 13 is a device for a user operating the terminal device 10 to input instructions or information. The input device 13 may be realized by, for example, a keyboard, a mouse, a reader, or the like. When the terminal device 10 is a mobile terminal or the like, it is realized by a touch-sensitive device 131 or the like in which an instruction is input by touching an operation surface. The input device 13 converts an instruction input from a user into an electrical signal, and outputs the electrical signal to the control unit 190. Note that the input device 13 may include, for example, a reception port that receives electrical signals input from an external input device.

The output device 14 is a device for presenting information to a user operating the terminal device 10. The output device 14 is realized by, for example, a display 141 or the like. The display 141 displays data according to the control of the control unit 190. The display 141 is realized by, for example, an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display.

The audio processing unit 17 performs, for example, digital-to-analog conversion processing of audio signals. The audio processing unit 17 converts the signal provided from the microphone 171 into a digital signal, and provides the converted signal to the control unit 190. Furthermore, the audio processing section 17 provides an audio signal to the speaker 172. The audio processing unit 17 is realized, for example, by a processor for audio processing. The microphone 171 receives a voice input and provides a voice signal corresponding to the voice input to the voice processing unit 17. The speaker 172 converts the audio signal provided from the audio processing unit 17 into audio and outputs the audio to the outside of the terminal device 10 .

The camera 160 is a device that receives light with a light receiving element and outputs image data as a photographing signal. The camera 160 images an object (subject) in a certain direction and within a certain imaging range with respect to the terminal device 10, and outputs image data as a result of imaging this object. At this time, if the camera 160 has a function that allows adjustment of the imaging range, more precisely, the angle of view, the camera 160 also outputs information regarding this angle of view. Such a function is called a zoom function. Furthermore, it is assumed that the camera 160 is fixed to the housing of the terminal device 10, as an example, so that the imaging direction of the camera 160 with respect to the terminal device 10 is uniquely determined.

The position information sensor 150 is a sensor that detects the position of the terminal device 10, and is generally a GNSS device, such as a GPS module. A GPS module is a receiving device used in a satellite positioning system. The satellite positioning system receives signals from at least three or four satellites, and detects the current position of the terminal device 10 equipped with a GPS module based on the received signals. The position information sensor 150 may detect the current position of the terminal device 10 from the position of a wireless base station to which the terminal device 10 connects via the communication unit 120.

The acceleration sensor 155 is a sensor that detects acceleration applied to the terminal device 10. Preferably, the acceleration sensor 155 has a function of detecting inclination around each axis (X-axis, Y-axis, Z-axis) of a three-dimensional coordinate whose origin is the position of the terminal device 10. The acceleration sensor 155 having such a function can detect the attitude of the terminal device 10, that is, the direction with respect to the X-axis, Y-axis, and Z-axis, by detecting the gravitational acceleration of the universal gravitational force with respect to the earth. Since the imaging direction of the camera 160 with respect to the terminal device 10 is uniquely determined, the imaging direction of the camera 60 in the three-dimensional space can be determined based on the detection result of the acceleration sensor 155.

The 3D scanner 165, which is an example of a 3D position acquisition device, scans an object in a certain direction and within a certain detection range with respect to the terminal device 10, acquires 3D position data on the surface of this object, and Output 3D position data. LiDAR is known as an example of such a three-dimensional scanner 165. LiDAR irradiates (scans) a wide range of laser beams onto a measurement target, receives reflected light, and acquires three-dimensional position data on the surface of the measurement target based on the presence or absence of this reflected light and the timing of reception of the reflected light. It is something to do. The three-dimensional position data acquired by LiDAR is three-dimensional point group data that is the reflection position of the laser beam. The acquisition range (measurement range) of three-dimensional position data by the three-dimensional scanner 165 is set to include at least the imaging range by the camera 160. As described above, when the camera 160 has a zoom function, three-dimensional position data can be obtained in a measurement range that includes at least the imaging range when the angle of view of the camera 160 is at its maximum (that is, closer to the wide-angle side). It is said that Alternatively, if the measurement range of the three-dimensional scanner 165 is variable, the measurement range of the three-dimensional scanner 165 may be set to match the imaging range of the camera 160.

The storage unit 180 is realized by, for example, the memory 15, the storage 16, etc., and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, an application program 181, image data 182, scan data 183, image DB (DataBase), and pointing point DB 185.

Image data 182 is data obtained by capturing an image of a subject with camera 160. The scan data 183 is data obtained by scanning a three-dimensional space by the three-dimensional scanner 165. The image DB 184 is a DB for managing information that is linked to each image data 182 when the image data 182 is captured by the camera 160. The designated point DB 185 is a DB for managing information linked to a specific location on the surface of an object in three-dimensional space designated by the owner of the terminal device 10. Details of the image DB 184 and the designated point DB 185 will be described later.

The control unit 190 is realized by the processor 19 reading an application program 181 stored in the storage unit 180 and executing instructions included in the application program 181. The control unit 190 controls the operation of the terminal device 10. The control unit 190 operates according to the program to control the operation reception unit 191, the transmission/reception unit 192, the presentation control unit 193, the imaging control unit 194, the scanner control unit 195, the scan data association unit 196, and the specification. It functions as a three-dimensional position acquisition unit 197.

The operation reception unit 191 performs processing for accepting instructions or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives information based on instructions input from a keyboard, mouse, or the like.

Further, the operation reception unit 191 receives voice instructions input from the microphone 171. Specifically, for example, the operation reception unit 191 receives an audio signal input from the microphone 171 and converted into a digital signal by the audio processing unit 17. The operation reception unit 191 obtains an instruction from the user by, for example, analyzing the received audio signal and extracting a predetermined noun.

The transmitting/receiving unit 192 performs processing for the terminal device 10 to transmit and receive data with an external device such as the instruction target device 20 according to a communication protocol. Specifically, for example, the transmitter/receiver 192 transmits the content input by the user to the external device. Further, the transmitting/receiving unit 192 receives information regarding the user from an external device.

The presentation control unit 193 controls the output device 14 in order to present information generated by each functional unit of the terminal device 10 to the user. Specifically, for example, the presentation control unit 193 causes the display 141 to display information generated by each functional unit of the terminal device 10. Furthermore, the presentation control unit 193 causes the speaker 172 to output information generated by each functional unit of the terminal device 10 .

The imaging control unit 194 performs general control of the camera 160. In particular, the imaging control unit 194 operates the camera 160 to image a subject with the camera 160 while the system S of the present embodiment is in operation (while the application program 181 is running), and thereby causes the camera 160 to constantly output an imaging signal. An image is displayed on the screen of the display 141 of the output device 14 based on the captured image signal.

Then, the imaging control unit 194 stores the image data 182 output from the camera 160 in the storage unit 180 based on an instruction from the owner of the terminal device 10 or the like. The imaging control unit 194 also acquires the measured values of the position information sensor 150 and the acceleration sensor 155 when the camera 160 acquires the image data 182, and stores the acquired measured values and the file name of the image data 182 in the image DB 184. do. Thereafter, the imaging control unit 194 causes the acquired image data 182 to be displayed on the display 141 of the output device 14. The imaging control unit 194 may store the measurement value as metadata of the image data 182.

The scanner control unit 195 performs general control of the three-dimensional scanner 165. In particular, the scanner control unit 195 operates the three-dimensional scanner 165 to constantly output three-dimensional position data while the system S of this embodiment is operating (while the application program 181 is being executed). Then, the scanner control unit 195 transmits the three-dimensional position data acquired by the three-dimensional scanner 165 at the timing when the imaging control unit 194 acquires the image data 182 from the camera 160 based on an instruction from the owner of the terminal device 10 or the like. It is stored in the scan data 183. At this time, the scanner control unit 195 checks whether the acquisition range of the three-dimensional position data by the three-dimensional scanner 165 is a range that includes at least the imaging range by the camera 160, and if necessary, Set and adjust the acquisition range of 3D position data.

The scan data association unit 196 associates the image data 182 and the scan data 183 based on the image data 182 and scan data (three-dimensional position data) 183 acquired by the imaging control unit 194 and the scanner control unit 195. .

Specifically, the scan data association unit 196 refers to the image DB 184 and the scan data 183, and associates each point of the three-dimensional point group data that makes up the three-dimensional position data with the pixels that make up the image data 182. Match. Although there is no particular limitation on the method of associating the image data 182 with the three-dimensional position data, one example is a method of obtaining a conversion formula for projecting the three-dimensional position data onto the image data 182, which is two-dimensional data. It will be done.

That is, the acquisition range of 3D position data by the 3D scanner 165 includes at least the imaging range of the camera 160, and the 3D position data is acquired by the 3D scanner 165 in response to the operation of acquiring image data 182 from the camera 160. Since the movement is being performed, it is almost certain that the image data 182 and the three-dimensional position data are data regarding the same object. Therefore, the scan data correspondence unit 196 projects the three-dimensional position data onto the image data 182, which is two-dimensional plane data, and obtains a conversion formula at this time, thereby associating the three-dimensional position data with the image data 182. I do. Then, the scan data association unit 196 temporarily stores the obtained conversion formula in the storage unit 180.

Alternatively, if the imaged object has a simple shape and the number of objects included in the image data 182 is limited, the scan data association unit 196 may apply this image to the image data 182. The image data 182 and the three-dimensional position data may be correlated by analyzing what shape the object is included in the data 182 and correlating the surface shape of the object with the three-dimensional position data. . As an example, in artificial intelligence technology, particularly deep learning technology, an object detection (object recognition) technology that detects an object (subject) present in image data is known. Well-known object detection technologies include, for example, YOLO (You Only Look Once), R-CNN (Region Based Convolutional Neural Networks), and their advanced forms such as Fast R-CNN, Faster R-CNN, and SSD (Single Shot MultiBox). Detector), etc. When using object detection technology, the subject (object) must be known, but if it is a common object (for example, car 3 shown in Figure 1), object detection technology can be used to detect the object. Since the position and approximate shape can be grasped, the object in the image data 182 can be detected and correlated with three-dimensional position data.

The designated position three-dimensional position acquisition unit 197 accepts the designation of specific data (specific pixels) of the image data 182 when the owner of the terminal device 10 touches the screen of the display 141 , and thereby allows the camera 160 to Accepts the designation of a specific location on the surface of the imaged object. Then, the specified position three-dimensional position acquisition unit 197 acquires the specified position on the surface of the received object, based on the position information of the terminal device 10 linked to the image data 182, the imaging direction of the camera 160, and the three-dimensional position data. Then, the three-dimensional position data of this specific location is obtained.

Specifically, since the imaging control unit 194 displays an image based on the imaging signal output from the camera 160 on the display 141, the designated three-dimensional position acquisition unit 197 displays the image data 182 on the display 141. In this state, the terminal device 10 waits for the owner of the terminal device 10 to touch the screen. A touch on the screen by the owner of the terminal device 10 is detected by the touch sensitive device 131 of the input device 13 . The designated position three-dimensional position acquisition unit 197 receives a touch on the screen by the owner of the terminal device 10 and detects and calculates the position of the screen touched by the owner based on the detection signal from the touch sensitive device 131. Then, by calculating which pixel of the image data 182 the detected touch position corresponds to, it is detected which position of the image data 182 the detected touch position corresponds to. The designated three-dimensional position acquisition unit 197 stores the detection result in the designated point DB 185.

Next, the specified position three-dimensional position acquisition unit 197 scans to determine which point of the three-dimensional position data (three-dimensional point group data) the position of the image data 182 corresponding to the touch position corresponds to. The identification is made based on the association result of the data association unit 196. Thereby, the specified position three-dimensional position acquisition unit 197 detects which position on the surface of the object in the three-dimensional space (that is, the position of a specific part) the position touched by the owner of the terminal device 10 is, Based on this detection result, three-dimensional position data (three-dimensional position data as a point) of a specific location on the surface of the object can be specified.

Furthermore, the designated three-dimensional position acquisition unit 197 refers to the image DB 184 to obtain the position information of the terminal device 10 and the imaging direction of the camera 160 when the image data 182 was acquired. Details of the image DB 184 will be described later, but the image DB 184 stores the detection results of the position information sensor 150 and the acceleration sensor 155 when the image data 182 is captured, linked to the image data 182. ing. The detection result of the position information sensor 150 is the position information of the terminal device 10, and the detection result of the acceleration sensor 155 is the attitude of the terminal device 10 at the time when the image data 182 was acquired (3 with the position of the terminal device 10 as the origin). The inclinations of the dimensional coordinates around the X, Y, and Z axes can be calculated. Since the imaging direction of the camera 160 with respect to the terminal device 10 is known, the imaging direction of the camera 160 can be calculated from the detection result of the acceleration sensor 155.

Then, the specified position three-dimensional position acquisition unit 197 acquires three-dimensional position data of the specific place based on the position information of the terminal device 10, the imaging direction of the camera 160, and the three-dimensional position data of the specific place on the surface of the object. Absolute position (latitude, longitude, altitude) can be determined. Thereafter, the control unit 190 of the terminal device 10 provides the absolute position of the three-dimensional position data of the specific location obtained by the specified position three-dimensional position acquisition unit 197 to the pointing target device 20, and It is possible to instruct various operations using the absolute position as the target position. As an example of such an instruction, if the instruction target device 20 is a car, as in the example shown in FIG. 1, an instruction to set the absolute position as the parking target position and park the car at this target position can be given. .

<2 Data structure>
4 to 5 are diagrams showing the data structure of a database stored in the terminal device 10. Note that FIGS. 4 and 5 are examples, and do not exclude data not described.

FIG. 4 is a diagram showing the data structure of the image DB 184. As shown in FIG. 4, each record of the image DB 184 includes, for example, an item "image ID", an item "image file", an item "imaging point information", and an item "imaging direction information". Information stored in the image DB 184 is input and updated by the imaging control unit 194. The information stored in the image DB 184 can be changed and updated as appropriate.

The item “image ID” is an ID for identifying the image data 182. The item “image file” is information regarding the file name of each image data 182 stored in the storage unit 180. The item "imaging point information" is information regarding the position of the terminal device 10 when the image data 182 specified by the image ID was captured. The item "imaging direction information" is information regarding the imaging direction of the camera 160 when the image data 182 specified by the image ID was captured.

The item "imaging point information" includes the item "latitude information", the item "longitude information", and the item "altitude information". The item "latitude information" is information about the latitude of the terminal device 10 when the image data 182 specified by the image ID was captured, and the item "longitude information" is information about the latitude of the terminal device 10 when the image data 182 specified by the image ID was captured. The item "altitude information" is information regarding the altitude of the terminal device 10 when the image data 182 specified by the image ID was captured.

The item "imaging direction information" includes the item "tilt around the X-axis", the item "tilt around the Y-axis", and the item "tilt around the Z-axis". The item "Tilt around the X-axis" is information about the tilt of the terminal device 10 around the X-axis when the image data 182 specified by the image ID is captured, and the item "Tilt around the Y-axis" is information about the tilt of the terminal device 10 around the X-axis when the image data 182 specified by the image ID is captured. This is information regarding the tilt of the terminal device 10 around the Y-axis when the image data 182 specified by the ID is captured, and the item "Tilt around the Z-axis" is information about the tilt of the terminal device 10 when the image data 182 specified by the image ID is captured. This is information regarding the inclination of the terminal device 10 around the Z-axis when the terminal device 10 is rotated.

FIG. 5 is a diagram showing the data structure of the pointing point DB 185. As shown in FIG. 5, each record of the pointing point DB 185 includes, for example, the item "pointing point ID", the item "screen pointing point", and the item "scanner pointing point". The information stored in the designated point DB 185 is input and modified by the specified position three-dimensional position acquisition unit 197. The information stored in the pointing point DB 185 can be changed and updated as appropriate.

The item "indication point ID" is specified when the owner of the terminal device 10 touches the screen to specify the position (pixel) of a specific part of the image data 182. This is an ID for specifying a point. The item “Screen indication point” is information regarding the coordinate value on the screen (on the image data 182) of the point designated by the owner of the terminal device 10, and the item “Scanner indication point” is information regarding the coordinate value of the point specified by the owner of the terminal device 10. This is information regarding the coordinate values of the three-dimensional position data (obtained by the three-dimensional scanner 165) of the point specified by the owner.

The item "Screen indication point" includes the item "X coordinate point" and the item "Y coordinate point." The item "X coordinate point" is information regarding the X coordinate value on the screen of the point specified by the owner of the terminal device 10, and the item "Y coordinate point" is information regarding the X coordinate value of the point specified by the owner of the terminal device 10. This is information regarding the Y coordinate value of the point on the screen.

The item "scanner pointing point" includes the item "X coordinate point", the item "Y coordinate point", and the item "Z coordinate point". The item "X coordinate point" is information regarding the X coordinate value of the three-dimensional position data corresponding to the point specified by the owner of the terminal device 10, and the item "Y coordinate point" is information about the X coordinate value of the three-dimensional position data corresponding to the point specified by the owner of the terminal device 10 The item "Z coordinate point" is information about the Y coordinate value of the three-dimensional position data corresponding to the point specified by the owner of the terminal device 10. This is information regarding the Z coordinate value.

<3 Operation example>
An example of the operation of the terminal device 10 will be described below.

FIG. 6 is a flowchart illustrating an example of the operation of the terminal device 10.

In step S600, the control unit 190 operates the camera 160 to display imaging information regarding the range that can be imaged by the camera 160, that is, the subject being imaged at that time by the image sensor of the camera 160. Specifically, for example, the control unit 190 receives an imaging signal output from the imaging device of the camera 160 by the imaging control unit 194, and causes the display 141 to display an image based on this imaging signal.

FIG. 8 is a diagram showing an example of an image displayed on the display 141 of the terminal device 10 in step S600. On the screen 800 of the terminal device 10, an image 801 based on an image signal captured by the image sensor of the camera 160 at that time is displayed. The screen 800 also displays an instruction to the owner of the terminal device 10 to perform a predetermined operation (in this case, an instruction to image an area including a target point for instructing the instruction target device 20 to perform an operation). A display area 802 is provided.

In step S601, the control unit 190 operates the three-dimensional scanner 165 to scan a measurement range that includes at least the imaging range of the camera 160. Specifically, the control unit 190 causes the scanner control unit 195 to operate the three-dimensional scanner 165 to receive reflected light from the surface of an object within the measurement range, thereby scanning the real space including the object. conduct.

Next, in step S602, the control unit 190 instructs the owner of the terminal device 10 to input an instruction to designate a specific location on the surface of the object that will be the target position by touching the screen of the display 141 of the terminal device 10. Specifically, the control unit 190 causes the designated position three-dimensional position acquisition unit 197 to display on the display 141 a display instructing the owner of the terminal device 10 to input a designation of a specific location on the surface of the object. Then, in step S603, the control unit 190 waits for the owner of the terminal device 10 to touch the screen, and if it is detected that the owner has touched the screen (YES in step S603), the process proceeds to step S604.

FIG. 9 is a diagram showing an example of the screen displayed on the display 141 of the terminal device 10 in step S602. On the screen 900 of the terminal device 10, an image 901 currently captured by the camera 160 is displayed, and an instruction to perform a predetermined operation is given to the owner of the terminal device 10 (in this case, the terminal An area 902 is provided in which an instruction (instructing the user to touch the screen of the device 10 to specify a specific location on the surface of an object as a target position) is displayed.

Then, as shown in FIG. 10, the owner of the terminal device 10 confirms that the target point for instructing the instruction target device 20 is on the screen 1000, and then displays the screen on the display 141 of the terminal device 10. When the user touches , and inputs the designation of a specific location on the surface of the object that will be the target position (input of designation of the specific location 1003 in FIG. 10), the program moves to step S604.

In step S604, the control unit 190 acquires the image data 182 output from the camera 160, stores the acquired image data 182 in the storage unit 180, and also controls the position information sensor 150 and the Necessary information is recorded in the image DB 184 based on the detection result of the acceleration sensor 155 and the image data 182.

Specifically, for example, the control unit 190 receives image data 182 from the camera 160 using the imaging control unit 194, and stores the accepted image data 182 in the storage unit 180. Next, the control unit 190 receives the detection results of the position information sensor 150 and the acceleration sensor 155 at the time of acquiring the image data 182 by the imaging control unit 194, and receives the received detection results of the position information sensor 150 and the acceleration sensor 155, Then, necessary information is written into the image DB 184 based on the image data 182 stored in the storage unit 180.

There is no particular restriction on the time interval between the timing of acquisition of image data 182 by camera 160 and the timing of acquisition of detection results of position information sensor 150 and acceleration sensor 155, but after acquisition of image data 182, terminal device 10 Since movement may affect the accuracy of identifying the three-dimensional position data of a specific location on the surface of the object, it is preferable to acquire the detection results of the position information sensor 150 and the acceleration sensor 155 immediately after acquiring the image data 182. .

In step S606, the control unit 190 acquires three-dimensional position data of the surface of an object in real space within a predetermined measurement range from the terminal device 10. Specifically, in step S606, the control unit 190 uses the scanner control unit 195 to acquire three-dimensional position data of the surface of the object in the real space within a predetermined measurement range from the three-dimensional scanner 165, and uses the acquired three-dimensional The position data is stored in the storage unit 180 as scan data 183.

7, in step S700, the control unit 190 causes the scan data association unit 196 to associate the image data 182 obtained in step S604 with the scan data 183 obtained in step S606. The details of the data association operation by the scan data association unit 196 have already been explained, so the explanation will be omitted here.

In step S701, the control unit 190 acquires three-dimensional position data of a specific location on the surface of the object where the designation input was made in step S603. Specifically, the control unit 190 uses the specified position three-dimensional position acquisition unit 197 to calculate the coordinate position of the image data 182 where the specified input has been made and the three-dimensional position data corresponding to this coordinate position. The calculated coordinate position and three-dimensional position data are stored in the designated point DB 185 by the specified position three-dimensional position acquisition unit 197. Next, the specified position three-dimensional position acquisition unit 197 obtains and calculates the position information of the terminal device 10 linked to the image data 182 and the imaging direction of the camera 160, and calculates the three-dimensional position data of the specific point on the surface of the object. Calculate absolute position.

Thereafter, in step S702, the control unit 190 instructs the instruction target device 20 to operate toward the absolute position of the three-dimensional position data of the specific location on the surface of the object acquired in step S702.

<4 Effects of embodiment>
As described above in detail, according to the system S of the present embodiment, based on the designation input of a specific location on the surface of an object from the owner of the terminal device 10, the absolute position of the three-dimensional position data of this specific location is determined. can be obtained. In particular, according to the system S of this embodiment, acquisition of image data 182 and acquisition of scan data 183 are performed in a series of operations without acquiring three-dimensional coordinate data in advance using a three-dimensional position acquisition device such as LiDAR. It is possible to instruct the instruction target device 20 to move to the target position without any special preparation.

<5 Additional notes>
Note that the configurations of the embodiments described above are explained in detail in order to explain the present disclosure in an easy-to-understand manner, and the embodiments are not necessarily limited to those having all of the configurations described. Furthermore, it is possible to add, delete, or replace a part of the configuration of each embodiment with other configurations.

As an example, in the system S of the embodiment described above, all operations from acquiring image data 182, acquiring scan data 183, and acquiring three-dimensional position data of a specific location on the surface of an object are performed by the terminal device 10. The scan data association unit 196 and designated position three-dimensional position acquisition unit 197 in FIG. 3 may be provided in an information processing device separate from the terminal device 10. That is, the hardware configuration of the system S is not limited to the configuration shown in FIG. 2.

In addition, in the system S of the embodiment described above, an automobile is taken as an example of the instruction target device 20, and an instruction to park the automobile at a target position is given. The instructions are not limited to this.

As an example, in a medical field, by touching the screen of the terminal device 10, the outline of a treatment target, for example, a region to be resected, is traced, and a (remote) medical robot is used as the instruction target device 20, and this medical robot is used to specify the area. It is also possible to remove the Currently, only doctors can perform medical procedures due to restrictions under the Medical Practitioners Act, but with the assistance of a doctor, it will be possible for medical robots to perform ablation procedures (medical procedures).

Alternatively, at a construction site, by touching the screen of the terminal device 10, the area where civil engineering work is to be performed, such as excavating the slope of the land, is instructed, and the excavator car is used to excavate the specified area by using the excavator as the device 20 to be instructed. It is also possible to instruct.

Furthermore, a flying object such as a drone can also be suitably used as the instruction target device 20. When a drone is selected as the instruction target device 20, the system S can instruct the flight path and target position of this drone.

Furthermore, in the system S of the above-described embodiment, as shown in steps S600 and S601, while the system S is in operation, the camera 160 constantly performs the imaging operation and the 3D scanner 165 performs the 3D position data acquisition operation. It is also possible that the three-dimensional scanner 165 performs a three-dimensional position data acquisition operation in accordance with at least the designation of a specific location by the owner of the terminal device 10, and does not operate the three-dimensional scanner 165 at other times.

Further, the operation of associating the image data 182 and the scan data 183 in step S700 may be performed all the time when the three-dimensional scanner 165 is constantly performing the three-dimensional position data acquisition operation.

Further, each of the above-mentioned configurations, functions, processing units, processing means, etc. may be partially or entirely realized in hardware by designing, for example, an integrated circuit. Further, the present invention can also be realized by software program codes that realize the functions of the embodiments. In this case, a storage medium on which a program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored on the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the embodiments described above, and the program code itself and the storage medium storing it constitute the present invention. Storage media for supplying such program codes include, for example, flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, and non-volatile memory cards. , ROM, etc. are used.

Furthermore, the program code that implements the functions described in this embodiment can be implemented using a wide range of program or script languages, such as assembler, C/C++, Perl, Shell, PHP, and Java (registered trademark).

Furthermore, by distributing the software program code that realizes the functions of the embodiment via a network, it can be stored in a storage means such as a computer's hard disk or memory, or a storage medium such as a CD-RW or CD-R. Alternatively, a processor included in the computer may read and execute the program code stored in the storage means or the storage medium.

The matters explained in each of the above embodiments are additionally described below.

(Additional note 1)
A program for operating a computer (10) including a processor (19), a memory (15), an imaging device (160), and a three-dimensional position acquisition device (165), the program causing the processor (19) to: a step (S604) of acquiring image data (182) obtained by imaging the real space with the imaging device (160); and a step (S604) of acquiring image data (182) obtained by imaging the real space with the imaging device (160); A step (S605) of acquiring the position data (183) by the three-dimensional position acquisition device (165), a step (S600) of displaying the image data (182) on the display device (141), and the step of displaying the image data (182) and 3 a step of associating the dimensional position data (183) (S700); and a step of accepting the designation of a specific location on the surface of the object by accepting the designation of the specific location of the image data (182) displayed on the display device (141). (S603), and 3D position data (183) of the specific location based on the position information of the imaging device (160), the imaging direction of the imaging device (160), the designation of the specific location, and the 3D location data (183). A program that executes the step of acquiring (S701).
(Additional note 2)
The program according to supplementary note 1 further causes the program to execute a step (S702) of instructing a moving body to move toward three-dimensional position data (183) of a specific location.
(Additional note 3)
In the step of accepting the designation of a specific location (S603), the designation of a plurality of specific locations is accepted, and in the step of acquiring the three-dimensional position data (183) of the specific location (S701), the three-dimensional location data (183) of the plurality of specific locations is accepted. ) and instructing to move the moving object (S702), instructing the moving object to move so as not to straddle the line specified by the three-dimensional position data (183) of a plurality of specific locations. The program described in Appendix 2.
(Additional note 4)
In the step of accepting the designation of a specific location (S603), the designation of a plurality of specific locations is accepted, and in the step of acquiring the three-dimensional position data (183) of the specific location (S701), the three-dimensional location data (183) of the plurality of specific locations is accepted. 183) and instructing the moving body to move (S702), the moving body is instructed to perform predetermined processing on the surface specified by the three-dimensional position data (183) of a plurality of specific locations. The program described in Supplementary Note 2 that provides instructions to operate.
(Appendix 5)
An information processing device (10) equipped with a processor (19), a memory (15), an imaging device (160), and a three-dimensional position acquisition device (165), the processor (19) is configured to move the real space to the imaging device (165). 160), and three-dimensional position data (183) of the surface of an object existing in the real space, including at least the imaging range of the imaging device (160). A step (S605) of acquiring by the three-dimensional position acquisition device (165), a step (S600) of displaying the image data (182) on the display device (141), and a step (S600) of displaying the image data (182) and the three-dimensional position data (183). (S700); a step (S603) of accepting the designation of a specific location on the surface of the object by accepting the designation of the specific location of the image data (182) displayed on the display device (141); A step (S701) of acquiring three-dimensional position data (183) of a specific place based on the position information of the device (160), the imaging direction of the imaging device (160), the designation of the specific place, and the three-dimensional position data (183). ).
(Appendix 6)
A method executed by a computer (10) including a processor (19), a memory (15), an imaging device (160), and a three-dimensional position acquisition device (165), the processor (19) A step (S604) of acquiring image data (182) obtained by imaging with the imaging device (160), and three-dimensional position data ( 183) by the three-dimensional position acquisition device (165) (S605), a step (S600) of displaying the image data (182) on the display device (141), and a step (S600) of displaying the image data (182) and the three-dimensional position data. (183) (S700), and a step (S603) of accepting the designation of a specific location on the surface of the object by accepting the designation of the specific location of the image data (182) displayed on the display device (141). Then, 3D position data (183) of the specific location is acquired based on the location information of the imaging device (160), the imaging direction of the imaging device (160), the designation of the specific location, and the 3D location data (183). A method of performing step (S701).
(Appendix 7)
A means (194) for acquiring image data (182) obtained by imaging a real space with an imaging device (160), and a three-dimensional surface of an object existing in the real space that includes at least the imaging range of the imaging device (160). means (195) for acquiring position data (183) by a three-dimensional position acquisition device (165); means (197) for displaying image data (182) on a display device (141); and image data (182). means (196) for associating the dimensional position data (183) with the image data (182) displayed on the display device (141); (197), the 3D position data (183) of the specific location based on the position information of the imaging device (160), the imaging direction of the imaging device (160), the designation of the specific location, and the 3D location data (183). A system (1) comprising means (197) for acquiring.

S, system 10, terminal device 12, communication IF 13, input device 14, output device 15, memory 16, storage 17, audio processing section 19, processor 20, instruction target device 60, camera 80, network 120, communication section 131, Touch sensitive device 141, display 150, position information sensor 155, acceleration sensor 160, camera 165, three-dimensional scanner 180, storage unit 181, application program 182, image data 183, scan data 184, image DB 185, pointing point DB 190, Control unit 191, Operation reception unit 192, Transmission/reception unit 193, Presentation control unit 194, Imaging control unit 195, Scanner control unit 196, Scan data association unit 197, Specified position three-dimensional position acquisition unit

Claims (7)

A program for operating a computer including a processor, a memory, an imaging device, and a three-dimensional position acquisition device,
The program causes the processor to:
acquiring image data obtained by imaging a real space with the imaging device;
acquiring, by the three-dimensional position acquisition device, three-dimensional position data of a surface of an object existing in the real space that includes at least an imaging range of the imaging device;
Displaying the image data on a display device;
associating the image data with the three-dimensional position data;
accepting the designation of a specific location on the surface of the object by accepting the designation of the specific location of the image data displayed on the display device;
A program that causes a program to execute a step of acquiring the three-dimensional position data of the specific location based on position information of the imaging device, an imaging direction of the imaging device, designation of the specific location, and the three-dimensional location data. The program further includes:
2. The program according to claim 1, wherein the program executes the step of instructing a moving body to move toward the three-dimensional position data of the specific location. In the step of accepting the designation of the specific location, accepting the designation of a plurality of the specific locations;
In the step of acquiring the three-dimensional position data of the specific location, acquiring the three-dimensional position data of a plurality of the specific locations;
3. The program according to claim 2, wherein in the step of instructing to move the moving object, an instruction is given to move the moving object so as not to straddle a line specified by the three-dimensional position data of the plurality of specific locations. . In the step of accepting the designation of the specific location, accepting the designation of a plurality of the specific locations;
In the step of acquiring the three-dimensional position data of the specific location, acquiring the three-dimensional position data of a plurality of the specific locations;
2. The step of instructing to move the movable body includes instructing the movable body to perform a predetermined process on a surface specified by the three-dimensional position data of the plurality of specific locations. The program described in. An information processing device including a processor, a memory, an imaging device, and a three-dimensional position acquisition device,
The processor includes:
acquiring image data obtained by imaging a real space with the imaging device;
acquiring, by the three-dimensional position acquisition device, three-dimensional position data of a surface of an object existing in the real space that includes at least an imaging range of the imaging device;
Displaying the image data on a display device;
associating the image data with the three-dimensional position data;
accepting the designation of a specific location on the surface of the object by accepting the designation of the specific location of the image data displayed on the display device;
Information processing that executes the step of acquiring the three-dimensional position data of the specific location based on the position information of the imaging device, the imaging direction of the imaging device, the designation of the specific location, and the three-dimensional location data. Device. A method performed by a computer comprising a processor, a memory, an imaging device, and a three-dimensional position acquisition device, the method comprising:
The processor includes:
acquiring image data obtained by imaging a real space with the imaging device;
acquiring, by the three-dimensional position acquisition device, three-dimensional position data of a surface of an object existing in the real space that includes at least an imaging range of the imaging device;
Displaying the image data on a display device;
associating the image data with the three-dimensional position data;
accepting the designation of a specific location on the surface of the object by accepting the designation of the specific location of the image data displayed on the display device;
A method, comprising: acquiring the three-dimensional position data of the specific location based on the position information of the imaging device, the imaging direction of the imaging device, the designation of the specific location, and the three-dimensional location data. means for acquiring image data obtained by imaging a real space with an imaging device;
means for acquiring, by a three-dimensional position acquisition device, three-dimensional position data of a surface of an object existing in the real space that includes at least an imaging range of the imaging device;
means for displaying the image data on a display device;
means for associating the image data and the three-dimensional position data;
means for accepting the designation of a specific location on the surface of the object by accepting the designation of the specific location of the image data displayed on the display device;
A system comprising means for acquiring the three-dimensional position data of the specific location based on position information of the imaging device, an imaging direction of the imaging device, designation of the specific location, and the three-dimensional location data.