JP2006308493A - Portable terminal device, response communications device, and display system and method of retrieved object - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a system that is concurrently equipped with a position measuring device of a retrieved object and a photographic device and can overlap-display an image photographed on a display and a pointer indicating the retrieved object, its method and the like. <P>SOLUTION: The portable terminal device comprises the display, a photographing means for photographing a video projected on the display, a retrieved object position measuring means for measuring the arrival direction and distance of a response signal from the retrieved object, and a display control means that displays the video photographed by the photographic means on the display, determines whether the retrieved object is within the range of the video displayed on the display, and, when it is determined to be within the range, overlaps the video and the pointer indicating the retrieved object and displays them on the display. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a mobile terminal device capable of searching for a search target, a response communication device that can be attached to the search target, and a search target display system and method, and in particular, an image of an optical camera and a search on a display of the mobile terminal device. It relates to displaying the target pointer in an overlapping manner.

  Due to the development of wireless technology and the miniaturization technology of electronic devices, it has become possible to attach a communication device equipped with a sensor and a calculation capability to a small article that has been thought to be difficult to mount. For example, by attaching a minute RFID tag to each product, it is possible to carry out detailed tracking and distribution tracking surveys.

  If such usage further progresses, there is a possibility that a minute communication device will be attached to every article and there will be several hundred communication devices per square meter. In such a case, it is expected that the problem of how a human discovers, identifies, and manages an article (search target) with such a communication device will become serious.

  The following techniques exist as conventional techniques applicable to such a problem, but all have disadvantages.

  Since the passive RFID technology has a very short communication distance, it is not suitable for an application of specifying the location of an article around us. Furthermore, even with the existing active RFID technology that can increase the communication distance, the position where the tag exists cannot be specified at present. Furthermore, when there are a plurality of tags in the communicable area, there is not provided a method in which a human can intuitively recognize at which position the tag is currently communicating.

  On the other hand, radar communication devices are used for applications that recognize distant objects, but widely used radar technology measures the reflected waves of electromagnetic waves transmitted at high power to distant objects as a primary radar. It is used for the purpose and is not suitable for searching for small objects at a short distance such as indoors.

  In recent years, Micro-power Impulse Radar technology and UWB radar technology have been developed, and miniaturization of radar technology at a short distance is being promoted.

  However, since these techniques are primary radars that measure a reflected wave from an object, information relating to the ID of the object cannot be obtained. Therefore, the above problem cannot be solved in terms of identification and recognition.

  Similarly, there are short-range radar technologies such as fish detectors, landmine detectors, and anti-collision vehicle-mounted radars, but these are also primary radars, so the relative position of the object can be detected, but information on the ID of the object is obtained. The above problem is not solved because it cannot be done.

  In addition, there is a secondary radar device for aircraft in that the searching side outputs a signal, but it is not considered for use at a short distance, and it is displayed overlapping with the actual image to search for human objects. There is no function to support.

  For applications that detect nearby items, there is a technology that recognizes items by image recognition, but it is necessary to increase the resolution of the image in order to improve the recognition rate, and the amount of calculation for identification is large. Since it is large, it is not suitable for mounting on a portable terminal device. Also, with image recognition, it is not possible to find an article that is not on the line of sight in the visible light region, such as behind an object.

  The present invention provides a searchable portable terminal device that combines a search target position measurement device such as a short-range secondary radar communication device and a shooting device such as a camera, and shows an image captured on a display and a search target. It is an object of the present invention to provide a system, a method, and the like capable of overlapping display with a pointer.

According to one aspect of the present invention for achieving the above-described object, a searchable portable terminal device includes a display; a photographing unit that captures an image displayed on the display;
Search target position measuring means for measuring the arrival direction and distance of a signal from the search target; and a video captured by the imaging means are displayed on the display, and the search target is within the range of the video displayed on the display. Display control means for determining whether or not the image is within the range and causing the video and the pointer indicating the search target to overlap and display on the display.

  According to another aspect of the present invention, a response communication device that can be attached to a search target includes: a transmission / reception antenna; a receiver that receives a search signal from the searchable device; and a search for a specific communication device to which the received search signal is addressed. A response processing unit that performs response processing in the case of a message; and a synchronization processing unit that controls transmission of a response signal in synchronization with the next expected reception timing based on the timing of the received signal; .

According to another aspect of the present invention, a search target display system for displaying a search target on a display includes a searchable portable terminal device and a response communication device attached to the search target.
The mobile terminal device has a display; a photographing means for photographing a video image displayed on the display; a search target position measuring means for measuring the arrival direction and distance of a response signal from the search target; and a video photographed by the photographing means on the display To determine whether the search target is within the range of the video displayed on the display, and if it is determined that the search target is within the range, the video and the pointer indicating the search target are overlapped. Display control means for displaying on the display.
The response communication device is a transmission / reception antenna; a receiver that receives a search signal from a mobile terminal device; a response processing unit that performs a response process when the received search signal is a specific communication device search message addressed to itself; A synchronization processing unit that controls to transmit a response signal in synchronization with the next expected reception timing based on the timing of the signal.

According to another aspect of the present invention, a search target display method for displaying a search target on a display includes:
Transmitting a search signal in a searchable mobile terminal device;
In the response communication device attached to the search target, the step of receiving the search signal; the step of performing response processing when the received search signal is a specific communication device search message addressed to itself; the timing of the received signal And controlling to send a response signal in synchronization with the next expected reception timing;
In the portable terminal device, the step of measuring the arrival direction and distance of the response signal from the search target; the video captured by the imaging unit is displayed on the display, and the search target is within the range of the video displayed on the display. Determining whether or not the image is within the range, and displaying the image and the pointer indicating the search target on the display in an overlapping manner.

  According to the embodiment of the present invention, an article to be searched for is displayed by overlappingly displaying an image obtained by an optical camera and a result of measuring an arrival direction and a distance of electromagnetic waves or sound waves by a secondary radar communication device on a display of a mobile terminal device. You can visually grasp where is. Further, by displaying attribute information such as name and ID acquired from the response communication device, it is possible to obtain information related to the article. By connecting to an external network such as the Internet, it is possible to obtain more detailed information related to the corresponding article, and to confirm history information such as when and where was searched last time.

  The embodiments of the present invention can be applied in various ways as follows.

-Search for the location of the property that is locked in-Search for lost golf balls covered with deep grass-Search for shopping at department stores and supermarkets-Search for works intended for appreciation at museums and museums

  First, an outline of an embodiment of the present invention will be described with reference to FIG.

  When the user of the mobile terminal device with a search function enables the radar search mode provided in the mobile terminal device, the optical camera, the display device, and the secondary radar communication device are activated. Further, when the user activates the search switch, a search signal having directivity for searching for a response communication device is transmitted from the secondary radar communication device using electromagnetic waves or sound waves. Search signals are broadly classified into search signals for searching for a specific response communication device and search signals for searching for an unspecified response communication device.

  The response communication device that has received the search signal analyzes the content of the search signal, and when it is a search signal addressed to itself or an unspecified majority search signal, transmits the response signal in synchronization with the search signal.

  The mobile terminal device that has received the response signal from the response communication device measures the arrival direction of the response signal using various techniques such as an array antenna. The mobile terminal device measures the distance from the synchronization information. Furthermore, the mobile terminal device acquires attribute information such as the ID and name of the response communication device from the content of the response signal.

  The obtained information on the direction of arrival is collated with the lens angle of view of the optical camera at that time, and converted into coordinates on the display. The information thus obtained is displayed so as to overlap with the video from the optical camera. That is, the position of the response communication device is displayed on the display in real time together with the camera image, and attribute information such as distance, ID, and name is displayed in a balloon format.

  The user can check more detailed data via the Internet connection or the like by selecting the information of the response communication device displayed on the display with a cursor or the like.

  FIG. 1 is a diagram illustrating a state in which the user has a search target pointer displayed on the display of the mobile terminal device. A star-shaped pointer indicates the current position of the response communication device attached to the key, and the balloon displays the name, affiliation, and distance to the response communication device as attribute information.

  Embodiments according to the present invention will be described below with reference to the drawings.

  First, a mobile terminal device with a search function according to an embodiment of the present invention will be described with reference to FIG. The mobile terminal device 200 with a search function includes an optical lens 210, a CCD 212, a CCD driver 214, operation buttons 260, a display 242 for displaying an image captured by the image capturing unit, an image processor 244, a RAM 246, a CPU 248, and a search target. It comprises a two-dimensional array antenna 230 as a position measuring means, a transceiver 232, a radar driver 234, and a ROM 250.

  Optical information obtained from the subject is received by the CCD 212 via the optical lens 210, processed by the CCD driver 214, and sent to the image processor 244.

  On the other hand, an electromagnetic wave signal or a sound wave signal transmitted from the response communication device attached to the search target existing in the subject is received and transmitted / received by the array antenna 230 or ESPAR antenna arranged in two dimensions in the horizontal direction and the vertical direction. To the device 232. Information from the transmitter / receiver 232 is transmitted to the CPU 248 via the radar driver 234, collated with the angle of view setting of the optical lens 210 in the CPU 248 or the radar driver 234, and the search target is displayed on the display 242. It is determined whether it is within the range. Further, the result of determining the coordinates in the display 242 is transmitted to the image processor 244.

  The radar driver 234 also sends the distance from the response communication device and attribute information such as ID and name to the image processor 244. The image processor 244 calculates the image information for overlap display using the information obtained from the CCD driver 214 and the information obtained from the radar driver 234 using a memory such as a DRAM, and displays the result on a display such as an LCD. To 242.

  The user sends an instruction to the CPU through the operation button 260, and the CCD driver 214 and the radar driver 234 perform calculation using the CPU and a flash memory.

  A response communication device according to an embodiment of the present invention will be described with reference to FIG. The response communication device 300 that can be mounted on the search target performs a response process when the search signal from the transmission / reception antenna 310 and the searchable device is a receiver 320, and the received search signal is a specific communication device search message addressed to itself. Response processing unit 330, modulator 340, transmitter 350, synchronization processing unit 360 for controlling to transmit a response signal in synchronization with the next expected reception timing based on the timing of the received signal, storing attribute information ROM 370 to be configured.

  The search signal received by the receiver 320 of the response processing unit 300 is sent to the response processing unit 330. The response processing unit 330 performs response processing if the received search signal is a specific communication device search message addressed to itself or an unspecific communication device search message. A response signal based on the response processing is modulated by the modulator 340, sent to the transmitter 350, and transmitted from the antenna 310.

  Here, attribute information such as name and ID used for the response process is read from the ROM 370.

  In addition, the timing of the received search signal is sent to the synchronization processing unit 360, and is subjected to synchronization processing so as to perform transmission in synchronization with the next expected reception timing. This process enables distance measurement in the searchable device. The time interval used for prediction is described in the search message.

  With reference to FIG. 4, the starting procedure of the searchable portable terminal device according to the embodiment of the present invention will be described.

  When the user presses a switch that is a part of the operation button 260 of the portable terminal device (S410), the optical camera is activated as a photographing unit (S420), and if not activated, an error message is displayed (S430).

  After the optical camera is activated, the radar communication device is activated (S450), and when it is not activated normally, an error message is displayed (S430). If it is normally activated, the search message is changed depending on whether or not to search for a specific communication device (S450). When searching for a specific communication device, a search signal for searching for the specific communication device is created (S470). At that time, it is necessary to acquire attribute information such as an ID of a specific communication device to be searched for in advance by another operation (S485).

  In the case of searching for an unspecified communication device, a search signal received by all response communication devices is created like a broadcast message (S480). After performing these operations, the state transitions to the state after startup (S490).

  With reference to FIG. 5, the procedure of the arrival angle / distance measurement and the result display in the search portable terminal device will be described.

  After being activated (S490), the search portable terminal device 200 starts transmission of a search signal (S520) by turning on a search switch that is a part of the operation button 260 (S512). The search signal is set to be transmitted once every fixed interval T seconds while the search switch is on. When the search portable terminal device receives the response signal from the response communication device (S530), it starts the following three types of processing.

  First processing: The arrival angle of radio waves or sound waves is measured (S550), and the searched response communication device is within the range of the image on the display 242 by collating with the angle of view of the current optical camera image (S546). It is determined whether or not there is (S552). If it is within the range of the display 242, the coordinates are calculated (S580), and the pointer to be searched is displayed in an overlapping manner on the display 242 (S548). If it is outside the range of the display 242, a message indicating that is calculated (S544), and an overlap display is displayed on the display 242 (S548).

  The arrival angle is measured by measuring delay times or phase differences of a plurality of incoming waves as shown in FIG. 6 by arranging at least two pairs of receiving array antennas and ESPAR antennas in the horizontal and vertical directions. To calculate the arrival angle (S550). By arranging the array antennas in two dimensions in the horizontal direction and in the vertical direction, it is possible to determine from which position of the two-dimensional coordinates on the display 242 the wave has arrived.

  Second process: A process of analyzing a response signal from the response communication apparatus and acquiring attribute information such as an ID and a name through an Internet connection or the like (S560). The result is sent to both the internal and external display methods (S544, S580) of the display 242 and is overlapped on the display 242 (S548).

  The attribute information includes, for example, an ID, a name, a position, a distance, the number of relay nodes in multihop communication, a related digital content address such as a URL, and relationship information (usage history, etc.) with a user.

  Third process: a process of measuring the distance from the response communication device. The distance measurement is performed when the signal synchronization flag is included in the received signal (S540). The distance is calculated by the difference between the delay time of the search signal and the response signal (S542), and the result is displayed on the display 242 in an overlapping manner (S548). If the synchronization is established, the distance is calculated from the delay difference. If the synchronization is not established, a measurement impossible message or nothing is displayed. By combining distance measurement and arrival angle measurement in this way, it becomes possible to display the three-dimensional position of the response communication device.

  The information display modes obtained by the above measurement and information acquisition include, for example, a real-time display mode, an image and information simultaneous recording mode, and a recorded playback mode.

  With reference to FIG. 7, the procedure when the user selects a pointer on the display and requests a detailed display in the search portable terminal device will be described.

  The user can select one or a plurality of pointers using a cursor or the like from the pointers (marks indicating the location of the response communication device) displayed on the display 242 of the search portable terminal device (S710). In addition, examples of the method for specifying the subject for which the detailed information is to be output include the following.

-Subject in the center of the shooting screen-Enter the desired conditions and the subject that matches the condition (subject that the user wants)
-Subject specified by the user using a keypad or the like When information on the ID of the selected response communication device exists on the memory (local memory) in the mobile terminal device (S720), the information ( Name and other attribute information) are acquired and displayed in an overlapping manner in the display 242 (S770). If there is no information relating to the ID in the local memory (S740), connection is made to the external network using a wireless packet network or the like, and attribute information or the like is acquired based on the ID information. Information acquired from the outside is stored in a local memory area such as a RAM (S760), and is displayed in an overlapping manner in the display 242 (S770). If nothing is found in the external network search, an error message is displayed in an overlapping manner (S750).

  Images used for visual output include still images and moving images, which can be selected.

  In the case of a still image, the image at the moment when the shutter is pressed with the optical camera and the positioning result by the secondary radar communication device are simultaneously saved, and the saved information is later retrieved and displayed on the display 242 to obtain the result shown in FIG. You can follow the steps shown. Furthermore, the stored information can be attached to an electronic mail or the like and transmitted to the outside.

  In the case of a moving image, an image continuously captured by the optical camera and a positioning result of the secondary radar communication device are displayed in real time on the display 242 so that the display procedure shown in FIG. 7 can be followed. The moving image can also be transmitted through a network together with the positioning result of the secondary radar communication device.

  It is also possible to select audio output, by reading out the attributes of response communication devices that are displayed in an overlapping manner, or by changing the volume of the sound when changing the orientation of the mobile terminal device It is also possible to express the direction and distance.

  Further, it is possible to output information obtained by the optical camera and the secondary radar communication device to an external device.

  The pointer indicating the search target output to the display 242 includes a direction pointer indicating the direction of the search target when the search target is outside the range of the image on the display 242, and the search target is within the image on the display 242. There are two types of search target pointers that indicate the search target. In the case of the direction pointer, the horizontal direction of the image and the depth direction of the image are displayed on the display 242. Information related to the search target includes characters, images, and sounds (such as operation sounds and calls).

  With reference to FIG. 8, a processing procedure until the response communication apparatus 300 transmits a response signal after receiving the search signal from the search portable terminal apparatus 200 will be described.

  The response communication device 300 wakes up from an idle state (S810) for signal reception at all times or intermittently, and determines the presence or absence of a search signal from the search portable terminal device (S820). If there is no reception, a wakeup timer is set to wake up next, and the state transitions to an idle state for a predetermined time and sleeps (S830). If there is reception (S820), it is determined whether or not synchronization with the timing of transmitting its own response signal is established (S340). If the signal is not synchronized, the signal synchronization flag is turned off, a response signal is transmitted, and a synchronization process with the response signal is attempted by a pull-in process (S850). If the response signal is synchronized (S340), the signal synchronization flag is turned on and the response signal is transmitted (S870).

  In the pull-in process, the next reception timing is predicted from the transmission timing included in the search signal, and the reception and transmission timing are matched. The signal synchronization flag is used when measuring the distance on the searching portable terminal device side. After the response signal is transmitted, it returns to the idle state (S810) and waits for the next reception.

  FIG. 9 shows a protocol procedure for exchanging the search signal and the response signal. This protocol is a request / response type procedure, and one response signal is returned from the response communication device for each search signal from the search portable terminal device. At this time, if the timing of receiving the search signal on the response communication device side is different from the timing of returning the response signal (as shown at the top of FIG. 9), the synchronization flag is turned off, The search portable terminal device can determine that synchronization is not achieved.

  When synchronization is established (as shown in the second and third responses in FIG. 9), the search portable terminal device can determine that synchronization is established by turning on the synchronization flag, and the distance Measurement processing will be performed. As synchronization methods, there are complete synchronization and offset synchronization. In the case of complete synchronization, it is indicated that the timing of reception and transmission are completely coincident. In the case of offset synchronization, transmission is performed at a certain interval from reception. FIG. 9 shows an example of complete synchronization.

  FIG. 10 shows the format of the search signal.

  The search signal includes the search signal type, ID, and transmission timing in addition to the header part and the footer part. The search signal type indicates the type of whether the search is a specific communication device search or non-specific communication device search. In the case of searching for a specific communication device, an ID for specifying the ID is entered. The transmission timing indicates in which cycle the search signal is transmitted.

  FIG. 11 shows a response signal format.

  The response signal includes synchronization information, ID, and attribute value information in addition to the header portion and the footer portion. The synchronization information includes type information indicating whether it is complete synchronization or offset synchronization, and a flag indicating whether it is currently synchronized. The ID information includes the ID value of the response communication device itself. The attribute value can include arbitrary information such as name and price.

  The portable terminal device, the response communication device, and the search target display system and method according to the present invention can be used to display an image captured on a display and a pointer indicating the search target in an overlapping manner.

It is the schematic of the display of the portable terminal device with a search function according to the Example of this invention. It is a block diagram which shows the structure of the portable terminal device with a search function according to the Example of this invention. It is a block diagram which shows the structure of the response communication apparatus according to the Example of this invention. It is a flowchart which shows the starting procedure of the searchable portable terminal device according to the Example of this invention. It is a flowchart which shows the procedure of the arrival angle and distance measurement in the search portable terminal device according to the Example of this invention, and a result display. It is a conceptual diagram which shows the method of calculating an arrival angle by measuring the delay time or phase difference of several incoming waves. It is a flowchart which shows the procedure when a user selects the pointer on a display and requests | requires detailed display in the search portable terminal device according to the Example of this invention. It is a flowchart which shows the process sequence after transmitting a response signal after a response communication apparatus receives the search signal from a search portable terminal device according to the Example of this invention. 6 is a chart illustrating a protocol procedure for exchanging a search signal and a response signal according to an embodiment of the present invention. 2 shows a format of a search signal according to an embodiment of the present invention. 4 shows a response signal format according to an embodiment of the present invention.

Explanation of symbols

200 Mobile Terminal Device with Search Function 210 Optical Lens 212 CCD
214 CCD driver 260 Operation button 242 Display 244 Image processor 246 RAM
248 CPU
230 Two-dimensional array antenna 232 Transceiver 234 Radar driver 250 ROM
280 Mobile Internet Communication Function 300 Response Communication Device 310 Antenna 320 Receiver 330 Response Processing Unit 340 Modulator 350 Transmitter 360 Synchronization Processing Unit 370 ROM

Claims (7)

A searchable mobile terminal device:
display;
Photographing means for photographing the image projected on the display;
A search target position measuring means for measuring the arrival direction and distance of a signal from the search target; and an image captured by the photographing means is displayed on the display, and the search target is displayed on the display. Display control means for determining whether or not the image is within the range and causing the video and the pointer indicating the search target to overlap and display on the display when determined to be within the range;
A searchable portable terminal device comprising: A searchable mobile terminal device according to claim 1,
The searchable portable terminal device, wherein the search target position measuring means is a secondary radar communication means for measuring the arrival direction and distance of electromagnetic waves or sound waves emitted from the search target. A searchable mobile terminal device according to claim 1,
The display control means also displays the attribute information of the search target on the display;
The searchable portable terminal device characterized by the above. A response communication device that can be attached to a search target:
Transmit / receive antenna;
A receiver for receiving a search signal from a searchable device;
A response processing unit that performs response processing when the received search signal is a specific communication device search message addressed to itself; and transmits a response signal in synchronization with the next expected reception timing based on the timing of the received signal A synchronization processor for controlling
A response communication device comprising: 5. The response communication device according to claim 4, further comprising storage means for storing own attribute information for use in the response process;
A response communication device comprising: A search target display system for displaying a search target on a display:
Consists of a searchable mobile terminal device and a response communication device attached to the search target,
The portable terminal device is
display;
Photographing means for photographing the image projected on the display;
A search target position measuring means for measuring the arrival direction and distance of the response signal from the search target; and an image photographed by the photographing means is displayed on the display, and the search target is displayed on the display. Display control means for determining whether or not the image is within the range of the image and displaying the image on the display by overlapping the image and a pointer indicating the search target when it is determined that the image is within the range;
Consisting of
The response communication device is
Transmit / receive antenna;
A receiver for receiving a search signal from the mobile terminal device;
A response processing unit that performs response processing when the received search signal is a specific communication device search message addressed to itself; and transmits a response signal in synchronization with the next expected reception timing based on the timing of the received signal A synchronization processor for controlling
Composed of,
A search target display system characterized by that. A search target display method for displaying a search target on a display:
In a searchable mobile terminal device,
Transmitting a search signal;
In the response communication device attached to the search target,
Receiving the search signal;
Performing a response process when the received search signal is a specific communication device search message addressed to itself;
Controlling to send a response signal in synchronization with the next expected reception timing based on the timing of the received signal;
In the portable terminal device,
Measuring an arrival direction and a distance of a response signal from the search target;
The image captured by the imaging means is displayed on a display, and it is determined whether or not the search target is within the range of the image displayed on the display. Overlapping with a pointer indicating the search target and displaying on the display;
Search target display method comprising:

Images