JP6051068B2 - Object search system and object search device - Google Patents

Description

  The present invention relates to an object search system and an object search apparatus including at least a self-propelled electronic device and an operation device communicating with the self-propelled electronic device.

Self-propelled electronic devices that run and work autonomously are becoming widespread. One example is a self-propelled vacuum cleaner. Other examples include a self-propelled ion generator and an air cleaner. Such a self-propelled electronic device is equipped with a communication function and reading means for reading identification information from an IC tag, and the identification information is read from an IC tag attached to a predetermined object and transmitted to a user-owned mobile phone. A type device has been proposed (see, for example, Patent Document 1).
Further, there has been proposed one that identifies an object arranged in a cleaning target area and executes a cleaning operation associated with the identified object (for example, see Patent Document 2).

JP 2007-317112 A JP 2005-31893 A

If the IC tag is attached to the object as in Patent Document 1, the position of the object changes, or the self-propelled device reads the identification information from the IC tag even if the position of the object is unknown. Thus, the object can be photographed by the camera, and the position information of the self-propelled device and the photographed image can be transmitted to the user's mobile phone to notify the state of the object. However, the function cannot be used unless the object is previously attached with an IC tag.
The present invention has been made in consideration of the above-described circumstances. For example, even an object that does not have a function of generating an identification signal, sound, or the like is used for traveling of a self-propelled electronic device. An object search system capable of finding an object in a range is provided.

This invention
(1) A system including a self-propelled electronic device that travels and performs work, and an operation device that communicates with the self-propelled electronic device, and that captures images at each traveled position and generates a recorded image A recording unit, a storage unit for storing the recorded image, a tag adding unit for recognizing an object shown in the recorded image and tagging the recorded image with a keyword corresponding to the object, and inputting a keyword as a query A query input unit for accepting, a search unit for searching and extracting the recorded image tagged with a keyword related to the input query from the storage unit, and a search result for outputting the extracted recorded image An object search system including an output unit is provided.
(2) An object search device capable of communicating with a self-propelled electronic device that travels and performs work, and a communication unit that receives a recorded image taken by the self-propelled electronic device from the self-propelled electronic device; A storage unit for storing the received recorded image, a tag adding unit for recognizing an object shown in the recorded image and tagging the recorded image with a keyword corresponding to the object, and a designated keyword being tagged And a search unit that searches and extracts the recorded image from the storage unit.

The object search system according to (1) of the present invention includes an image recording unit that shoots at each traveled position to generate a recorded image, and recognizes an object that appears in the recorded image and assigns a keyword corresponding to the object to the keyword. A tag adding unit for tagging a recorded image, and a search unit for searching and extracting a recorded image tagged with a keyword related to the input query from the storage unit. Even an object that does not have a function of generating sound or the like can find an object within a range in which the self-propelled electronic device has traveled.
The object search device according to the above (2) includes a communication unit that receives a recorded image captured by the self-propelled electronic device, a keyword that corresponds to the object recognized in the recorded image, and a keyword corresponding to the object. A tag adding unit for tagging and a search unit for searching and extracting the recorded image tagged with the specified keyword from the storage unit, so that an identification signal or sound is generated. Even if the object does not have the function to perform, it is possible to find the object in the range in which the self-propelled electronic device has traveled.

It is a block diagram which shows schematic structure of one Example of the object search system of this invention. (Embodiment 1) It is a perspective view which shows roughly an example of the external appearance of the self-propelled cleaner 1 shown in FIG. It is a bottom view which shows roughly the bottom face of the self-propelled cleaner shown in FIG. It is a flowchart which shows the process in cleaning operation among the object searches which the object search system of this invention performs. (Embodiment 1) It is a flowchart which shows the process at the time of object search execution among the object searches which the object search system of this invention performs. (Embodiment 1) It is a block diagram which shows schematic structure of the object search system of this embodiment. (Embodiment 2) In this embodiment, it is a flowchart which shows the process which a self-propelled cleaner moves to the imaging position of the selected image. (Embodiment 3)

Hereinafter, the present invention will be described in more detail with reference to the drawings. In addition, the following description is an illustration in all the points, Comprising: It should not be interpreted as limiting this invention.
(Embodiment 1)
<< 1-1. Self-propelled vacuum cleaner configuration >>
FIG. 1 is a block diagram showing a schematic configuration of an embodiment of the object search system of the present invention. As shown in FIG. 1, the object search system 100 according to the present invention is roughly composed of a self-propelled cleaner 1 which is one aspect of a self-propelled electronic device and a communication terminal device 110 which is one aspect of an operation device. The Note that the number of self-propelled electronic devices included in the object search system 100 is not limited to one, and may be plural. Similarly, the number of communication terminal devices is not limited to one and may be a plurality.
The self-propelled cleaner 1 mainly includes a rotary brush 9, a side brush 10, a device control unit 11, a rechargeable battery 12, an obstacle detection unit 14, and a dust collection unit 15. Furthermore, the power unit 21, the right drive wheel 22R, the left drive wheel 22L, the intake port 31, the exhaust port 32, the device operation unit 51, the device storage unit 61, the position recording unit 71, the device communication unit 81, the image recording unit 41, and the electric motor A blower 33 is provided.

The self-propelled cleaner 1 cleans the floor surface by sucking in air containing dust on the floor surface and exhausting the air from which dust has been removed while traveling on the floor surface of the place where it is installed. And when cleaning is completed, it has the function to return autonomously to the charging station which is not illustrated.
FIG. 2 is a perspective view schematically showing an example of the appearance of the self-propelled cleaner 1 shown in FIG.
FIG. 3 is a bottom view schematically showing the bottom surface of the self-propelled cleaner shown in FIG. 2.

As shown in FIG. 2, the self-propelled cleaner 1 includes a disk-shaped housing 2.
The housing 2 includes a bottom plate 2a, a top plate 2b having a lid 3 that can be opened and closed to put in and out a dust collecting container accommodated in the housing 2, and a bottom plate 2a and a top plate 2b. And a side plate 2c having a ring shape in plan view provided along the outer periphery. An exhaust port 32 is formed near the boundary between the front portion and the middle portion of the top plate 2b. The side plate 2c is divided into two parts, front and rear, the side plate front part functions as a bumper, and a collision sensor (not shown) for detecting a collision of the side plate front part is provided inside. Further, as shown in FIG. 2, a front ultrasonic sensor 14F is disposed on the front side, and a left ultrasonic sensor 14L is disposed on the left side. Although hidden in FIG. 2, a right ultrasonic sensor is arranged on the right side. The front, left and right ultrasonic sensors and the collision sensor are included in the obstacle detection unit 14 of FIG.

  As shown in FIG. 3, the bottom plate 2a is formed with a plurality of holes that expose the front wheel 27, the right driving wheel 22R, the left driving wheel 22L, and the rear wheel 26 from the inside of the housing 2 and project outside. Yes. Further, the rotary brush 9 is located behind the intake port 31, the side brush 10 is located on the left and right sides of the intake port 31, the front wheel floor surface detection sensor 18 is located in front of the front wheel 27, the left wheel floor surface detection sensor 19L is located forward of the left drive wheel 22L, and the right drive is driven. A right wheel floor surface detection sensor 19R is disposed in front of the wheel 22R. The front wheel floor surface detection sensor 18, the left wheel floor surface detection sensor 19L, and the right wheel floor surface detection sensor 19R are included in the obstacle detection unit 14 of FIG.

The self-propelled cleaner 1 travels in the direction in which the right driving wheel 22R and the left driving wheel 22L rotate forward in the same direction and moves forward, and the front ultrasonic sensor 14F is disposed. Further, the left and right drive wheels rotate backward in the same direction and move backward, and turn by rotating in opposite directions. For example, when the self-propelled cleaner 1 reaches the periphery of the cleaning area by each sensor of the obstacle detection unit 14 and when an obstacle is detected on the course, the left and right drive wheels are decelerated and then stopped. Thereafter, the left and right drive wheels are rotated in opposite directions to turn and change directions. In this way, the self-propelled cleaner 1 self-propels while avoiding obstacles over the entire installation location or the entire desired range.
Here, the front means the forward direction of the self-propelled cleaner 1 (upward along the paper surface in FIG. 3), and the rear means the backward direction of the self-propelled cleaner 1 (the paper surface in FIG. 3). Along the bottom).

Hereinafter, each component shown in FIG. 1 will be described.
1 is a part that controls the operation of each component of the self-propelled cleaner 1, and is mainly realized by a microcomputer including a CPU, a RAM, an I / O controller, a timer, and the like.
The CPU organically operates each hardware based on a control program stored in advance in a device storage unit 61, which will be described later, and developed in the RAM, and executes the cleaning function, the traveling function, and the like of the present invention.
The device control unit 11 realizes functions as the tag addition unit 11a and the search unit 11b by executing a predetermined processing program stored in advance in the device storage unit 61.

The rechargeable battery 12 is a part that supplies power to each functional element of the self-propelled cleaner 1, and is a part that mainly supplies power for performing a cleaning function and travel control. For example, a rechargeable battery such as a lithium ion battery, a nickel metal hydride battery, or a Ni—Cd battery is used.
The rechargeable battery 12 is charged by bringing the exposed charging terminals into contact with each other in a state where the self-propelled cleaner 1 is brought close to a charging station (not shown).

The obstacle detection unit 14 detects that the self-propelled cleaner 1 is in contact with or approaches an obstacle such as an indoor wall, desk, or chair while the self-propelled cleaner 1 is traveling, and detects a step such as a descending staircase leading to a fall. . The obstacle detection unit 14 detects proximity to the obstacle using an ultrasonic sensor. Instead of the ultrasonic sensor or together with the ultrasonic sensor, other types of non-contact sensors such as an infrared distance measuring sensor may be used.
The obstacle detection unit 14 includes the above-described collision sensor. The collision sensor is disposed, for example, inside the side plate 2c of the housing 2 in order to detect that the self-propelled cleaner 1 has come into contact with an obstacle during traveling. The CPU knows that the side plate 2c has collided with the obstacle based on the output signal from the collision sensor.

Further, the front wheel floor surface detection sensor 18, the left wheel floor surface detection sensor 19L, and the right wheel floor surface detection sensor 19R as the obstacle detection unit 14 detect large steps such as descending stairs.
Based on the signal output from the obstacle detection unit 14, the CPU recognizes the position where an obstacle or a step exists. Based on the recognized obstacle and step position information, the next direction to travel is determined while avoiding the obstacle and step. The left wheel floor surface detection sensor 19L and the right wheel floor surface detection sensor 19R detect the descending stairs when the front wheel floor surface detection sensor 18 fails to detect a step or fails, and the self-propelled cleaner 1 Prevent falling to the downstairs.

The image recording unit 41 includes a camera 41C and an object recognition unit 41A.
The camera 41C outputs the surrounding situation as an image signal to the object recognition unit 41A. The object recognition unit 41 </ b> A recognizes an object when the object is reflected in the image signal using a known pattern recognition technique. Further, information for calculating (that is, determining) the direction and distance of the object may be provided. The direction and distance of the object can be calculated on the basis of information indicating at which position in the image of the camera 41C the object is shown. Alternatively, the direction and distance of the object can be determined by, for example, taking a so-called 3D image by photographing the surroundings from different viewpoints using two cameras on the left and right, analyzing the depth of the 3D image, and analyzing the direction and the depth distance. Information may be acquired.

  The power unit 21 is a part that realizes traveling by a drive motor that rotates and stops the left and right drive wheels of the self-propelled cleaner 1. By configuring the drive motor so that the left and right drive wheels can be rotated independently in both forward and reverse directions, the traveling state of the self-propelled cleaner 1 such as advancing, retreating, turning, and acceleration / deceleration is realized.

The intake port 31 and the exhaust port 32 are portions that perform intake and exhaust of air for cleaning, respectively.
The dust collection part 15 is a part which performs the cleaning function which collects indoor garbage and dust, and is mainly provided with the dust collection container which is not shown in figure, the filter part, and the cover part which covers a dust collection container and a filter part. Further, the dust collecting container has an inflow port that leads to an inflow path that communicates with the intake port 31, and an exhaust port that communicates with an exhaust path that communicates with the exhaust port 32. An electric blower 33 is disposed in the discharge path. The electric blower 33 sucks air from the intake port 31, guides the air into the dust collecting container through the inflow passage, and discharges the air after dust collection from the exhaust port 32 to the outside through the discharge passage. generate.

  A rotary brush 9 that rotates about an axis parallel to the bottom surface is provided behind the intake port 31, and a side brush 10 that rotates about a vertical rotation axis is provided on both the left and right sides of the intake port 31. ing. The rotating brush 9 is formed by implanting a brush spirally on the outer peripheral surface of a roller that is a rotating shaft. The side brush 10 is formed by providing a brush bundle radially at the lower end of the rotating shaft. The rotating shaft of the rotating brush 9 and the rotating shaft of the pair of side brushes 10 are pivotally attached to a part of the bottom plate 2a of the housing 2, and a brush motor 34 provided in the vicinity thereof, a pulley, a belt, etc. It is connected via a power transmission mechanism that includes it. This is merely an example, and a dedicated drive motor that rotates the side brush 10 may be provided.

Moreover, the self-propelled cleaner 1 according to this embodiment may have an ion generation function as an additional function.
The device operation unit 51 is a part where the user inputs an instruction for the operation of the self-propelled cleaner 1, and is provided on the surface of the casing of the self-propelled cleaner 1 as an operation panel or an operation button.
Further, the communication terminal device may function as a remote control unit separately from the operation panel and operation buttons provided in the cleaner body.

  The device operation unit 51 includes a main power switch and a power switch. The main power switch turns on / off the power supply from the rechargeable battery 12 to the device control unit 11 and the like in a circuit. The power switch turns on / off the power of the self-propelled cleaner 1. As the device operation unit 51, other switches (for example, a start switch for starting a cleaning operation, a charge request switch, an operation mode switch, a timer switch) may be further provided. When receiving an instruction from a communication terminal device functioning as a remote controller instead of the device operation unit 51, the device control unit 11 responds to the instruction, for example, controls the power unit 21 to run in the direction instructed by the user. Or stop running.

The device storage unit 61 is a part that stores information necessary for realizing various functions of the self-propelled cleaner 1 and a control program, and a non-volatile semiconductor storage element such as a flash memory or a storage medium such as a hard disk is stored therein. Used.
The device storage unit 61 includes, for example, battery information indicating the remaining capacity of the rechargeable battery 12, history of the travel route of the self-propelled cleaner 1, position information 63 indicating the current position and direction, and a self-propelled cleaner. Operation mode information indicating one operation mode is stored. The operation mode information stores an operation mode, a standby mode, and a sleep mode. The operation mode is data indicating that a cleaning operation is being performed. The standby mode is data indicating that the state of the self-propelled cleaner is the standby mode. The standby mode is a mode in which cleaning can be started in response to an instruction from the user, for example, when an activation switch of the operation unit is turned on. The sleep mode is data indicating that the sleep mode is in the power saving state.

The position recording unit 71 records the travel direction and distance of the self-propelled cleaner 1 with rotary encoders 71R and 71L attached to the right drive wheel 22R and the left drive wheel 22L, respectively. However, in order to correct errors due to slippage between the right driving wheel 22R and the left driving wheel 22L and the floor surface, the acceleration sensor 71a and the gyro sensor 71b are mounted for correction.
The device communication unit 81 has a function (communication with a long-distance communication function) for communicating with another device that exists in a remote place via the communication network 90, and another device that exists within a communicable range (for example, within the same facility). And a communication means (communication function) for performing communication between devices.
Examples of the communication network 90 used for the above-described long-distance communication function include the Internet, a telephone line network, a mobile communication network, a CATV communication network, a satellite communication network, and the like.
In addition, as the short-range communication function, for example, WiFi (registered trademark) that interconnects wireless devices using IEEE802.11 (IEEE802.11a or IEEE802.11b), which is one of the wireless LAN standards, is used. Use a communication function of a device, a communication function based on a wireless LAN standard other than IEEE 802.11, a communication function using a short-range wireless communication standard such as Bluetooth (registered trademark) or ZigBee (registered trademark), an infrared communication function, etc. it can. Note that the communication network 90 includes both the long-range communication function and the short-range communication function.

<< 1-2. Configuration of communication terminal device >>
As illustrated in FIG. 1, the communication terminal device 110 includes a terminal control unit 111, a terminal communication unit 112, a terminal display unit 113, a terminal operation unit 114, and a terminal storage unit 115. The configuration of the communication terminal device 110 is not particularly limited as long as it is a communication terminal device having the functions of the above-described units. For example, a smartphone, a tablet terminal, a mobile phone, a PDA (Personal Digital Assistance), a personal computer, a mobile phone A type game machine or the like can be used.

  The terminal control unit 111 is a control unit of the communication terminal device 110 that controls the operation of each unit of the communication terminal device 110. The terminal control unit 111 includes, for example, an arithmetic processing unit such as a CPU and a dedicated processor, and a storage unit such as a RAM (Random Access Memory), a ROM (Read Only Memory), and an HDD (Hard Disk Drive) (not shown). ) And the like. The terminal control unit 111 controls the operation of each unit of the communication terminal device 110 by reading and executing various information stored in the storage unit and a program for performing various controls.

  The terminal communication unit 112 has a function (communication with long distance communication) for communicating with other devices located in a remote place via the communication network 90 and another device existing within a communicable range (for example, in the same facility). And a communication means (communication function) for performing communication between devices. As the terminal communication part 112, the thing similar to the apparatus communication part 81 of the self-propelled cleaner 1 can be used, for example.

  The terminal display unit 113 displays various information to be presented to the user in response to an instruction from the terminal control unit 111. The configuration of the terminal display unit 113 is not particularly limited, and for example, a liquid crystal display panel, an organic EL panel, a plasma display panel, or the like can be used. The terminal display unit 113 also functions as the search result output unit 113a. That is, the terminal display unit 113 displays the recorded image extracted as the search result in association with the position.

The terminal operation unit 114 receives an operation input from the user and transmits it to the terminal control unit 111. The configuration of the terminal operation unit 114 is not particularly limited, and various conventionally known operation input means can be used. In the present embodiment, a touch panel provided with the function of the terminal display unit 113 and the function of the terminal operation unit 114 is used. The terminal operation unit 114 also functions as the query input unit 114a. That is, the terminal operation unit 114 receives an input by a user of a keyword as a query.
The terminal storage unit 115 is a storage unit that stores various types of information used in the communication terminal device 110. The configuration of the terminal storage unit 115 is not particularly limited, and a known storage unit can be used.

<< 1-3. Image recording during cleaning operation >>
Hereinafter, an example of an object search processing procedure performed by the object search system 100 according to the present invention will be described with reference to flowcharts.
4 and 5 are flowcharts of processing relating to object search executed by the object search system 100 of the present invention. FIG. 4 shows a process of storing a recorded image during the cleaning operation of the self-propelled cleaner as the self-propelled electronic device prior to the object search. FIG. 5 shows processing executed on the communication terminal device side and the self-propelled cleaner side as the operation device when performing object search.

  First, it demonstrates based on FIG. The self-propelled cleaner 1 as a self-propelled electronic device performs autonomous traveling for the cleaning operation which is the original function, but photographs various places in the traveling room with the camera 41C, and records the recorded images. Stored in the device storage unit 61. The device control unit 11 secures a FIFO area for storing the recorded image in the device storage unit 61, and sequentially stores the recorded image in the FIFO area. When the FIFO capacity is full, the device control unit 11 sequentially erases the previously recorded images from the recorded images stored in the FIFO area, and stores a new recorded image in the vacant area.

In the standby state, the device control unit 11 monitors whether a start switch (not shown) included in the device operation unit 51 is turned on to instruct the start of the cleaning operation (step S11 in FIG. 4). If the instruction | indication of cleaning operation start is made (Yes of step S11), the apparatus control part 11 will control the power part 21 so that it may drive | work, avoiding the obstacle detected by the obstacle detection part 14. FIG. Further, the electric blower 33 and the brush motor 34 are rotated to perform a cleaning operation (step S13).
During traveling, every time a predetermined period elapses (step S15), the camera 41C takes an image and acquires a recorded image (step S17).
Subsequently, the object recognition unit 41A performs object recognition of the recorded image that has been shot, and adds the position information acquired by the position recording unit 71 to the recorded image. The position recording unit 71 sequentially updates information to a position that changes with traveling. The memory area of the position information 63 may be secured in the device storage unit 61, and the position information updated by the position recording unit 71 may be sequentially stored and updated in the position information 63. Further, the device control unit 11 adds a tag to the recorded image as the tag addition unit 11a (step S19).
The function of the tag addition unit 11a is to add a keyword related to the object recognized by the object recognition unit 41A to the recorded image. For example, when a key that has fallen on the floor in front of the travel path is shown in an image captured by the camera 41C, the object recognition unit 41A uses a well-known pattern recognition technique to classify an object (object) of the class “key”. Recognize that is reflected. The class here is a general concept of an object such as “key”, “clip”, and “pen”. As described above, a technique for extracting an object included in an image at a class level by applying pattern matching is known as general object recognition. As illustrated in FIG. 1, the device storage unit 61 may store in advance as a tag database 65 a feature amount and a tag of an object to be recognized.

  In pattern matching, generally, features of an object to be recognized are extracted as feature amounts by a predetermined feature amount extraction process and stored in a memory. As the feature quantity extraction processing, various methods such as SIFT (Scale-Invariant Feature Transform) for obtaining local feature quantities that are invariant to scale change and rotation of an image based on the arrangement of shading as local feature quantities of an object are known. It has been. When a recorded image is given, the same feature amount extraction process is applied to the recorded image to extract the feature amount, and if there is something similar to the feature amount stored in advance in the memory, it corresponds. It is determined that the object (object) is reflected in the image.

  For general object recognition, a technique for recognizing only an object instance, for example, a key having a specific shape among various objects included in a class of “key” is known as a specific object recognition technique.

As described above, the device control unit 11 adds position information and a tag at the time of shooting to the recorded image and stores them in the device storage unit 61. In addition, the self-propelled cleaner 1 may be provided with a real time clock (not shown in FIG. 1), and the date and time at which the photographing is pointed may be further added to the recorded image.
Subsequently, the device control unit 11 determines whether to end or continue the cleaning operation (step S21). When continuing (No of step S21), a routine returns to above-mentioned step S13, and continues a cleaning operation | movement and intermittent imaging | photography.
When the cleaning operation is finished (Yes in step S21), the device control unit 11 returns the self-propelled cleaner 1 to the charging stand arranged in the room (step S23). When returning to the charging stand, the cleaning operation process is terminated (step S25). Various conditions are conceivable as conditions for ending the cleaning operation. In particular, when the remaining capacity of the rechargeable battery 12 is less than a predetermined value, it is determined that it is necessary to return to the charging stand. Alternatively, or in addition thereto, it may be determined that the cleaning operation is terminated when a predetermined period has elapsed since the start of the cleaning operation. Furthermore, when the spot mode for cleaning the periphery of the position designated as the operation mode is selected, the cleaning operation may be determined to be finished when a predetermined range is cleaned around the designated position.

<< 1-4. Object search process >>
Next, processing at the time of object search will be described. FIG. 5 shows two flowcharts together. The flowchart on the left side is a process executed by the terminal control unit 111 on the communication terminal apparatus 110 side, and the flowchart on the right side is a process executed by the device control unit 11 on the self-propelled cleaner 1 side. The communication terminal device 110 and the self-propelled cleaner 1 perform processing in cooperation while performing communication via the communication network 90.

  First, it is assumed that the user of the communication terminal device 110 performs an operation on the terminal operation unit 114 to select a menu for executing object search. Then, the terminal control unit 111 that monitors the operation input of the terminal operation unit 114 (step S31), responds to the operation (Yes in step S31), and displays a screen that accepts an input of a keyword as a query. (Step S33). At this stage, the terminal operation unit 114 functions as the query input unit 114a. Note that display data of a menu screen for executing an object search and a screen for accepting an input of a query are stored in advance in the terminal storage unit 115. It is assumed that the user performs an operation for inputting a query on the screen to the terminal operation unit 114. Then, the terminal control unit 111 that monitors the operation input (Step S35) responds to the operation (Yes in Step S35) and sends the input query from the terminal communication unit 112 via the communication network 90. Send to the traveling vacuum cleaner 1. Then, a request is made to search for an image to which a tag related to the query is added (step S37).

Let's move on to the description of the self-propelled cleaner 1 side. In the self-propelled cleaner 1, the device control unit 11 that monitors whether the device communication unit 81 has received a request for object search (step S51), responds to the request (Yes in step S51). A recorded image to which a tag related to the query is added is extracted from the device storage unit 61 (step S53). This process of the device control unit 11 corresponds to the function as the search unit 11b.
Returning to the description of the communication terminal device 110 side. When one or more recorded images are extracted as the search results, the device control unit 11 transmits the images from the device communication unit 81 to the communication terminal device 110 via the communication network 90 (step S53). If there is no corresponding image, a search result to that effect is transmitted to communication terminal apparatus 110.

The terminal control unit 111 of the communication terminal device 110 monitors whether or not the terminal communication unit 112 has received a search result from the self-propelled cleaner 1 (step S39). In response to receiving the search result (Yes in step S39), the search result is displayed on the terminal display unit 113 (step S41). At this stage, the terminal display unit 113 functions as the search result output unit 113a.
When the image corresponding to the search result is found, the terminal control unit 111 displays the list on the terminal display unit 113 together with the position information added to the image. If there is no corresponding image, a message to that effect is displayed on the terminal display unit 113. When the date and time when the image was taken is added to the search result image, the date and time may be displayed together with the image.

  The above is the processing of the device control unit 11 and the terminal control unit 111, but the user finds an image that matches the one that the user is looking at by looking at the image displayed on the terminal display unit 113, and the position information displayed together with the image Based on this, you can identify where you are looking for.

(Embodiment 2)
In the first embodiment, the object search includes the self-propelled cleaner 1 and the communication terminal device 110 that communicate with each other. In this embodiment, a configuration in which an information processing device is added to the communication device will be described.
FIG. 6 is a block diagram showing a schematic configuration of the object search system 100 of this embodiment. The same parts as those in FIG. 1 are denoted by the same reference numerals.
One of the differences from FIG. 1 is a configuration in which a server device 150 is added and is communicably connected to the self-propelled cleaner 1 and the communication terminal device 110 via a communication network 90.

  The server device 150 includes a server control unit 151, a server communication unit 153, and a server storage unit 155. The tag addition unit and the search unit provided in the device control unit 11 of the self-propelled cleaner 1 in FIG. 1 are provided in the server control unit 151 of the server device 150 in FIG. 1 is stored in the server storage unit 155 in FIG. The tag database may be stored in the server storage unit 155. Here, the server device 150 is an example of an object search device according to the present invention.

  In the first embodiment, step S19 in FIG. 4, during the cleaning operation, the device control unit 11 adds position information and a tag to the photographed recorded image and stores them in the device storage unit 61. In this embodiment, instead, the device control unit 11 adds position information to the recorded image that has been shot, and then transmits the information from the device communication unit 81 to the server device 150 via the communication network 90. When the server communication unit 153 receives the transmitted image, the server control unit 151 adds a tag to the recorded image received from the self-propelled cleaner 1 and stores it in the server storage unit 155. At this stage, the server control unit 151 functions as the tag addition unit 151a.

The server storage unit 155 may store the tag database 157, and the tag addition unit 151a may perform tagging with reference to the tag database 157.
Further, in step S37 of the first embodiment and FIG. 5, the terminal control unit 111 sends a query to the self-propelled cleaner 1 and requests to search for an image, and the search result is self-propelled in step S39. In this embodiment, instead of receiving from the vacuum cleaner 1, a query is sent to the server device 150 to request an image search. In step S39, the search result is received from the server apparatus 150. Further, the processing of steps S51 and S53 executed by the device control unit 11 in FIG. 5 is executed by the server control unit 151 of the server device 150 in this embodiment. That is, the server control unit 151 extracts a recorded image to which a tag related to the query is added from the server storage unit 155 as the search unit 151b.

  The server device 150 may be provided by, for example, a manufacturer of the self-propelled cleaner 1 or a service provider. Or you may prepare. A plurality of self-propelled cleaners and a plurality of communication terminal devices may share the server device 150. In that case, the recorded image may be stored in the server storage unit 155 for each self-propelled cleaner, and the terminal control unit 111 may designate which self-propelled cleaner search image is to be searched.

(Embodiment 3)
In this embodiment, after the search result image is displayed on the terminal display unit 113, the position of the self-propelled cleaner 1 is received based on the position information added to the selected image by accepting the selection of the image by the user. The mode of moving to will be described.

  FIG. 7 is executed in this embodiment on the communication terminal device side and the self-propelled cleaner side as the operation devices when the self-propelled cleaner moves to the position where the selected image is taken. It is a flowchart which shows a process. Processing will be described with reference to FIG. In FIG. 7, the terminal control unit 111 of the communication terminal device 110 performs the same processing as steps S <b> 31 to S <b> 41 in FIG. 5 and displays the search result image on the terminal display unit 113. Then, it is monitored whether any displayed image is selected by the user (step S43). If any image is selected (Yes in step S43), the position information added to the selected image is transmitted to the self-propelled cleaner 1 (step S45).

  The device control unit 11 of the self-propelled cleaner 1 performs the same processing as steps S51 to S53 in FIG. 5 and then monitors whether or not the device communication unit 81 has received the position information from the communication terminal device 110. (Step S55). When the position information is received (Yes in step S55), the device control unit 11 starts the traveling of the self-propelled cleaner 1 in response to the position indicated by the position information. When the movement to the target location is completed while avoiding the obstacle (Yes in step S59), the device control unit transmits the arrival at the target location to the communication terminal device 110 (step S61).

Returning to the description of the processing of the terminal control unit 111. The terminal control unit 111 waits for the terminal communication unit 112 to receive a notification that the self-propelled cleaner 1 has arrived at the target location (step S47). When the notification is received (Yes in step S47), the terminal control unit 111 causes the terminal display unit 113 to display a message indicating that the self-propelled cleaner 1 has arrived at the target location (step S49), and ends the process. .
The device control unit 11 performs control so that when the self-propelled cleaner 1 arrives at the target location, the surrounding state is captured by the camera 41C and transmitted to the communication terminal device 110. The terminal control unit 111 When the communication unit 112 receives the image, the terminal display unit 113 may be controlled to display the received image.

As mentioned above,
(I) An object search system according to the present invention is a system including at least a self-propelled electronic device that travels and performs work, and an operation device that communicates with the self-propelled electronic device, and images are taken at each traveled position. An image recording unit for generating a recorded image by performing a recording process, a storage unit for storing the recorded image, and a tag adding unit for recognizing an object shown in the recorded image and tagging the recorded image with a keyword corresponding to the object A query input unit that accepts input of a keyword as a query, a search unit that searches and extracts the recorded image tagged with a keyword related to the input query from the storage unit, And a search result output unit for outputting the recorded image.

In the present invention, the self-propelled electronic device travels and performs operations such as cleaning, air cleaning, and ion generation while traveling. A self-propelled cleaner is mentioned as an example of the specific aspect. In the above-described embodiment, the self-propelled electronic device corresponds to a self-propelled cleaner.
The operation device is used by a user to search for an object. The specific aspect is a smart phone or a tablet terminal, for example. In the above-described embodiment, the operation device corresponds to a communication terminal device.
The object search system of the present invention includes a self-propelled electronic device and an operation device as a minimum configuration, but may further include an information processing device that communicates with at least one of them. The information processing apparatus may be configured by a plurality of apparatuses in which a plurality of apparatuses cooperate to execute one process in a distributed manner. In the second embodiment, the information processing apparatus corresponds to a server apparatus.

The image recording unit is provided in a self-propelled device. An example of the specific mode is a camera. In the above-described embodiment, the self-propelled cleaner includes the camera 41C as an image recording unit.
The storage unit is a readable / writable memory that stores each recorded image generated by the image recording unit. Specific examples thereof include a flash memory and a hard disk drive. The storage unit may be included in the self-propelled electronic device, but is not limited thereto. As described above, when the system includes an information processing apparatus, for example, the information processing apparatus may include the storage unit. The device storage unit included in the self-propelled cleaner in the first embodiment provides the function of the storage unit of the present invention.

The tag adding unit adds a search tag to the recorded image. A specific aspect thereof is, for example, a CPU that executes processing for adding a tag to each recorded image. Although the structure which a self-propelled cleaner provided with the said memory | storage part also has a tag addition part is preferable, it is not limited to this. As described above, when the system includes an information processing apparatus, the information processing apparatus may be provided, for example. In the first embodiment, the function of the tag addition unit is realized when the device control unit executes tag addition processing. Note that the terminal storage unit may store a tag database referred to by the tag addition unit.
The query input unit is provided in the operation device, and inputs a search keyword when a user searches for an object. The specific aspect is the operation input means with which an operating device is provided, for example. In the above-described embodiment, the query input unit corresponds to a terminal operation unit included in a communication terminal device as an operation device.

The search unit performs a process for searching for a recorded image related to the input query. A specific aspect thereof is, for example, a CPU that executes search processing. A configuration in which a self-propelled cleaner provided with the storage unit also includes a search unit is preferable, but is not limited thereto. As described above, when the system includes an information processing apparatus, the information processing apparatus may include the information processing apparatus. In the above-described embodiment, the search unit corresponds to a device control unit included in the self-propelled cleaner.
The search result output unit is provided in the operating device and presents the image extracted by the search unit to the user. The specific aspect is a display part with which an operating device is provided, for example. In the above-described embodiment, the search result output unit corresponds to a terminal display unit included in the communication terminal device.

  (Ii) An object search device according to the present invention is an object search device capable of communicating with a self-propelled electronic device that travels and performs work, and a recorded image taken by the self-propelled electronic device is recorded on the self-propelled electronic device. A communication unit that receives from the electronic device, a storage unit that stores the received recorded image, a tag addition unit that recognizes an object reflected in the recorded image and tags the recorded image with a keyword corresponding to the object, And a search unit for searching and extracting the recorded image tagged with the specified keyword from the storage unit.

Furthermore, the preferable aspect of this invention is demonstrated.
(Iii) a position recording unit that records a position where the self-propelled electronic device travels, and the recording unit stores the recorded image in association with the position where the recorded image was captured; The result output unit may output the position associated with the recorded image together with the recorded image. If it does in this way, the search result output part can output the position where the record image was photoed with the record image of the search result.
The position recording unit is provided in the self-propelled device and records a position where the self-propelled device travels. As a specific aspect thereof, for example, a self-propelled device may be equipped with a gyro sensor and an acceleration sensor to acquire the traveling direction and distance. Alternatively, the traveling direction and distance may be acquired by a rotary encoder attached to a wheel or motor that causes the self-propelled device to travel. Alternatively, the self-propelled type may be equipped with a camera and acquire a traveling direction and distance from an image captured by the camera. A combination of these may also be used. The camera may also function as an image recording unit. In the embodiment, a self-propelled cleaner as a self-propelled device includes a rotary encoder, a gyro sensor, and an acceleration sensor as a position recording unit.

(Iv) a time providing unit that provides a time at which the recorded image was captured; the storage unit stores each recorded image in association with the time; and the search result output unit is extracted The position and time associated with the recorded image may be output together with the recorded image.
In this way, it is possible to confirm when the extracted recorded image was taken, that is, when the object in the recorded image was present.
The time providing unit preferably has a configuration included in the self-propelled electronic device like the position recording unit, but is not limited thereto, and may be included in the operation device. Moreover, when the system includes an information processing apparatus as described above, the information processing apparatus may be provided.

(V) You may further provide the object extraction part which cuts out the part in which the said object was reflected from the recording image in which the recognized object was reflected, and memorize | stores it in the said memory | storage part.
In this way, even when the target object is small in a part of the recorded image, the target object can be cut out from the unnecessary part and presented to the user in an easy-to-understand manner.
The object extraction unit preferably has a configuration provided in the operating device as in the tag addition unit, but is not limited thereto, and may be provided in a self-propelled electronic device. Moreover, when the system includes an information processing apparatus as described above, the information processing apparatus may be provided.

(Vi) The tag adding unit includes a tag database in which an image of an object assumed as a query or a feature amount extracted from the image is associated with a keyword, and is associated with an image or feature amount in the tag database. It may be checked whether or not a portion is included in each recorded image, and if a related portion is included, a keyword associated with the image or feature amount may be tagged to the recorded image.
If it does in this way, a tag can be added to each record picture about a subject registered beforehand in a tag database, without bothering a user's hand. A specific aspect of the tag database is a database stored in advance in a memory. A configuration in which a self-propelled cleaner provided with the storage unit also includes a search unit is preferable, but is not limited thereto. As described above, when the system includes an information processing apparatus, the information processing apparatus may include the information processing apparatus. In the above-described embodiment, the search unit corresponds to a device control unit included in the self-propelled cleaner.

(Vii) an input unit that accepts selection by the user of one or more recorded images extracted by the search unit; a travel control unit that moves the self-propelled electronic device to a position associated with the selected recorded image; May be further provided.
In this way, it is possible to move the self-propelled electronic device to a place where the object appears in the selected recorded image and guide the user to the place where the object exists.
The input unit is provided in an operating device. Although the said travel control part has the structure with which a self-propelled electronic device is provided similarly to a position recording part, it is not limited to this, You may provide with an operating device. Moreover, when the system includes an information processing apparatus as described above, the information processing apparatus may be provided.

Preferred embodiments of the present invention include combinations of any of the above-described plurality of embodiments.
In addition to the embodiments described above, there can be various modifications of the present invention. These modifications should not be construed as not belonging to the scope of the present invention. The present invention should include the meaning equivalent to the scope of the claims and all modifications within the scope.

1: Self-propelled cleaner, 2: Housing, 2a: Bottom plate, 2b: Top plate, 2c: Side plate, 3: Lid, 9: Rotating brush, 10: Side brush, 11: Device control unit, 11a: Tag Additional part, 11b: Search part, 12: Rechargeable battery, 14: Obstacle detection part, 14F: Front ultrasonic sensor, 14L: Left ultrasonic sensor, 15: Dust collecting part, 18: Front wheel floor surface detection sensor, 19L : Left wheel floor surface detection sensor, 19R: Right wheel floor surface detection sensor, 21: Power unit, 22L: Left drive wheel, 22R: Right drive wheel, 26: Rear wheel, 27: Front wheel, 31: Inlet, 32: Exhaust Mouth, 33: Electric blower, 34: Brush motor, 41: Image recording unit, 41A: Object recognition unit, 41C: Camera, 51: Device operation unit, 61: Device storage unit, 6 : Position information, 65: Tag database, 71: Position recording unit, 71L, 71R: Rotary encoder, 71a: Acceleration sensor, 71b: Gyro sensor, 81: Device communication unit, 90: Communication network, 100: Object search system, 110 : Communication terminal device, 111: terminal control unit, 112: terminal communication unit, 113: terminal display unit, 114: terminal operation unit, 115: terminal storage unit, 150: server device, 151: server control unit, 151a: tag addition , 151b: Search unit, 153: Server communication unit, 155: Server storage unit, 157: Tag database

Claims (6)

A system comprising a self-propelled electronic device that travels and performs work, and an operating device that communicates with the self-propelled electronic device,
The self-propelled electronic device is
An image recording unit that shoots at each traveled position and generates a recorded image;
A storage unit for storing the recorded image;
A tag adding unit for recognizing an object shown in the recorded image and tagging the recorded image with a keyword corresponding to the object;
A keyword inputted by the user, the recorded image-tagged associated with keywords as a query, and a search unit for extracting search from among the storage unit,
The operating device is:
A query input unit that accepts an input of a keyword as the query;
Object search system characterized in that it comprises a search result output unit for outputting the recorded image extracted by the search unit. The self-propelled electronic device further includes a position recording section for recording the position in which the self-propelled electronic device travels,
The storage unit stores the recorded image in association with a position where the recorded image is captured,
The object search system according to claim 1, wherein the search result output unit outputs the position associated with the recorded image together with the recorded image. The self-propelled electronic device further includes a time providing unit that provides a time when the recording image was taken,
The storage unit stores the recorded image in association with the time,
The object search system according to claim 1, wherein the search result output unit outputs the time associated with the recorded image together with the recorded image. The operating device further includes an input unit that accepts selection by the user of one or more recorded images extracted by the search unit,
The self-propelled electronic device, object search according to any one of claims 1 to 3, further comprising a running control unit for moving the self-propelled electronic device to that associated with the recording image selected position system.   A system comprising a self-propelled electronic device that runs and performs work, a server device that communicates with the self-propelled electronic device, and a communication terminal device that communicates with the server device,
  The self-propelled electronic device is
    An image recording unit that generates a recorded image by shooting at each traveled position,
  The server device is
    A storage unit for storing the recorded image;
    A tag adding unit for recognizing an object shown in the recorded image and tagging the recorded image with a keyword corresponding to the object;
    A search unit that searches for and extracts the recorded image, which is a keyword input by a user and to which a tag related to the keyword as a query is added, from the storage unit;
  The communication terminal device
    A query input unit that accepts an input of a keyword as the query;
    An object search system comprising: a search result output unit that outputs a recorded image extracted by the search unit. An object search device capable of communicating with a self-propelled electronic device that runs and performs work,
A communication unit that receives a recorded image taken by the self-propelled electronic device from the self-propelled electronic device; and
A storage unit for storing the received recorded image;
A tag adding unit for recognizing an object shown in the recorded image and tagging the recorded image with a keyword corresponding to the object;
An object retrieval apparatus comprising: a retrieval unit that retrieves and extracts the recorded image tagged with a designated keyword from the storage unit.