KR102543742B1 - Mobile terminal and computer readable recording medium recording operating method of mobile terminal - Google Patents

Abstract

A mobile terminal according to an embodiment of the present invention extracts a robot cleaner and a camera for acquiring an image of the robot cleaner's surroundings, a display displaying the acquired image, and a plurality of candidate markers from the displayed image, and extracts the plurality of extracted markers. Positional information of each of the candidate markers is calculated, and based on the calculated positional information and preset motion information of the robot cleaner, one of the plurality of candidate markers is selected as an effective marker corresponding to the marker attached to the robot cleaner. You can include a controller that acquires as a marker.

Description

A computer-readable recording medium recording a mobile terminal and a method of operating the mobile terminal

The present invention relates to a mobile terminal, and more particularly, to a mobile terminal for detecting a marker attached to a robot cleaner.

In general, in order to calculate the 3D position and posture of the subject, the image of the subject itself or a marker attached/printed on the subject is extracted/recognized by a vision system.

To this end, the marker of the subject needs a characteristic shape that is easy to distinguish from the surrounding environment, which acts as a design limiting factor.

Moreover, the vision system is inherently very sensitive to ambient lighting environments such as artificial light, natural light, and reflected light, and thus, it is difficult to overcome various types of noise.

A related prior patent (registered patent publication KR 10-1575597) detects a marker depending only on image features in an acquired image. However, this method has a disadvantage in that it is difficult to detect effective markers when a plurality of regions having characteristics similar to actual markers exist in the acquired image.

For example, assuming that an LED without any particular pattern (just bright and small circular) is used as a marker, natural light from a window, artificial light/light source such as a fluorescent lamp/stand/TV, robot vacuum cleaner/floor/floor/light source, etc. Various types of reflected light falling on surrounding objects act as serious noise in LED detection.

An object of the present invention is to detect a marker attached to a robot cleaner so as to be less sensitive to the influence of ambient noise.

An object of the present invention is to clearly distinguish a marker attached to a robot cleaner from other noise elements.

An object of the present invention is to filter noise elements having similar image characteristics to actual markers of a robot cleaner.

An object of the present invention is to more accurately detect a marker of a robot cleaner.

An object of the present invention is to provide motion information of various robot cleaners in order to detect valid markers.

A mobile terminal according to an embodiment of the present invention may obtain a valid marker from among a plurality of marker candidates by using location information of each of the plurality of marker candidates and preset motion information of the robot cleaner.

A mobile terminal according to an embodiment of the present invention may detect a valid marker by comparing a position value of each of a plurality of marker candidates with preset motion information.

A mobile terminal according to an embodiment of the present invention may determine a corresponding candidate marker as a noise element when a change in position value of each of a plurality of marker candidates does not coincide with preset motion information.

The mobile terminal according to an embodiment of the present invention may determine the candidate marker as a valid marker when the position value of the candidate marker matches preset motion information a certain number of times or more through consecutive image frames.

A mobile terminal according to an embodiment of the present invention may display a motion list providing various motions of the robot cleaner.

A further scope of the applicability of the present invention will become apparent from the detailed description that follows. However, since various changes and modifications within the spirit and scope of the present invention can be clearly understood by those skilled in the art, it should be understood that the detailed description and specific examples such as preferred embodiments of the present invention are given as examples only.

According to an embodiment of the present invention, it is possible to overcome the design constraints of the marker attached to the robot cleaner and detect an effective marker without a significant effect on noise in the surrounding environment.

According to an embodiment of the present invention, candidate markers having characteristics similar to valid markers can be more clearly distinguished.

According to an embodiment of the present invention, candidate markers having characteristics similar to effective markers can be more accurately filtered.

According to an embodiment of the present invention, there is an effect of more accurately detecting an effective marker through successive image frames.

According to an embodiment of the present invention, a user can set motions of a robot cleaner in various ways to more accurately detect a marker.

1 is a block diagram for explaining a mobile terminal related to the present invention.
2 is a block diagram for explaining the configuration of a robot cleaner according to an embodiment of the present invention.
3 is a flowchart illustrating a method of operating a mobile terminal according to an embodiment of the present invention.
4 is a diagram for explaining a process of extracting an effective marker from among a plurality of candidate markers when a preset motion is linear movement according to an embodiment of the present invention.
5 to 8 explain a process of extracting effective markers in more detail using a 3D coordinate system according to an embodiment of the present invention.
9 is a diagram for explaining an example of program coding for detecting an effective marker when a preset motion of a robot cleaner is rotational movement in place according to an embodiment of the present invention.
10A to 10D are diagrams for explaining a scenario in which a process of recognizing a marker attached to a robot cleaner through a mobile terminal according to an embodiment of the present invention is provided.

Hereinafter, the embodiments disclosed in this specification will be described in detail with reference to the accompanying drawings, but the same or similar elements are given the same reference numerals regardless of reference numerals, and redundant description thereof will be omitted. The suffixes "module" and "unit" for components used in the following description are given or used together in consideration of ease of writing the specification, and do not have meanings or roles that are distinct from each other by themselves.

Mobile terminals described in this specification include mobile phones, smart phones, laptop computers, digital broadcasting terminals, personal digital assistants (PDAs), portable multimedia players (PMPs), navigation devices, and slate PCs. , tablet PC, ultrabook, wearable device (eg, watch type terminal (smartwatch), glass type terminal (smart glass), HMD (head mounted display)), etc. may be included there is.

However, those skilled in the art can easily recognize that the configurations according to the embodiments described in this specification may be applied to fixed terminals such as digital TVs, desktop computers, digital signage, etc., except when applicable only to mobile terminals. will be.

1 is a block diagram for explaining a mobile terminal related to the present invention.

Referring to FIG. 1 , a mobile terminal 100 may include a camera 110 , a display 130 , a wireless communication unit 150 , a memory 170 and a controller 190 .

The components shown in FIG. 1 are not essential to implement a mobile terminal, so the mobile terminal described herein may have more or fewer components than those listed above.

The camera 110 may be located on the opposite side of the front of the mobile terminal 100 displaying the screen.

The camera 110 may acquire images of the robot cleaner and its surroundings.

The camera 110 includes at least one of a camera sensor (eg, CCD, CMOS, etc.), a photo sensor (or image sensor), and a laser sensor.

The camera 110 and the laser sensor may be combined with each other to sense a touch of a sensing target for a 3D stereoscopic image.

A photo sensor may be stacked on a display device, and the photo sensor is configured to scan a motion of a sensing target close to the touch screen. More specifically, the photo sensor scans the content placed on the photo sensor by mounting photo diodes and transistors (TRs) in rows/columns and using electrical signals that change according to the amount of light applied to the photo diodes. That is, the photo sensor calculates the coordinates of the sensing object according to the change in light, and through this, the location information of the sensing object can be obtained.

The display unit 130 may implement a touch screen by forming a mutual layer structure or integrally with the touch sensor. Such a touch screen may function as a user input unit providing an input interface between the mobile terminal 100 and the user, and may provide an output interface between the mobile terminal 100 and the user.

The display unit 130 displays (outputs) information processed by the mobile terminal 100 . For example, the display unit 130 may display execution screen information of an application program driven in the mobile terminal 100 or UI (User Interface) and GUI (Graphic User Interface) information according to such execution screen information. .

The wireless communication unit 150 may perform wireless communication with the robot cleaner 200 to be described later. To this end, the wireless communication unit 150 may include a short-range communication module.

The short-range communication module is for short-range communication, and includes Bluetooth™, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, and Near Field Communication (NFC). Communication), Wi-Fi (Wireless-Fidelity), Wi-Fi Direct, and wireless USB (Wireless Universal Serial Bus) technologies may be used to support short-distance communication.

Such a short-range communication module may be used between the mobile terminal 100 and a wireless communication system, between the mobile terminal 100 and another mobile terminal 100, or between the mobile terminal 100 and another mobile terminal 100 through wireless area networks. Wireless communication between networks in which the mobile terminal 100 (or an external server) is located may be supported. The local area wireless communication network may be a local area wireless personal area network (Wireless Personal Area Networks).

The wireless communication unit 150 may transmit a control command for controlling the operation of the robot cleaner 200 to the robot cleaner 200 .

The memory 170 stores data supporting various functions of the mobile terminal 100 . The memory 170 may store a plurality of application programs (application programs or applications) running in the mobile terminal 100 , data for operation of the mobile terminal 100 , and commands.

At least some of these application programs may be downloaded from an external server through wireless communication.

In addition, at least some of these application programs may exist on the mobile terminal 100 from the time of shipment for the basic functions of the mobile terminal 100 (eg, incoming and outgoing calls, outgoing functions, message receiving, and outgoing functions).

Meanwhile, the application program may be stored in the memory 170, installed on the mobile terminal 100, and driven by the controller 190 to perform an operation (or function) of the mobile terminal.

The memory 170 may store programs for operation of the controller 190 and may temporarily store input/output data (eg, phonebook, message, still image, video, etc.). The memory 170 may store data related to vibration and sound of various patterns output when a touch is input on the touch screen.

The memory 170 may be a flash memory type, a hard disk type, a solid state disk type, a silicon disk drive type, or a multimedia card micro type. ), card-type memory (eg SD or XD memory, etc.), RAM (random access memory; RAM), SRAM (static random access memory), ROM (read-only memory; ROM), EEPROM (electrically erasable programmable read -only memory), a programmable read-only memory (PROM), a magnetic memory, a magnetic disk, and an optical disk. The mobile terminal 100 may be operated in relation to a web storage that performs a storage function of the memory 170 on the Internet.

The controller 190 controls general operations of the mobile terminal 100 in addition to operations related to the application program. The controller 190 may provide or process appropriate information or functions to a user by processing signals, data, information, etc. input or output through the components described above or by running an application program stored in the memory 170.

In addition, the controller 190 may control at least some of the components reviewed in conjunction with FIG. 1 in order to drive an application program stored in the memory 170 . Furthermore, the controller 190 may combine and operate at least two or more of the components included in the mobile terminal 100 to drive the application program.

Meanwhile, the controller 190 may perform different controls or the same control according to the type of the touch object that touches the touch screen (or a touch key provided other than the touch screen). Whether to perform different controls or the same control according to the type of the touch object may be determined according to an operating state of the mobile terminal 100 or an application program currently being executed.

Meanwhile, as described above, the controller 190 controls operations related to applications and general operations of the mobile terminal 100 . For example, if the state of the mobile terminal satisfies a set condition, the controller 190 may execute or release a locked state that restricts a user's control command input to applications.

2 is a block diagram for explaining the configuration of a robot cleaner according to an embodiment of the present invention.

Referring to FIG. 2 , the robot cleaner 200 according to an embodiment of the present invention may include an obstacle detecting unit 210, a wireless communication unit 250, a driving driving unit 270, and a controller 290.

The obstacle detecting unit 210 may detect obstacles disposed around the robot cleaner 200 .

The obstacle detecting unit 210 may include an ultrasonic sensor, an infrared sensor, a laser sensor, and/or a vision camera. The obstacle detecting unit 210 may include a camera that acquires an image by capturing a cleaning area irradiated with a laser by a laser sensor. In this case, a laser light pattern is extracted from an image acquired through a camera, and based on changes in the position, shape, posture, etc. of the pattern in the image, an obstacle situation in the cleaning area and a map of the cleaning area may be generated. .

The wireless communication unit 250 may communicate with the wireless communication unit 150 of the mobile terminal 100 wirelessly. A description of the wireless communication unit 250 is replaced with a description of the wireless communication unit 150 of FIG. 1 .

The driving driving unit 270 may include a left wheel driving unit 271 driving the left wheel of the robot cleaner 200 and a right wheel driving unit 273 driving the right wheel.

The controller 290 may control the overall operation of the robot cleaner 200 .

The controller 290 can independently control the operation of the left wheel driving unit 271 and the right wheel driving unit 273 . Accordingly, the robot cleaner 200 moves straight, backward, or turns.

3 is a flowchart illustrating a method of operating a mobile terminal according to an embodiment of the present invention.

Hereinafter, a method of operating a mobile terminal according to the embodiment of FIG. 3 will be described in connection with the embodiments of FIGS. 1 and 2 .

In addition, the following description will be made on the assumption that the number of markers attached to the robot cleaner 200 is one.

In addition, in the following embodiments, the robot cleaner 200 is described as an example, but it is not necessary to be limited thereto, and any type of electronic device that can be controlled to move by itself can be applied.

Referring to FIG. 3 , the camera 110 of the mobile terminal 100 obtains an image (S301). The acquired image may include an image of the robot cleaner and an image of the robot cleaner's surroundings.

The camera 110 of the mobile terminal 100 may be disposed on the back of the display 130 displaying the screen.

The pre-processing unit 191 of the mobile terminal 100 pre-processes the acquired image of the robot cleaner (S303). In one embodiment, the pre-processing unit 191 may adjust the brightness of an image acquired through the camera 110 . Also, the pre-processing unit 191 may remove noise from the acquired image. In addition, the pre-processing unit 191 may perform an image improvement task such as color correction of the obtained image.

The preprocessed image may be displayed through the display 130 of the mobile terminal 100 .

The candidate marker extraction unit 193 of the mobile terminal 100 extracts a plurality of candidate markers from the preprocessed image of the robot cleaner (S205).

The candidate marker extractor 193 may extract a plurality of candidate markers based on predetermined marker information. The preset marker information may include a shape of the marker and a color of the marker.

Pre-set marker information may be stored in the memory 170 .

In an embodiment, the candidate marker extractor 193 may extract candidate markers corresponding to predetermined marker information from an image displayed on the display 130 .

For example, when a marker included in the preset marker information has a circular shape and a red color, the candidate marker extractor 193 selects a candidate having a circular shape and a red color from the preprocessed image. markers can be extracted.

That is, the candidate marker extractor 193 first obtains features having a circular shape from the preprocessed image, and then extracts a feature having a red color as a marker from the obtained features.

The location information calculation unit 195 of the mobile terminal 100 calculates location information of each of the extracted candidate markers (S207).

In an embodiment, the location information calculation unit 195 may calculate a 3D location value of each of the plurality of extracted candidate markers.

The location information calculating unit 195 may calculate a 3D location value of each of the plurality of candidate markers based on the reference location. Here, the reference location may be a location where the mobile terminal 100 is located.

The valid marker acquisition unit 197 of the mobile terminal 100 acquires one or more valid markers among a plurality of candidate markers based on the calculated location information and preset motion information of the robot cleaner 200 (S209).

The user may control the operation of the robot cleaner 200 through the mobile terminal 100 so that the robot cleaner 200 takes a preset motion. The controller 190 of the mobile terminal 100 may receive a user input. The controller 190 may transmit a control command corresponding to a user input to the robot cleaner 200 through the wireless communication unit 150 .

The robot cleaner 200 may take a preset motion corresponding to the control command according to the control command received from the mobile terminal 100 .

In an embodiment, the preset motion information of the robot cleaner 200 may include information about the motion of the robot cleaner 200, such as a moving direction and a moving distance of the robot cleaner 200.

In one embodiment, the motion of the robot cleaner 200 may be rotational movement of the robot cleaner 200 in place. The in-place rotation movement may be a movement in which the robot cleaner 200 rotates in place around a point.

In another embodiment, the motion of the robot cleaner 200 may be a linear movement of the robot cleaner 200 . Linear movement may be movement in which the robot cleaner 200 draws a straight line from a first point to a second point.

In another embodiment, the motion of the robot cleaner 200 may be a curved movement of the robot cleaner 200 . The curved movement may be a movement in which the robot cleaner 200 draws a curve from a first point to a second point.

A process of acquiring one or more valid markers among a plurality of candidate markers will be described with reference to the following drawings.

4 is a diagram for explaining a process of extracting an effective marker from among a plurality of candidate markers when a preset motion is linear movement according to an embodiment of the present invention.

In FIG. 4 , it is assumed that the preset motion of the robot cleaner 200 is a linear movement from a first point to a second point while drawing a straight line.

Also, it is assumed that the user is photographing the robot cleaner 200 and its surroundings while the mobile terminal 100 is fixed.

In addition, it is assumed that the number of candidate markers extracted in FIG. 4 is 5, but this is only an example.

Referring to FIG. 4 , a plurality of image frames 400_1 to 400_n acquired through the camera 110 of the mobile terminal 100 are illustrated.

Each of the plurality of image frames 400_1 to 400_n may include a plurality of candidate markers 411a to 411e.

The valid marker acquisition unit 197 of the mobile terminal 100 may extract valid markers based on the position values of each of the plurality of candidate markers 411a to 411e and pre-stored motion information of the robot cleaner 200. .

The valid marker acquisition unit 197 may extract valid markers matching preset motion information from among the plurality of candidate markers 411a to 411e.

Specifically, the effective marker acquisition unit 197 may compare the position value of each candidate marker included in the first image frame 400_1 with the position value of each candidate marker included in the second image frame 400_2.

As a result of the comparison, the effective marker acquisition unit 197 may determine candidate markers 411a to 411d having the same position value as noise markers to be filtered. A noise marker may be a noise component that needs to be filtered out.

As a result of the comparison, the valid marker acquisition unit 197 may determine the corresponding candidate marker 411e as a valid marker when the position value is changed to a value consistent with linear movement.

That is, the valid marker acquisition unit 197 determines the position value of the first candidate marker 411a included in the second image frame 400_2 and the position of the second candidate marker 411e included in the first image frame 400_1. When the value is not changed by a specific value corresponding to linear movement, the first candidate marker 411a may be extracted as a noise marker.

The valid marker acquisition unit 197 determines the position value of the second candidate marker 411e included in the second image frame 400_2 to the position value of the second candidate marker 411e included in the first image frame 400_1. When the value is changed by a specific value corresponding to linear movement, the second candidate marker 411e may be extracted as an effective marker.

Since the mobile terminal 100 acquires image frames in a fixed state, position values of candidate markers other than effective markers in each image frame will be fixed. The remaining candidate markers may be environmental noise components to be filtered out.

That is, position values of candidate markers having image characteristics similar to those of markers attached to the actual robot cleaner 200 will not change as the robot cleaner 200 linearly moves.

Even if the position values of the candidate markers change, if they do not change to specific values corresponding to motion information of the robot cleaner 200 set in advance, the candidate markers may be recognized as noise markers.

In an embodiment, the valid marker acquisition unit 197 may determine the second candidate marker 411e as a final valid marker when the second candidate marker 411e is extracted as a valid marker a predetermined number of times or more.

For example, when the number of image frames shown in FIG. 4 is n and a valid marker extraction process is performed a total of n−1 times according to position value comparison, the second candidate marker 411e is selected as an effective marker five or more times. When extracted, the second candidate marker 411e can be recognized as the last valid marker.

In this way, when an effective marker is extracted using a plurality of image frames instead of two image frames, a marker attached to the robot cleaner 200 can be more accurately extracted.

Figure 3 will be described again.

The controller 190 of the mobile terminal 100 determines whether the acquired valid markers are two or more (S211).

In one embodiment, as a result of performing steps S201 to S209, two or more effective markers may be extracted.

If there are two or more acquired valid markers, the controller 190 of the mobile terminal 100 stores the location information of the two or more valid markers in the memory 170 (S213).

The mobile terminal 100 may perform step S201 again after storing the location information of two or more valid markers.

Meanwhile, when two or more valid markers are extracted, the controller 190 of the mobile terminal 100 may change preset motion information into other motion information in step S309.

Specifically, when the controller 190 of the mobile terminal 100 sets the motion of the robot cleaner 200 as a linear movement in the process of extracting two effective markers, the robot cleaner 200 in the process of extracting the next effective marker ) can be changed to rotational movement in place.

That is, when a plurality of valid markers are extracted, the controller 190 may change preset first motion information into preset second motion information. The controller 190 may acquire an effective marker corresponding to a position matching the changed second motion information among a plurality of valid markers as a final valid marker.

Accordingly, a process of extracting one valid marker from among a plurality of valid markers can be performed more accurately through new motion information.

If the obtained valid marker is one, the controller 190 of the mobile terminal 100 controls the operation of the robot cleaner based on the acquired one valid marker (S215).

In one embodiment, the mobile terminal 100 may control driving of the robot cleaner 200 based on the obtained effective marker.

Next, a process of extracting valid markers according to an embodiment of the present invention will be described in detail.

5 to 8 explain a process of extracting effective markers in more detail using a 3D coordinate system according to an embodiment of the present invention.

In FIG. 5 , it is assumed that the position of the mobile terminal 100 is fixed and the preset motion of the robot cleaner 200 is rotational movement in place.

Referring to FIG. 5 , the mobile terminal 100 may acquire an image of the robot cleaner 200 through the camera 110 at a fixed location.

The coordinate system corresponding to the preset motion of the robot cleaner 200 is an X1-Y1 coordinate system (first coordinate system) based on the position of the robot cleaner 200, and the position of the camera 110 of the mobile terminal 100 as a reference. The coordinate system to be may be an X2-Y2 coordinate system (second coordinate system).

The first coordinate system and the second coordinate system may have a Z-axis in a vertical direction from the ground in common.

Referring to FIG. 6 , a first coordinate system based on the position of the robot cleaner 200 is shown. The movement of the effective marker 600 is shown on the first coordinate system. That is, the effective marker 600 is located at a first location (A) on the first image frame and at a second location (B) on the second image frame. The second image frame may be a frame obtained after the first image frame.

An arrow extending from the center of the effective marker 600 may indicate a movement direction of the robot cleaner 200 based on rotational movement of the robot cleaner 200 in place.

The effective marker 600 may be moved based on preset motion information (rotation in place). On the first coordinate system, a state in which the effective marker 600 moves based on the rotational movement in place is shown.

The effective marker acquisition unit 197 may obtain a first coordinate value corresponding to the first position A and a second coordinate value corresponding to the second position B of the effective marker 600 on the first coordinate system. .

Similarly, referring to FIG. 7 , a second coordinate system based on the location of the mobile terminal 100 is shown. The movement of the effective marker 600 based on the location of the mobile terminal 100 is shown on the second coordinate system. The valid marker 600 is located at a third location (C) on the first image frame and at a fourth location (D) on the second image frame.

On the second coordinate system, a state in which the effective marker 600 moves based on the rotational movement in place is shown.

The valid marker acquisition unit 197 may obtain a third coordinate value corresponding to the third position C and a fourth coordinate value corresponding to the fourth position D of the effective marker 600 on the second coordinate system. .

If the first and second coordinate systems are displayed on the same plane, they can be expressed as shown in FIG. 8 .

The valid marker acquisition unit 197 may perform coordinate system transformation to calculate a position value of each candidate marker for each of the image frames that are successively acquired.

The valid marker acquisition unit 197 may translate the coordinate system by a coordinate displacement value between the first and second coordinate systems based on the valid marker 600 on the second image frame.

Also, the valid marker acquisition unit 197 may rotate and transform the coordinate system by an angular displacement value in the direction between the first and second coordinate systems based on the effective marker 600 on the second image frame.

The effective marker acquisition unit 197 matches the two coordinate systems according to translational translation and rotation transformation and compares the coordinate values of the effective marker 600 in the first image frame with the coordinate values of the effective marker 600 in the second image frame. can

The effective marker acquisition unit 197 may determine whether the effective marker 600 is a marker matching preset motion information of the robot cleaner 200 according to the comparison result.

When the difference between the coordinate values of the effective marker 600 on the first image frame and the coordinate value of the effective marker 600 on the second image frame is a specific value corresponding to preset motion information in a state in which the two coordinate systems are matched, The valid marker acquisition unit 197 may acquire the corresponding valid marker 600 as a final valid marker.

In a state in which the two coordinate systems are matched, the difference between the coordinate values of the effective marker 600 on the first image frame and the coordinate value of the effective marker 600 on the second image frame is not a specific value corresponding to preset motion information. In this case, the valid marker acquisition unit 197 may determine the valid marker 600 as a noise marker.

Hereinafter, when the preset motion of the robot cleaner 200 is rotation in place, a process of detecting an effective marker will be described through mathematical formulas and program coding.

9 is a diagram for explaining an example of program coding for detecting an effective marker when a preset motion of a robot cleaner is rotational movement in place according to an embodiment of the present invention.

In order to detect an effective marker, Equations 1 and 2 below are also required.

[Equation 1]

[Equation 2]

In FIG. 9, N represents the number of consecutive frames to be used for valid marker detection.

In [Equation 1], Ti represents the time at which an image frame preceding the ith from the current image frame of the obtained image is acquired.

In [Equation 1], Pi,j represents the posture of the j-th candidate marker existing in the image frame before the i-th from the current image frame of the acquired image. In particular, the posture of the candidate marker may represent the direction angle toward which the candidate marker is directed.

In [Equation 1], Vr represents a rotational angular velocity of a preset rotation in place.

In [Equation 1], ei,j may indicate a predicted direction angle that a j-th candidate marker of the current image frame will have in an image frame previous to the i-th.

In FIG. 9 , m may be a critical angle for determining whether the predicted direction angle is within a preset range. The critical angle may be less than 0, but this is merely an exemplary value.

In [Equation 2], c may represent the number of counts per image frame when the predicted direction angle is within the critical angle. For example, c may be 5, but this is only an exemplary value. C may be equal to N-1 or less than N-1.

Since the total number of image frames is N, the maximum value that C can have may be N-1. When C is N-1, the corresponding candidate marker can be determined as the most accurate effective marker.

10A to 10D are diagrams for explaining a scenario in which a process of recognizing a marker attached to a robot cleaner through a mobile terminal according to an embodiment of the present invention is provided.

Referring to FIG. 10A , the display 130 of the mobile terminal 100 may display a motion list 1010 including preset motion items before extracting valid markers of the robot cleaner 200 . The motion list 1010 may include a plurality of motion items 1011 to 1015 .

The motion list 1010 may be a list for setting motions of the robot cleaner 200 in order to extract valid markers attached to the robot cleaner 200 .

The first motion item 1011 may be an item corresponding to rotational movement in place.

The second motion item 1013 may be an item corresponding to linear movement.

The third motion item 1015 may be an item corresponding to a curved movement.

When any one of the plurality of motion items 1011 to 1015 is selected, the mobile terminal 100 may transmit motion information corresponding to the selected item to the robot cleaner 200 .

The robot cleaner 200 may operate to take a specific motion based on motion information received from the mobile terminal 100 .

For example, when the first motion item 1011 is selected, the mobile terminal 100 may transmit a control command requesting rotation in place to the robot cleaner 200 . The robot cleaner 200 may perform a motion of rotating in place according to the received control command.

Meanwhile, after the first motion item 1011 is selected, the mobile terminal 100 may activate the camera 110 to display an image 1030 of the robot cleaner obtained from the camera 110 .

The image 1030 of the robot cleaner may include an actual marker image 1031 .

Thereafter, the mobile terminal 100 may obtain an image 1300 of the robot cleaner performing a rotational movement in place. At this time, the mobile terminal 100 may display a pop-up window 1040 requesting fixing the position of the camera through the display 130 .

This is because, when the position of the camera is moved, accurate detection of an effective marker may be hindered.

According to the embodiments of FIGS. 3 to 9 , when a final valid marker is obtained, the mobile terminal 100 may display a pop-up window 1050 indicating that the marker has been recognized through the display 130 .

That is, when one valid marker is detected among a plurality of candidate markers, the mobile terminal 100 can display the pop-up window 1050 .

After that, the mobile terminal 100 may display operation control buttons 1071 and 1073 for controlling the operation of the robot cleaner 200 .

The user can move the robot cleaner 200 or clean a desired place through the operation control buttons 1071 and 1073.

The above-described present invention can be implemented as computer readable code on a medium on which a program is recorded. The computer-readable medium includes all types of recording devices in which data that can be read by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. there is

The above detailed description should not be construed as limiting in all respects and should be considered illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (13)

In the mobile terminal,
A camera for acquiring images of the robot cleaner and its surroundings;
a display displaying the obtained image; and
Extracting a plurality of candidate markers from the displayed image;
Calculating location information of each of the plurality of extracted candidate markers;
A controller for obtaining one of the plurality of candidate markers as an effective marker corresponding to a marker attached to the robot cleaner based on the calculated location information and preset motion information of the robot cleaner;
The preset motion information of the robot cleaner
Including the moving direction and moving distance of the robot cleaner
mobile terminal. According to claim 1,
The controller
calculating position values of the plurality of candidate markers included in each of a plurality of image frames;
Acquiring a corresponding candidate marker as the valid marker when the position value of each of the plurality of candidate markers is changed to a value corresponding to the preset motion information.
mobile terminal. According to claim 2,
The controller
When the position value of each of the plurality of candidate markers is not changed to a value corresponding to the preset motion information, determining the corresponding candidate marker as a noise marker to be filtered
mobile terminal. According to claim 2,
The controller
When the number of the plurality of image frames is N, and the position value of each of the plurality of candidate markers is changed to a value corresponding to the preset motion information more than a preset number of times, the corresponding candidate marker is obtained as the valid marker. doing
mobile terminal. According to claim 1,
When two or more effective markers are extracted from among the plurality of candidate markers, further comprising a memory for storing positional information of each of the extracted two or more effective markers
mobile terminal. According to claim 1,
The controller
Displaying a motion list including the plurality of motion items through the display;
Further comprising a wireless communication unit for transmitting a control command for controlling the motion of the robot cleaner with a motion corresponding to a selected item among the plurality of motion items
mobile terminal. According to claim 1,
The controller
Transmitting a control command for controlling an operation of the robot cleaner based on the obtained valid marker
mobile terminal. In a recording medium on which a program for performing a method of operating a mobile terminal is recorded,
The operating method of the mobile terminal is
Obtaining an image of a robot cleaner and its surroundings;
displaying the obtained image;
extracting a plurality of candidate markers from the displayed image;
calculating location information of each of the plurality of extracted candidate markers;
Acquiring one of the plurality of candidate markers as an effective marker corresponding to a marker attached to the robot cleaner based on the calculated location information and preset motion information of the robot cleaner;
The preset motion information of the robot cleaner
Including the moving direction and moving distance of the robot cleaner
recording medium. According to claim 8,
Calculating the location information
Calculating position values of the plurality of candidate markers included in each of a plurality of image frames;
Obtaining the effective marker
Acquiring a corresponding candidate marker as the valid marker when the position value of each of the plurality of candidate markers is changed to a value corresponding to the preset motion information.
recording medium. According to claim 9,
Further comprising determining the corresponding candidate marker as a noise marker to be filtered when the position value of each of the plurality of candidate markers is not changed to a value corresponding to the preset motion information.
recording medium. According to claim 9,
Obtaining the effective marker
When the number of the plurality of image frames is N, and the position value of each of the plurality of candidate markers is changed to a value corresponding to the preset motion information more than a preset number of times, the corresponding candidate marker is obtained as the valid marker. including the steps of
recording medium. According to claim 8,
When two or more effective markers are extracted from among the plurality of candidate markers, storing positional information of each of the two or more effective markers extracted
recording medium. According to claim 8,
displaying a motion list including the plurality of motion items through a display of the mobile terminal; and
Transmitting a control command for controlling the motion of the robot cleaner with a motion corresponding to a selected item among the plurality of motion items
recording medium.

Images