KR20230126495A - Method, system and non-transitory computer-readable recording medium for generating a map of controlled object - Google Patents

Abstract

The present disclosure relates to a method for generating a map of a control target object, an apparatus, and a non-transitory computer readable recording medium. A method for generating a map of a control target object according to an embodiment of the present disclosure includes receiving at least one image including a subject from at least one control target object; detecting an area image of a subject in each of the at least one image; generating posture coordinate information by analyzing the region image; generating mapped coordinate information by mapping at least one control target object to posture coordinate information; and generating direction coordinate information by converting the mapped coordinate information based on the subject; includes

Description

Method, apparatus, and non-temporary computer readable recording medium for generating a map of a control object

The present disclosure relates to a method, apparatus, and non-transitory computer readable recording medium for generating a map of a control object, and more particularly, to generate a map of a control object by determining a location of a control object based on a user. It relates to a method, apparatus, and non-transitory computer readable recording medium.

Recently, as interest in the Internet of Things increases and development of related technologies is actively progressed, technologies for controlling devices or objects based on the Internet of Things technology have been introduced. In particular, technologies for supporting control of devices, objects, and the like based on user's motion or manipulation in various use environments such as mobile devices, tablets, laptops, PCs, home appliances, and automobiles have been introduced.

In order to support controlling an object according to its physical location in an IoT environment with a plurality of controllable objects, a technique for measuring the location of an object and generating a digital twin map based on the measured location of the object is known. For example, a technique of scanning a space by moving a 2D camera or a 3D camera and detecting the position of an object together in the process of scanning the space is known. However, this method requires a lot of effort in the process of scanning the space, and since it is impossible to detect the location of the object in real time, there is a problem in that an error occurs when the location of the object is changed after scanning. In addition, a technique for detecting the location of an object by generating inaudible band sounds (eg, ultrasonic waves) at various locations and estimating a distance from the current location is known. However, this method has a problem in that the position of the signal generator must be accurately set, and an error occurs when the position of the signal generator is changed.

Therefore, there is still a need to develop a technology that can improve user convenience in controlling objects by detecting the physical locations of objects in a simple and accurate way and generating a digital twin map of objects based on this. am.

An object of the present disclosure is to provide a method and apparatus for generating a map of a control target object based on a relative location of the control target object, in order to solve the above-described problems of the prior art.

In addition, an object of the present disclosure is to provide a method and apparatus for generating a 3D control target object map by detecting a position of the control target object on a reference space.

A method for generating a map of a control target object according to an embodiment of the present disclosure includes receiving at least one image including a subject from at least one control target object; detecting an area image of a subject in each of the at least one image; generating posture coordinate information by analyzing the region image; generating mapped coordinate information by mapping at least one control target object to posture coordinate information; and generating direction coordinate information by converting the mapped coordinate information based on the subject; includes

According to an embodiment of the present disclosure, the generating of the attitude coordinate information may be performed by detecting yaw and pitch angles of the area image.

According to an embodiment of the present disclosure, the generating of mapped coordinate information may include matching a region image with a control target object obtained by capturing a corresponding region image.

According to an embodiment of the present disclosure, the generating of directional coordinate information may be performed by converting posture coordinate information according to yaw and pitch angles into directional coordinate information according to up, down, left, and right directions.

A method of generating a map of a control target object according to an embodiment of the present disclosure includes receiving at least one image including a first subject and a second subject from at least one control target object; detecting a first area image of a first subject and a second area image of a second subject from each of the at least one image; generating first posture coordinate information and second posture coordinate information by analyzing the first area image and the second area image; generating first mapped coordinate information and second mapped coordinate information by mapping at least one control target object to first posture coordinate information and second posture coordinate information, respectively; generating first direction coordinate information and second direction coordinate information by converting first mapped coordinate information and second mapped coordinate information based on the first subject and the second subject, respectively; Based on the first direction coordinate information, first half line information is generated by connecting each of at least one control target object with a half line based on the first subject based on the first direction coordinate information, and at least one control target object based on the second subject based on the second direction coordinate information. generating second half line information by connecting each of the control target objects in a half line; and determining a location of each of the at least one control target object based on the first and second linear information.

According to an embodiment of the present disclosure, the step of determining the position of the control target object may include determining an intersection point where a ray path of first ray information and a ray path of second linear information for each of the at least one control target object intersect with each other. determining the position of the object; or determining a point of a shortest connecting line between a ray path of the first ray information and a ray path of the second linear information for each of the at least one control target object as the position of each of the at least one control target object. there is.

According to an embodiment of the present disclosure, determining the location of the object to be controlled may include determining the posture of each of the at least one object to be controlled. Here, in the step of determining the posture of the object to be controlled, the posture of each of the at least one object to be controlled may be determined from the posture of each camera module of the at least one object to be controlled.

According to one embodiment of the present disclosure, determining the position of the object to be controlled may include determining a roll angle of a first area of a first subject and a roll angle of a second area of a second subject; and determining a roll angle of at least one control target object. Here, in determining the roll angle of the first area of the first subject and the roll angle of the second area of the second subject, an average of effective roll angles excluding outliers may be determined as the roll angle of the area of the subject. The determining of the roll angle of the at least one control target object may include determining the roll angle of the at least one control target object based on the roll angle of the first region of the first subject; and determining a roll angle of at least one control target object based on a roll angle of a second region of a second subject.

An apparatus for generating a map of a control target object according to an embodiment of the present disclosure includes receiving at least one image including a subject from at least one control target object and detecting an image of a region of the subject in each of the at least one image. The image receiving unit analyzes the area image to generate posture coordinate information, maps at least one control target object to the posture coordinate information to generate mapped coordinate information, and converts the mapped coordinate information based on the subject to create a first direction It includes a coordinate information generation unit that generates coordinate information and a control target object location determiner that determines the position of each of the at least one control target object based on the direction coordinate information.

An apparatus for generating a map of a control target object according to an embodiment of the present disclosure receives at least one image including a first subject and a second subject from at least one control target object, and obtains a first object from each of the at least one image. an image receiver configured to detect a first area image of a subject and a second area image of a second subject, analyze the first area image and the second area image, and generate first posture coordinate information and second posture coordinate information; At least one control target object is mapped to the first posture coordinate information and the second posture coordinate, respectively, to generate first mapped coordinate information and second mapped coordinate information, and the first mapped coordinate information and the second mapped coordinate information are generated. Information is converted based on the first subject and the second subject, respectively, to generate first direction coordinate information and second direction coordinate information, and at least one control target object based on the first subject based on the first direction coordinate information First half line information is generated by connecting each of them with a half line, and coordinate information is generated to generate second half line information by connecting each of at least one control target object with a half line based on the second subject based on the second direction coordinate information. and a control target object location determiner for determining a position of each of the at least one control target object based on the second linear information and the first linear information.

In addition to this, another method for implementing the present disclosure, another device, and a non-transitory computer readable recording medium recording a computer program for executing the method are further provided.

According to an embodiment of the present disclosure, by analyzing an image including a subject received from a control object and generating coordinate information of the control object, a map of the control object according to a physically recognized direction and location based on the subject can create

According to an embodiment of the present disclosure, the location of the control object may be specified as 3D coordinates by generating coordinate information of the control object based on two or more subjects.

1 is a diagram schematically illustrating a system environment for controlling a control target object based on a control target object map according to an embodiment of the present disclosure.
2 is a functional block diagram schematically illustrating a functional configuration of an apparatus for generating a control target object map according to an embodiment of the present disclosure.
3 is a diagram illustratively illustrating a state in which a subject is facing a specific control target object in a predetermined space in which a plurality of control target objects are located according to an embodiment of the present disclosure.
4 is a diagram exemplarily illustrating a method of generating posture coordinate information by analyzing a region image of a subject according to an embodiment of the present disclosure.
5 is a diagram exemplarily illustrating a method of generating direction coordinate information according to an embodiment of the present disclosure.
6 is an operational flowchart exemplarily illustrating a method of generating a map of a control target object according to an embodiment of the present disclosure.
7 is a diagram illustratively showing a state in which a first subject and a second subject are present in a predetermined space in which a plurality of control object objects are located according to another embodiment of the present disclosure.
8 is a diagram exemplarily illustrating a method of generating posture coordinate information by analyzing a region image according to another embodiment of the present disclosure.
9 is a diagram exemplarily illustrating a method of generating half-line information according to another embodiment of the present disclosure.
10 is a diagram exemplarily illustrating a method of determining a location of each control target object according to another embodiment of the present disclosure.
11 and 12 exemplarily illustrate a method of correcting a roll angle of an image of a region of a subject according to another embodiment of the present disclosure.
13 is an operational flowchart exemplarily illustrating a method of generating a map of a control target object according to another embodiment of the present disclosure.

Hereinafter, embodiments of the present disclosure will be described in detail with reference to the accompanying drawings. Hereinafter, if it is determined that there is a risk of unnecessarily obscuring the gist of the present disclosure, detailed descriptions of already known functions and configurations will be omitted. In addition, it should be understood that the contents described below relate only to one embodiment of the present disclosure, and the present disclosure is not limited thereto.

Terms used in the present disclosure are used only to describe specific embodiments and are not intended to limit the present disclosure. For example, a component expressed in the singular number should be understood as a concept including a plurality of components unless the context clearly means only the singular number. It should be understood that the term "and/or" as used in this disclosure encompasses any and all possible combinations of one or more of the listed items. Terms such as 'include' or 'have' used in the present disclosure are intended to designate that the features, numbers, steps, operations, components, parts, or combinations thereof described in the present disclosure exist, and these terms Use is not intended to exclude the possibility of the presence or addition of one or more other features, numbers, steps, operations, components, parts, or combinations thereof.

In an embodiment of the present disclosure, a 'module' or 'unit' refers to a functional part that performs at least one function or operation, and may be implemented with hardware or software or a combination of hardware and software. In addition, a plurality of 'modules' or 'units' may be integrated into at least one software module and implemented by at least one processor, except for 'modules' or 'units' that need to be implemented with specific hardware. there is.

In addition, unless defined otherwise, all terms used in this disclosure, including technical or scientific terms, have the same meaning as commonly understood by one of ordinary skill in the art to which this disclosure belongs. It should be noted that commonly used terms defined in dictionaries should be interpreted as having a meaning consistent with the meaning in the context of the related art, and should not be interpreted excessively limited or expanded unless clearly defined otherwise in the present disclosure. .

1 is a diagram schematically illustrating a system environment for controlling a control target object based on a control target object map according to an embodiment of the present disclosure.

As shown in FIG. 1 , according to an embodiment of the present disclosure, a system 100 for controlling a control target object based on a control target object map (hereinafter referred to as 'control target object control system 100') It may include a user terminal 110, a control target object 120, a communication network 130, and a map generating device 140 (hereinafter referred to as 'map generating device 140') of a control target object.

The user terminal 110 according to an embodiment of the present disclosure is a digital device having a function of communicating with the control target object 120 and/or the map generating device 140 through a communication network 130, and includes a memory unit. It may be a digital device equipped with a microprocessor and equipped with an arithmetic capability. In one embodiment, the user terminal 110 is a wearable device such as smart glasses, a smart watch, a smart band, a smart ring, a smart necklace, a smart earphone, a smart earphone, a smart earring, a head mounted display (HMD), a smart phone, It may be a more or less traditional device such as a smart pad, desktop computer, server, notebook computer, workstation, PDA, web pad, mobile phone, remote controller, and the like. In this drawing, three user terminals 110 are shown, but the present disclosure is not limited thereto.

According to an embodiment of the present disclosure, the user terminal 110 is configured to receive a user input and display a control screen of the control target object 120 or a link of the control screen of the control target object 120 corresponding thereto. A display means may be provided. In an embodiment, the user terminal 110 may receive map information of the control target object from the map generating device 140, and the control target object 120 is generated based on the relative position of the control target object in the control target object map. can be explored.

According to an embodiment of the present disclosure, the user terminal 110 may include an application supporting functions according to the present disclosure. Such an application may be downloaded from the controlled object control system 100 or an external application distribution server (not shown).

The control target object 120 according to an embodiment of the present disclosure is a device or object configured to be controlled by a user's manipulation, and the user terminal 110 or a map generating device by the communication network 130 or a predetermined processor. Communication with 140 or an external system (not shown) may be made. The control target object 120 may receive a control command transmitted from the user terminal 110 through the communication network 130 and may be configured to control its own motion or function according to the received control command. The control target object 120 according to an embodiment of the present disclosure may be, for example, an Internet of Things (IoT) device such as a television, an air conditioner, a speaker, a lamp, a humidifier, a chest of drawers, a window, or blinds.

According to one embodiment of the present disclosure, the control target object 120 may include a camera module (not shown) configured to capture an image. In one embodiment of the present disclosure, a camera module included in the control target object 120 may capture a predetermined space. Here, the predetermined space means a space including a subject. In one embodiment, the camera module may include at least one of a camera (eg, a 2D camera such as an RGB camera or an IR camera, a 3D camera, etc.), an image sensor, and an image sensor. Images obtained through the camera module may be transmitted to the outside (eg, the map generating device 140) through the communication network 130.

The communication network 130 according to an embodiment of the present disclosure may be configured regardless of communication aspects such as wired communication or wireless communication, and may include a local area network (LAN), a metropolitan area network (MAN), It may be configured with various communication networks such as a wide area network (WAN). Preferably, the communication network 130 referred to in this specification may be the well-known Internet or the World Wide Web (WWW). However, the communication network 130 may include at least a part of a known wired/wireless data communication network, a known telephone network, or a known wire/wireless television communication network without being limited thereto.

For example, the communication network 130 is a wireless data communication network, WiFi communication, WiFi-Direct communication, Long Term Evolution (LTE) communication, 5G communication, Bluetooth communication (low power Bluetooth (BLE) ; Bluetooth Low Energy) communication), infrared communication, ultrasonic communication, and the like may be implemented in at least a part thereof. For another example, the communication network 130 is an optical communication network, and may implement a conventional communication method such as LiFi (Light Fidelity) in at least a part thereof.

The map generating device 140 according to an embodiment of the present disclosure may generate a map of the control target object 120 . In one embodiment, the map generating device 140 analyzes an image including a subject received from the control target object 120 to generate posture coordinate information, and maps the control target object 120 to the corresponding posture coordinate information. Posture coordinate information of the object to be controlled 120 may be generated. In an embodiment, the map generating device 140 may generate direction coordinate information by converting coordinate information to which the control target object 120 is mapped based on the subject.

According to an embodiment of the present disclosure, the map generating device 140 is a digital device having a memory unit and a microprocessor equipped with an arithmetic capability, and may be a server system, and may be a user terminal 110 through a communication network 130 ) and the control target object 120 and communication may be performed.

2 is a functional block diagram schematically illustrating a functional configuration of an apparatus for generating a control target object map according to an embodiment of the present disclosure.

Referring to FIG. 2 , a map generating device 140 according to an embodiment of the present disclosure includes an image receiving unit 202, a coordinate information generating unit 204, a control target object location determining unit 206, a communication unit 208, and A storage unit 210 may be included. The components shown in FIG. 2 do not reflect all the functions of the map generating device 140 and are not essential, so the map generating device 140 includes more or fewer components than the illustrated components. can include

According to an embodiment of the present disclosure, at least a part of the image receiving unit 202, the coordinate information generating unit 204, the control target object location determination unit 206, the communication unit 208, and the storage unit 210 are external. It may be a program module that communicates with a system (not shown) of. These program modules may be included in the map generating device 140 in the form of an operating system, application program modules, or other program modules, and may be physically stored in various known storage devices. Also, these program modules may be stored in a remote storage device capable of communicating with the map generating device 140 . Meanwhile, these program modules include, but are not limited to, routines, subroutines, programs, objects, components, data structures, etc. that perform specific tasks or execute specific abstract data types according to the present disclosure.

Meanwhile, although the map generating device 140 has been described as above, this description is exemplary, and at least some of the components or functions of the map generating device 140 are realized in an external system (not shown) as needed, or It should be understood that it can be included within an external system.

According to an embodiment of the present disclosure, the image receiving unit 202 may perform a function of receiving at least one image including a subject from at least one control target object. Here, the subject is a reference object for determining the location of at least one control target object, and may be, for example, a person such as a user, or a general device or object such as a display or a chair. In one embodiment, the subject may be one or more than two.

3 is a diagram illustratively illustrating a state in which a subject is facing a specific control target object in a predetermined space in which a plurality of control target objects are located according to an embodiment of the present disclosure.

As shown in FIG. 3, a mirror (ⓐ), a speaker (ⓑ), a blind (ⓒ), a humidifier (ⓓ), a television (ⓔ), a lighting (ⓕ), and an air conditioner (ⓖ) are used as objects to be controlled in a predetermined space. and stands ⓗ are disposed in different positions. In FIG. 3 , the subject 310 is a user, and the user 310 is looking at a specific control target object (light ⓕ) among a plurality of control target objects.

Referring to FIGS. 2 and 3 together, the image receiving unit 202 may receive a plurality of images including a subject 310 from a plurality of control target objects ⓐ to ⓗ. Here, the plurality of images may be images captured by each camera module of the plurality of control target objects 120 .

According to an embodiment of the present disclosure, the image receiving unit 202 may perform a function of detecting a region image of a subject from each of a plurality of received images. Here, the area image of the subject is a reference area for estimating the posture of the subject. For example, if the subject is a user, the subject's area image may be the user's face area image. For another example, when the subject is a display, the area image of the subject may be a screen area image. For another example, when the subject is a chair, the area image of the subject may be a front backrest area image.

According to an embodiment of the present disclosure, the coordinate information generating unit 204 may perform a function of generating posture coordinate information by analyzing an area image of a subject.

4 is a diagram exemplarily illustrating a method of generating posture coordinate information by analyzing a region image of a subject according to an embodiment of the present disclosure.

As shown in FIG. 4 , the coordinate information generation unit 204 may detect a rotation direction and rotation angle of an area image of a subject (eg, a face area image of a user) and generate posture coordinate information of the area image. Specifically, the coordinate information generation unit 204 determines an angle (pitch angle) according to the degree of lifting or bending the face and an angle (yaw angle) according to the degree of turning the face left and right in each face region image. can be detected, and based on the pitch and yaw angles detected in each face region image, each face region image is placed in a yaw-pitch coordinate system (posture coordinate system) to position the face region image. information can be generated. In one embodiment, the attitude coordinate system may be composed of a yaw coordinate axis and a pitch coordinate axis orthogonal to the yaw coordinate axis, and a region image facing a specific control target object 120 in front without being biased to either side among the region images is the origin ( 0, 0).

According to an embodiment of the present disclosure, the coordinate information generating unit 204 may perform a function of generating mapped coordinate information by mapping at least one control target object to posture coordinate information. In an embodiment, the coordinate information generating unit 204 may generate mapped coordinate information by mapping at least one control target object 120 to the posture coordinate information of the face region image. Specifically, as shown in FIG. 4 , each region image (eg, face region image) may be matched with a control target object obtained by capturing a corresponding region image, and the matched control target object may be matched in a yaw-pitch coordinate system (posture). coordinate system) to generate mapped coordinate information.

According to an embodiment of the present disclosure, the coordinate information generating unit 204 may perform a function of generating direction coordinate information by converting mapped coordinate information based on a subject.

5 is a diagram exemplarily illustrating a method of generating direction coordinate information according to an embodiment of the present disclosure.

Referring to FIGS. 3 and 5 , the coordinate information generator 204 may generate direction coordinate information by converting posture coordinate information according to yaw and pitch angles into direction coordinate information according to up, down, left and right directions. Specifically, in the yaw-pitch coordinate system (attitude coordinate system), pitch down is up, pitch up is down, and yaw left is left. , A yaw right can be converted into a right to create an up, down, left, and right directional coordinate system, and directional coordinate information can be created by arranging each control target object in the directional coordinate system.

In one embodiment of the present disclosure, as shown in FIG. 5, based on the control target object (lighting (ⓕ)) located at the origin, the air conditioner (ⓖ) is on the right, the blind (ⓒ) is on the upper left, and the blind (ⓒ) is on the lower left. A map of a 2-dimensional object to be controlled where the television (ⓔ) is located may be created.

According to an embodiment of the present disclosure, since the relative position of the object to be controlled is determined with respect to the subject, the subject to be controlled may be searched according to the physical position actually recognized by the user. In an embodiment, when a 2D map of a control target object is provided to the user terminal 110, the user may search for the control target object by performing a user input including direction information to the user terminal 110. Here, the user input including the direction information may be, for example, dragging in a direction, clicking on a direction movement icon, swiping in a predetermined number of directions, or physically moving the user terminal 110 .

Hereinafter, a method of searching for a control target object will be described with reference to the map of the control target object shown in FIG. 5 by way of various examples. For example, when the user drags to the right while the control screen of the light ⓕ is displayed on the user terminal 110 (eg, smart watch), the light is positioned to the right of the light ⓕ on the map of the object to be controlled. A control screen of an air conditioner (ⓖ), which is a control target object closest to (ⓕ), may be displayed. For another example, when the user clicks a direction icon moving upward while the control screen of the humidifier (ⓓ) is displayed on the user terminal 110 (eg, a smartphone), from the humidifier (ⓓ) on the map of the control target object. A control screen of a blind (ⓒ), which is a control target object located in an upward direction and closest to the humidifier (ⓓ), may be displayed. For another example, when the user physically moves the tablet to the upper right corner while the link of the control screen of the speaker ⓑ is displayed on the user terminal 110 (eg, the tablet), the speaker (ⓑ) is displayed on the map of the control target object. A link to the control screen of a blind (ⓒ), which is a control target object located at the upper right of ⓑ) and closest to the speaker (ⓑ), may be displayed.

Referring back to FIG. 2 , the control target object position determiner 206 according to an embodiment of the present disclosure may perform a function of determining the position of each of the at least one control target object based on coordinate information. In an embodiment, the control target object position determiner 206 may determine the relative position of the control target object based on the direction coordinate information of each control target object, and generate a map according to the relative position of the control target object. can

According to an embodiment of the present disclosure, the communication unit 208 transmits/receives data from/to the image receiver 202, the coordinate information generator 204, the control target object location determiner 206, and the storage unit 210. It can also function to enable the map generating device 140 to communicate with an external communication network.

According to one embodiment of the present disclosure, the storage unit 210 may perform a function of storing data required to operate the map generating device 140 . The data stored in the storage unit 210 includes, for example, an image including the subject received by the image receiving unit 202, information about an area image of the subject detected in each image, and coordinate information generating unit 204. There may be generated posture coordinate information, direction coordinate information, radial line information, information about the position of each control target object determined by the control target object position determining unit 206 , and the like.

6 is an operational flowchart exemplarily illustrating a method of generating a map of a control target object according to an embodiment of the present disclosure.

First, a method of generating a map of a control target object starts with the map generating device 140 receiving an image including a subject (S602). In one embodiment, the map generating device 140 may receive an image captured by a camera module of the control target object 120 .

Next, in step S604, the map generating device 140 may detect an area image of the subject from each of the plurality of images received in step S602. Here, the subject may be the user, and the subject's area image may be the user's face area image.

Next, in step S606, the map generating device 140 may generate attitude coordinate information by analyzing the image of the area of the subject detected in step S604. In one embodiment, step S606 may be performed in a manner of detecting yaw angles and pitch angles of the area image of the subject.

Next, in step S608, the map generating device 140 may generate mapped coordinate information by mapping at least one control target object to the posture coordinate information generated in step S606. In an embodiment, the map generating device 140 may match each area image with a control target object that has captured the corresponding area image, and yaw-pitch the control target object matched to each area image according to the posture coordinate information. Mapped coordinate information can be created by mapping to a coordinate system (attitude coordinate system).

In step S610, the map generating device 140 may generate direction coordinate information by converting the mapped coordinate information generated in step S608 based on the subject. In an embodiment, the map generating device 140 may generate direction coordinate information by converting posture coordinate information according to yaw and pitch angles into direction coordinate information according to up, down, left and right directions.

In an additional step (not shown), the map generating device 140 may determine the relative position of the control target object based on the direction coordinate information and generate a map according to the relative position of the control target object.

Meanwhile, in the previous embodiment, a method for generating a 2D control target object map according to the relative position of the control target object has been described, but a 3D control target object map is generated by specifying the location of the control target object in a 3D coordinate system. It is also possible to do Hereinafter, a method of generating a 3D control target object map according to another embodiment of the present disclosure will be described in detail with reference to the drawings.

According to another embodiment of the present disclosure, the image receiving unit 202 may perform a function of receiving at least one image including a first subject and a second subject from at least one control target object. Here, the first subject and the second subject may be different objects at different locations or the same object at different locations at different times.

7 is a diagram illustratively showing a state in which a first subject and a second subject are present in a predetermined space in which a plurality of control object objects are located according to another embodiment of the present disclosure.

As shown in FIG. 7, a mirror (ⓐ), a speaker (ⓑ), a blind (ⓒ), a humidifier (ⓓ), a television (ⓔ), a lighting (ⓕ), and an air conditioner (ⓖ) are used as objects to be controlled in a predetermined space. and stands ⓗ are disposed in different positions. In FIG. 7 , a first subject 710 faces the light ⓕ as a user, and a second subject 720 faces the blind ⓒ as a display. In an embodiment, the image receiving unit 202 may receive images including the first subject 710 and the second subject 720 from the plurality of control target objects ⓐ to ⓗ.

According to another embodiment of the present disclosure, the image receiver 202 may perform a function of detecting an area image of each subject in each received image. In an embodiment, the image receiving unit 202 may detect an image of a first area of a first subject and an image of a second area of a second subject. In FIG. 7 , the first area image of the first subject 710 may be the user's face area image, and the second area image of the second subject 720 may be the screen area image of the display.

According to another embodiment of the present disclosure, the coordinate information generation unit 204 may perform a function of generating posture coordinate information by analyzing an area image of a subject. In an embodiment, the coordinate information generating unit 204 may generate first posture coordinate information by analyzing the first region image of the first subject. Also, in an embodiment, the coordinate information generating unit 204 may generate second posture coordinate information by analyzing the second region image of the second subject.

8 is a diagram exemplarily illustrating a method of generating posture coordinate information by analyzing a region image according to another embodiment of the present disclosure.

As shown in FIG. 8 , the coordinate information generation unit 204 may detect a rotation direction and a rotation angle of an area image of a subject, and may generate posture coordinate information based on the rotation direction and rotation angle. Specifically, as shown in (a) of FIG. 8 , a pitch angle and a yaw angle may be detected from each face region image, and each face region image may be determined based on the detected pitch angle and yaw angle. Posture coordinate information of the face region image can be created by placing it in the posture coordinate system according to . In addition, as shown in (b) of FIG. 8 , the pitch angle and the yaw angle may be detected from the screen area image of the display, and each screen area image is converted to the pitch angle and the yaw angle based on the detected pitch angle and yaw angle. Posture coordinate information can be created by arranging the posture coordinate system according to the posture coordinate system. In one embodiment, a front-facing region image that is not biased to any one side among each region image may be configured to be disposed at the origin (0,0) of the posture coordinate system.

According to another embodiment of the present disclosure, the coordinate information generation unit 204 may generate first mapped coordinate information by mapping at least one control target object to the first posture coordinate information of the first region image, respectively; The second mapped coordinate information may be generated by mapping at least one control target object to the second posture coordinate information of the second region image. As shown in (a) of FIG. 8 , a control target object matching the face region image (eg, a control target object that captures the corresponding face region image) may be mapped to the posture coordinate system of the face region image. As shown in (b) of 8, a control target object matching the screen region image (eg, a control target object that captures the screen region image) may be mapped to the posture coordinate system of the screen region image.

According to another embodiment of the present disclosure, the coordinate information generation unit 204 may generate first direction coordinate information by converting the first mapped coordinate information based on the first subject, and may generate the second mapped coordinate information. The second direction coordinate information may be generated by transforming the second subject based on the reference. In an embodiment, the coordinate information generation unit 204 may generate direction coordinate information by converting posture coordinate information according to yaw and pitch angles into direction coordinate information according to up, down, left and right directions.

According to another embodiment of the present disclosure, the coordinate information generation unit 204 may perform a function of generating half-line information by connecting each of at least one control target object with a half-line based on the subject based on the direction coordinate information. .

9 is a diagram exemplarily illustrating a method of generating half-line information according to another embodiment of the present disclosure.

As shown in (a) of FIG. 9 , the coordinate information generation unit 204 controls a plurality of control targets based on the first subject 910 in a reference space (3-dimensional coordinate system) based on the first direction coordinate information. First ray information may be generated by connecting each of the objects ⓐ to ⓗ with a ray. In addition, as shown in (b) of FIG. 9 , the coordinate information generator 204 includes a plurality of control target objects (ⓐ) based on the second subject 920 in the reference space based on the second direction coordinate information. to ⓗ) may be connected with a half line to generate second half line information.

According to another embodiment of the present disclosure, the control target object position determination unit 206 may perform a function of determining the position of each of at least one control target object.

10 is a diagram exemplarily illustrating a method of determining a location of each control target object according to another embodiment of the present disclosure.

First, as shown in (a) of FIG. 10, the location of each subject can be specified by referring to the positional relationship between each subject and a ray connected to the same control target object on the reference space (on the three-dimensional coordinate system). can In one embodiment, the control target object location determination unit 206 may include a ray connecting the first subject 1010 and each control target object (ⓐ to ⓗ) and the second subject 1020 and each control target object ( The location of each subject can be specified by adjusting the positional relationship between the ray lines connecting ⓐ to ⓗ). In one embodiment, the control target object location determining unit 206 intersects the same control target object and the same control target object so that the distance between the straight lines connected to the same control target object is the shortest on the reference space. The position of each subject can be specified by adjusting the positional relationship between the connected ray lines.

Next, as shown in (b) of FIG. 10 , the control target object position determination unit 206 may determine the position of each of the at least one control target object based on the ray information. In one embodiment, the control target object location determiner 206 determines an intersection point where a ray path of the first ray information and a ray path of the second linear information for each of the at least one control target object intersect with each other of the corresponding control target object. location can be determined. In one embodiment, when there is no intersection between the ray path of the first ray information and the ray path of the second ray information, the ray path of the first ray information and the ray of the second linear information for each of the at least one control target object One point of the shortest connection line of the path may be determined as a position of each of the at least one control target object. In one embodiment, when there is no intersection between the linear path of the first linear information and the linear path of the second linear information, the criterion is the midpoint of the shortest connecting line between the linear path of the first linear information and the linear path of the second linear information. It may be specified as a location where a control target object exists in space.

The control target object position determining unit 206 may also determine the posture of each control target object. In an embodiment, the control target object location determining unit 206 may determine the posture of each control target object by calculating the posture (yaw angle, pitch angle, roll angle) of the camera module of each control target object. Specifically, the control target object position determining unit 206 refers to parameters (horizontal and vertical angle of view information) of the camera module based on the pixel coordinates of the subject detected on the subject image captured by the camera module of each control target object. Thus, the posture (yaw angle, pitch angle, roll angle) of the camera module can be specified. For example, as shown in (c) of FIG. 10 , pixel coordinates P_e of the face area are detected on the image of the first subject 1010 captured by the camera module of the television (ⓔ), and The posture of the camera module of the television (ⓔ) can be specified by referring to information on the horizontal and vertical angles of view of the camera module of the television (ⓔ) based on the pixel coordinates (P_e). position can be estimated. Similarly, the pixel coordinates (P_h) of the face area are detected on the image of the first subject 1010 captured by the camera module of the stand (ⓗ), and the stand (ⓗ) detects the pixel coordinates (P_h) of the detected face area. The posture of the camera module of the stand ⓗ can be specified by referring to information on the horizontal and vertical angles of view of the camera module of , and the posture of the stand ⓗ can be estimated from the specified posture of the camera module.

Meanwhile, although it has been described here that the control target object position determining unit 206 determines the posture of the control target object, other components of the map generating device 140 implement a function of determining the posture of the control target object as needed. It could be.

In this way, the control target object location determiner 206 may determine not only the position but also the posture of each control target object in the 3D space, and accordingly, the control target object is disposed at the position of each control target object with the determined posture. By doing so, it is possible to create a 3D map of the object to be controlled.

Meanwhile, according to an embodiment of the present disclosure, a roll angle of an image of a region of a subject may be corrected in a process of determining a position of each of the at least one control target object.

According to another embodiment of the present disclosure, the control target object position determining unit 206 may perform a function of determining a roll angle of each subject and at least one control target object. In an embodiment, the control target object location determiner 206 may determine a roll angle of a first area of a first subject and a roll angle of a second area of a second subject. In one embodiment, the control target object position determiner 206 may determine the roll angle of the camera module of the control target object.

11 and 12 exemplarily illustrate a method of correcting a roll angle of an image of an area of a subject according to another embodiment of the present disclosure.

First, as shown in FIG. 11(a) , the control target object location determiner 206 may detect a roll angle of an image of the user's face region, which is the first subject, from the camera module of each control target object. . For example, the roll angle of the user's face area image detected by each control target object is -2° for the mirror (ⓐ), 8° for the speaker (ⓑ), -2° for the blind (ⓒ), and -2° for the humidifier ( ⓓ) is -2°, TV (ⓔ) is -26°, lighting (ⓕ) is -2°, air conditioner (ⓖ) is 11°, and stand (ⓗ) is -2°. Also, as shown in FIG. 11(b) , the control target object location determiner 206 may detect a roll angle of an image of the screen area of the display, which is the second subject, from the camera module of each control target object. . For example, the roll angle of the screen area image of the display detected by each control target object is 0° for the mirror (ⓐ), 10° for the speaker (ⓑ), 0° for the blind (ⓒ), and 0° for the humidifier (ⓓ). 0°, -24° for the television (ⓔ), 0° for the lighting (ⓕ), 13° for the air conditioner (ⓖ), and 0° for the stand (ⓗ).

In an embodiment, the control target object position determining unit 206 may determine a roll angle of a region of a subject based on a roll angle of the subject detected by a camera module of the control target object. Specifically, the roll angle of the region of the subject may be determined by calculating the average of effective roll angles excluding outliers that deviate greatly from the average among the roll angles detected in each control target object. Accordingly, in (a) of FIG. 11, the roll angle of the user's face area, which is the first subject, may be determined to be -2°, and in (b) of FIG. 11, the roll angle of the screen area of the display, which is the second subject, is 0. ° can be determined.

Subsequently, as shown in FIG. 12 , the control target object position determining unit 206 may determine a roll angle of a camera module of each control target object based on a roll angle of a subject region. Specifically, as shown in (a) of FIG. 12, when the roll angle of the face area image detected from the control target object is corrected based on the roll angle (ie, -2°) of the specific face area, the speaker ( ⓑ) is 10°, the television (ⓔ) is -24°, and the air conditioner (ⓖ) is 13°. A roll angle of the camera module may be determined. Similarly, as shown in (b) of FIG. 12, when the roll angle of the screen area image detected from the object to be controlled is corrected based on the roll angle (ie, 0°) of a specific screen area, the speaker (ⓑ) is 10°, the television (ⓔ) is -24°, and the air conditioner (ⓖ) is 13°. The roll angle of can be determined. That is, the roll angle of the camera module of the speaker ⓑ is 10°, the roll angle of the camera module of the television ⓔ is -24°, and the roll angle of the camera module of the air conditioner ⓖ is 13°.

13 is an operational flowchart exemplarily illustrating a method of generating a map of a control target object according to another embodiment of the present disclosure.

Referring to FIG. 13 , a method for generating a map of a control target object according to another embodiment of the present disclosure includes receiving an image (S1302), detecting an area image (S1304), and generating posture coordinate information (S1306). ), generating mapped coordinate information (S1308), generating direction coordinate information (S1310), generating radial line information (S1312), and determining the location of the object to be controlled (S1314). can

First, in step S1302, the map generating device 140 may receive at least one image including a first subject and a second subject from at least one control target object. Here, the first subject and the second subject may be different objects at different locations or the same object at different locations at different times.

Subsequently, in step S1304, the map generating device 140 may detect an area image of each subject from each received image. In an embodiment, the image receiving unit 202 of the map generating device 140 may detect an image of a first area of a first subject and an image of a second area of a second subject. Here, the image of the first area of the first subject may be an image of the user's face area, and the image of the second area of the second subject may be an image of the screen area of the display.

Next, in step S1306, the map generating device 140 may analyze the region image detected in step S1304 to generate posture coordinate information. In an embodiment, the coordinate information generating unit 204 of the map generating device 140 may generate first posture coordinate information by analyzing the image of the first area of the first subject, and may generate the image of the second area of the second subject. By analyzing the second posture coordinate information may be generated.

In step S1308, the map generating device 140 may generate mapped coordinate information by mapping at least one control target object to the posture coordinate information generated in step S1306. In an embodiment, the coordinate information generation unit 204 of the map generating device 140 may generate first mapped coordinate information by mapping at least one control target object to the first posture coordinate information, respectively; Second mapped coordinate information may be generated by mapping control target objects to posture coordinate information, respectively.

In step S1310, the map generating device 140 may generate direction coordinate information by transforming the mapped coordinate information based on the subject. In an embodiment, the coordinate information generation unit 204 may generate first direction coordinate information by transforming the first mapped coordinate information based on the first subject, and convert the second mapped coordinate information to the second subject. The second direction coordinate information may be generated by converting to a standard.

In step S1312, the map generating device 140 may generate half line information by connecting each of the at least one control target object with a half line based on the subject based on the direction coordinate information. In an embodiment, the coordinate information generation unit 204 of the map generating device 140 connects each of a plurality of control target objects with a ray based on the first subject based on the first direction coordinate information to generate the first ray information. The second half line information may be generated by connecting each of the plurality of control target objects with a half line based on the second subject based on the second direction coordinate information.

In step S1314, the map generating device 140 may determine the location of each of the at least one control target object. In an embodiment, the control target object location determiner 206 of the map generating device 140 determines that a linear path of first linear information and a linear path of second linear information for each of at least one control target object intersect each other. The intersection point can be determined as the position of the corresponding object to be controlled. In an embodiment, the control target object location determiner 206 of the map generating device 140 is configured to connect the shortest distance between a linear path of the first linear information and a linear path of the second linear information for each of the at least one control target object. One point of may be determined as a position of each of the at least one control target object.

Also, in step S1314, the map generating device 140 may determine the posture of each of the at least one control target object. In an embodiment, the map generating device 140 may calculate a posture of each camera module of at least one control target object, and determine a posture of each of the at least one control target object from the calculated posture. Specifically, the map generating device 140 refers to the parameter (horizontal and vertical angle of view information) of the camera module of the control object based on the pixel coordinates of the subject detected on the image taken by the camera module of each control object. The posture (yaw angle, pitch angle, roll angle) of the camera module of the object to be controlled may be calculated, and the posture of at least one object to be controlled may be determined therefrom.

Meanwhile, in step S1314, the map generating device 140 may determine a roll angle of each subject and a roll angle of each control target object. In an embodiment, the map generating apparatus 140 may determine an average of effective roll angles as the roll angle of the region of the subject, excluding outliers. In an embodiment, the map generating device 140 may determine the roll angle of at least one control target object based on the roll angle of the first area of the first subject, and based on the roll angle of the second area of the second subject, at least one The roll angle of one control target object can be determined.

In an additional step (not shown), the map generating device 140 may generate a map (3D map) of the control target object based on the position and posture of each of the at least one control target object.

As such, in the present embodiment, a map of the control target object may be created by generating coordinate information of each control target object based on two or more subjects. In particular, by connecting each subject and each control target object with a ray to generate ray information, and determining the position of the control target object by referring to the positional relationship between the ray paths of the ray information, the three-dimensional control target object You can create maps.

Embodiments according to the present disclosure described above may be implemented in the form of program instructions that can be executed through various computer components and recorded in a computer readable recording medium. A computer-readable recording medium may include program instructions, data files, data structures, etc. alone or in combination. Program instructions recorded on the computer-readable recording medium may be specially designed and configured for the present disclosure, or may be known and usable to those skilled in the art of computer software. Examples of computer-readable recording media include magnetic media such as hard disks, floppy disks and magnetic tapes, optical recording media such as CD-ROMs and DVDs, and magneto-optical media such as floptical disks. medium), and hardware devices specially configured to store and execute program instructions, such as ROM, RAM, flash memory, and the like. Examples of program instructions include high-level language codes that can be executed by a computer using an interpreter or the like as well as machine language codes generated by a compiler. A hardware device may be modified with one or more software modules to perform processing according to the present disclosure and vice versa.

Although the present disclosure has been described with specific details and limited examples, such as specific components, the above embodiments are only provided to help a more general understanding of the present disclosure, and the present disclosure is not limited thereto, and the present disclosure is not limited thereto. Those skilled in the art can make various modifications and variations from these descriptions.

Therefore, the spirit of the present disclosure should not be limited to the embodiments described above and should not be defined, and it can be said that not only the claims described below but also all modifications equivalent or equivalent to these claims fall within the scope of the spirit of the present disclosure. will be.

100: control target object control system
110: user terminal
120: object to be controlled
130: communication network
140: map generating device

Claims (14)

As a method of generating a map of a control target object,
receiving at least one image including a subject from at least one control target object;
detecting an area image of the subject from each of the at least one image;
generating posture coordinate information by analyzing the area image;
generating mapped coordinate information by mapping the at least one control target object to the posture coordinate information; and
converting the mapped coordinate information based on the subject to generate direction coordinate information;
A method of generating a map of a control target object including a. According to claim 1,
The generating of the posture coordinate information is performed by detecting yaw angles and pitch angles of the region image. According to claim 1,
The method of generating a map of a control target object, wherein the generating of the mapped coordinate information includes matching the region image with a control target object obtained by photographing a corresponding region image. According to claim 1,
The method of generating a map of a control target object, wherein the generating of the directional coordinate information is performed by converting posture coordinate information according to yaw and pitch angles into directional coordinate information according to up, down, left and right directions. As a method of generating a map of a control target object,
receiving at least one image including a first subject and a second subject from at least one control target object;
detecting a first area image of the first subject and a second area image of the second subject from each of the at least one image;
generating first posture coordinate information and second posture coordinate information by analyzing the first region image and the second region image, respectively;
generating first mapped coordinate information and second mapped coordinate information by mapping the at least one control target object to the first posture coordinate information and the second posture coordinate information, respectively;
converting the first mapped coordinate information and the second mapped coordinate information based on the first subject and the second subject, respectively, to generate first direction coordinate information and second direction coordinate generation information;
Based on the first direction coordinate information, first half line information is generated by connecting each of the at least one control target object with a half line based on the first subject, and the second subject based on the second direction coordinate information. generating second half line information by connecting each of the at least one control target object with a half line based on; and
determining a location of each of the at least one control target object based on the first and second linear information;
A method of generating a map of a control target object including a. According to claim 5,
In the step of determining the position of the object to be controlled,
determining an intersection point where a ray path of the first ray information and a ray path of the second linear information for each of the at least one control target object intersect as a position of the corresponding control target object; or
Determining a point of a shortest connecting line between a ray path of the first ray information and a ray path of the second linear information for each of the at least one control target object as the position of each of the at least one control target object;
Including, a method for generating a map of the control target object. According to claim 5,
The method of generating a map of a control target object, wherein the determining of the control target object position comprises determining a posture of each of the at least one control target object. According to claim 7,
In the determining of the posture of the object to be controlled, the posture of each of the at least one target object to be controlled is determined from the posture of each of the camera modules of the at least one object to be controlled. According to claim 5,
In the step of determining the position of the object to be controlled,
determining a roll angle of a first area of the first subject and a roll angle of a second area of the second subject; and
Further comprising determining a roll angle of the at least one control target object,
How to create a map of controlled objects. According to claim 9,
In the step of determining the roll angle of the first area of the first subject and the roll angle of the second area of the second subject,
A method of generating a map of a control target object in which an average of effective roll angles is determined as a roll angle of a region of a corresponding subject, excluding outliers. According to claim 9,
The step of determining the roll angle of the at least one control target object,
determining a roll angle of the at least one control target object based on a roll angle of a first region of the first subject; and
and determining a roll angle of the at least one control target object based on a roll angle of a second region of the second subject. A non-temporary computer readable recording medium storing a computer program for executing the method according to any one of claims 1 to 11. A device for generating a map of a control target object,
an image receiver configured to receive at least one image including a subject from at least one control target object, and to detect an image of a region of the subject in each of the at least one image;
Posture coordinate information is generated by analyzing the area image, mapped coordinate information is generated by mapping the at least one control target object to the posture coordinate information, and the mapped coordinate information is converted based on the subject to determine direction A coordinate information generation unit for generating coordinate information; and
A control target object location determiner configured to determine a position of each of the at least one control target object based on the direction coordinate information.
A device for generating a map of a control target object comprising a. A device for generating a map of a control target object,
Receiving at least one image including a first subject and a second subject from at least one control target object, and in each of the at least one image, a first area image of the first subject and a second area of the second subject an image receiver for detecting an image;
Analyzing the first region image and the second region image, respectively, first posture coordinate information and second posture coordinate information are generated, and the at least one control target is assigned to the first posture coordinate information and the second posture coordinate, respectively. An object is mapped to generate first mapped coordinate information and second mapped coordinate information, and the first mapped coordinate information and the second mapped coordinate information are based on the first subject and the second subject, respectively. transform to generate first direction coordinate information and second direction coordinate information, and connect each of the at least one control target object with a half line based on the first subject based on the first direction coordinate information to obtain the first half line information. And a coordinate information generation unit configured to generate second half line information by connecting each of the at least one control target object with a half line based on the second subject based on the second direction coordinate information, and
A control target object position determination unit determining a position of each of the at least one control target object based on the first and second linear line information.
A device for generating a map of a control target object comprising a.

Images