KR102260751B1 - Electronic device and control method thereof - Google Patents

Abstract

Disclosed is a control method of a coding robot. According to the present invention, the control method of the coding robot comprises: a step in which the coding robot acquires information of a subject to be recognized; a step in which the coding robot acquires control data based on the acquired information of the subject to be recognized; a step in which the coding robot determines whether the control data matches preset control information; a step in which the coding robot performs an operation corresponding to the preset control command when the control data matches the preset control information; and a step in which the coding robot analyzes the control data and matches one of the preset control commands when the control data do not match the preset control information. The control data are at least one of emotion data and speech data of the subject to be recognized. The present invention aims to provide an electronic device and a control method thereof, which are able to determine the emotion or speech of a subject to be recognized.

Description

Electronic device and control method thereof

The present invention relates to an electronic device and a method for controlling the same.

Recently, as coding education becomes compulsory, smart toys incorporating various information technology (IT) technologies are attracting attention, and one of them is the coding robot.

However, in the conventional coding robot, since the coding robot is operated only by the coding task, coding education must be preceded in order to operate the coding robot, and even if such coding education is preceded, if there is an error in the coding made or lack of coding skills, As the coding robot is not operated, there is a problem that interest in the coding robot is rapidly reduced.

Furthermore, since the conventional coding robot can only perform simple operations for simple coding commands, there is a limitation in that various interactions with the user of the coding robot are impossible.

Accordingly, the need for a coding robot that can operate easily but can implement various operations according to various inputs such as user's emotions is emerging.

Laid-open Patent Publication No. 10-2020-0093933, 2020.08.06

An object of the present invention is to provide an electronic device and a method for controlling the same.

The problems to be solved by the present invention are not limited to the problems mentioned above, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

A control method of a coding robot according to an aspect of the present invention for solving the above-described problems, the coding robot, the steps of obtaining information of the subject to be recognized; obtaining, by the coding robot, control data based on the acquired information of the subject to be recognized; determining, by the coding robot, whether the control data matches preset control information; performing, by the coding robot, an operation corresponding to the preset control command when the control data matches preset control information; and analyzing, by the coding robot, when the control data does not match preset control information, and matching the control data with any one of preset control commands. It is characterized in that the data is at least one of emotional data and voice data of the subject.

In this case, when the control data is the emotional data of the recognition target, obtaining the control data may include, by the coding robot, obtaining a face photographed image of the recognition target; obtaining, by the coding robot, a model variable based on the face photographed image; determining, by the coding robot, whether the recognition target matches the face of a pre-registered user based on the acquired model variables; obtaining, by the coding robot, a feature value corresponding to the photographed face image based on the photographed face image, when the recognition target matches the previously registered user; determining, by the coding robot, a face part including the eyes, nose, and mouth of the recognition target based on the acquired feature values; determining, by the coding robot, the emotion of the person to be recognized based on the determined face part; and obtaining, by the coding robot, the control data based on the determined emotion.

In this case, the step of performing the operation corresponding to the control command may include: obtaining, by the coding robot, coordinate information on a two-dimensional vector space corresponding to the determined emotion; obtaining a control command corresponding to coordinate information on the vector space; and performing an operation corresponding to the control command. may include.

In this case, the performing of the operation corresponding to the control command may include, by the coding robot, obtaining, by the coding robot, emotion change information of the recognition subject based on the photographed face image of the recognition subject; obtaining, by the coding robot, first coordinate information corresponding to the first emotion and second coordinate information corresponding to the second emotion when the emotion of the person to be recognized is changed from the first emotion to the second emotion; and obtaining, by the coding robot, a movement command corresponding to a vector value corresponding to the first coordinate and the second coordinate as the control command.

In this case, when the control command is a control command for moving the coding robot, the control method may include: recognizing, by the coding robot, an object around the coding robot; obtaining, by the coding robot, control information corresponding to the pre-stored sign when the recognized object matches a pre-stored sign; recognizing, by the coding robot, a lane around the coding robot; and moving, by the coding robot, along the recognized lane, and moving corresponding to the control information. The step of recognizing the lane may include: acquiring information on two consecutive line segments based on an image input to the coding robot, and acquiring a lane area based on the obtained line segment information; obtaining distance information between the obtained line segment and the coding robot; and controlling the coding robot based on the lane area and the distance information. may include.

In this case, when the control data is the voice data of the recognition target, the control method includes, in the coding robot, an operation speed control command and an operation repetition number control command for the coding robot based on the voice data generating a control block; obtaining, by the coding robot, at least one of motion speed information and motion repetition number information based on the voice data; and obtaining, by the coding robot, a control command corresponding to at least one of the operation speed and the number of repetitions of the coding robot based on at least one of the operation speed information and the operation repetition number information. may include.

In this case, the step of obtaining a control command corresponding to at least one of the operation speed and the number of operation repetitions of the coding robot may include, when the voice data matches the preset control information, by the coding robot, the preset control information obtaining the operation speed or the number of repetitions of the coding robot as the control command based on the corresponding operation speed information or operation repetition number information; generating, by the coding robot, syllable spacing information based on the voice data when the voice data does not match preset control information, and generating the motion speed information in response to the syllable spacing information; generating, by the coding robot, signal strength information based on the voice data when the voice data does not match preset control information, and generating the operation repetition number information in response to the signal strength information; and acquiring the operation speed or the number of repetitions of the coding robot as the control command based on the generated operation speed information or the number of repetitions.

Other specific details of the invention are included in the detailed description and drawings.

According to various embodiments of the present invention described above, the coding robot may determine the emotion or voice of the recognition target, and may be controlled based on a coding block corresponding to the determined emotion or voice.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is an exemplary diagram for explaining a coding robot according to an embodiment of the present invention.
2 is a block diagram showing the configuration of a coding robot according to an embodiment of the present invention.
3 is a flowchart illustrating a control method of a coding robot according to an embodiment of the present invention.
4 and 5 are exemplary views for explaining a method of determining a recognized user's emotion according to an embodiment of the present invention.

Advantages and features of the present invention and methods of achieving them will become apparent with reference to the embodiments described below in detail in conjunction with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but may be implemented in various different forms, and only the present embodiments allow the disclosure of the present invention to be complete, and those of ordinary skill in the art to which the present invention pertains. It is provided to fully understand the scope of the present invention to those skilled in the art, and the present invention is only defined by the scope of the claims.

The terminology used herein is for the purpose of describing the embodiments and is not intended to limit the present invention. As used herein, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, “comprises” and/or “comprising” does not exclude the presence or addition of one or more other components in addition to the stated components. Like reference numerals refer to like elements throughout, and "and/or" includes each and every combination of one or more of the recited elements. Although "first", "second", etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from another. Accordingly, it goes without saying that the first component mentioned below may be the second component within the spirit of the present invention.

Unless otherwise defined, all terms (including technical and scientific terms) used herein will have the meaning commonly understood by those of ordinary skill in the art to which this invention belongs. In addition, terms defined in a commonly used dictionary are not to be interpreted ideally or excessively unless specifically defined explicitly.

As used herein, the term “unit” or “module” refers to a hardware component such as software, FPGA, or ASIC, and “unit” or “module” performs certain roles. However, “part” or “module” is not meant to be limited to software or hardware. A “unit” or “module” may be configured to reside on an addressable storage medium or to reproduce one or more processors. Thus, by way of example, “part” or “module” refers to components such as software components, object-oriented software components, class components and task components, processes, functions, properties, Includes procedures, subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Components and functionality provided within “parts” or “modules” may be combined into a smaller number of components and “parts” or “modules” or as additional components and “parts” or “modules”. can be further separated.

Spatially relative terms "below", "beneath", "lower", "above", "upper", etc. It can be used to easily describe the correlation between a component and other components. A spatially relative term should be understood as a term that includes different directions of components during use or operation in addition to the directions shown in the drawings. For example, when a component shown in the drawing is turned over, a component described as “beneath” or “beneath” of another component may be placed “above” of the other component. can Accordingly, the exemplary term “below” may include both directions below and above. Components may also be oriented in other orientations, and thus spatially relative terms may be interpreted according to orientation.

In this specification, a computer means all types of hardware devices including at least one processor, and may be understood as encompassing software configurations operating in the corresponding hardware device according to embodiments. For example, a computer may be understood to include, but is not limited to, smart phones, tablet PCs, desktops, notebooks, and user clients and applications running on each device.

Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Each step described in this specification is described as being performed by a computer, but the subject of each step is not limited thereto, and at least a portion of each step may be performed in different devices according to embodiments.

1 is an exemplary diagram for explaining a coding robot according to an embodiment of the present invention.

The coding robot 100 according to the present invention may be operated by a coding language programmed in a computer language or a coding block programmed in a computer language. At this time, the coding is the operation of programming a sequential and logical flow using a computer language such as C language, and the coding block may mean a block including a coding instruction completed in order to perform a specific instruction. That is, the coding robot 100 may operate according to the command transmitted by the written coding or the selected coding block.

The coding robot 100 according to the present invention may include a camera, a microphone, a memory, a processor, and a driving unit.

2 is a block diagram showing in detail the configuration of a coding robot according to an embodiment of the present invention.

Specifically, the coding robot 100 may further include a communication unit 140 , a memory 150 , and a sensing unit 160 in addition to the input unit 110 , the output unit 120 , and the processor 130 .

The input unit 110 may include a camera 111 , a microphone 112 , a touch panel 113 , and the like to obtain various external inputs.

The camera 111 is a configuration for acquiring image data around the coding robot 100 . The camera 111 may capture still images and moving images. For example, the camera 111 may include one or more image sensors, lenses, an image signal processor (ISP), or a flash (eg, an LED or a xenon lamp, etc.), and may include a plurality of cameras 111 . In this case, in order to prevent unnecessary power consumption, the camera 111 may be activated only when a specific condition is satisfied. For example, the camera 111 may be activated only when a user is detected around the coding robot 100 . Alternatively, when the coding robot 100 is a part of the IoT system, the camera 111 may be activated when the front door is opened or the condition that the indoor lighting is turned on is satisfied. However, it is not limited to the above-described embodiment, and the camera 111 may be always activated when power is applied to the robot. When the camera 111 is always activated, the camera 111 may store an image or perform image processing only when a specific condition is satisfied. The specific condition is a condition when a user is detected around the coding robot 100, when the coding robot 100 is a part of an IoT system, the camera 111 is a condition in which the front door is opened or the indoor lighting is turned on can

The microphone 112 is a component for acquiring a sound around the coding robot 100 . The microphone 112 may receive an external sound signal to generate electrical voice information, and the coding robot 100 may include a plurality of microphones 112 . The coding robot 100 may estimate the direction of a sound source generating a sound signal by using a difference in arrival time of the sound signal input to each of the plurality of microphones. The microphone 112 may use various noise removal algorithms for removing noise generated in the process of receiving an external sound signal. As in the case of the camera 111 , the microphone 112 may be activated only when a specific condition is satisfied in order to prevent unnecessary power consumption. That is, the microphone 112 may be activated only when a user is detected around the coding robot 100 . Alternatively, when the coding robot 100 is a part of the IoT system, the microphone 112 may be activated when the front door is opened or the condition that the indoor lighting is turned on is satisfied. However, the present invention is not limited to the above-described embodiment, and as in the case of the camera 111 , the microphone 112 may be always activated.

The touch panel 113 is configured to receive various user inputs. The touch panel 113 may receive user data by a user manipulation. The touch panel may also be activated under the above conditions to prevent unnecessary power consumption. Alternatively, the touch panel 113 may be activated only when a user's touch is sensed. The touch panel 113 may be configured in combination with a display to be described later.

It goes without saying that the input unit 110 may have various configurations for receiving various user data in addition to the camera 111 , the microphone 112 , and the touch panel 113 described above.

The output unit 112 may include a driving unit 121 , an audio output unit 122 , and a display 123 .

The driving unit 121 is configured to output various motions in response to user data acquired through the input unit 110 . When the coding robot 100 according to an embodiment of the present invention is the humanoid coding robot 100, the driving unit 121 is a motor or actuator capable of driving at least one joint corresponding to a human joint. may include As another embodiment, when the coding robot 100 according to an embodiment of the present invention is a wheel-type robot, the driving unit 121 may include a motor or an actuator capable of driving the wheel.

The audio output unit 122 is configured to output various audio signals in response to user data acquired through the input unit 110 . The audio output unit 122 may output various types of audio data for which various processing operations such as decoding, amplification, and noise filtering have been performed by the audio processing unit (not shown), as well as various notification sounds or voice messages. In particular, the audio output unit 122 may be implemented as a speaker, but this is only an exemplary embodiment and may be implemented as an output terminal capable of outputting audio data.

The display 123 is configured to output an image in various responses to user data acquired through the input unit 110 . As an embodiment, the display 123 may output an avatar image for expressing emotions. That is, when the coding robot 100 wants to output a happy emotion in response to the user data obtained through the input unit 110 , it can output the happy emotion through the avatar image displayed on the display 123 . . Meanwhile, the display 123 for providing various images may be implemented with various types of display panels. For example, the display panel is a Liquid Crystal Display (LCD), Organic Light Emitting Diodes (OLED), Active-Matrix Organic Light-Emitting Diode (AM-OLED), Liquid Crystal on Silicon (LcoS), or Digital Light Processing (DLP). It may be implemented with various display technologies, such as. In addition, the display 123 may be coupled to at least one of a front area and a side area and a rear area of the coding robot 100 in the form of a flexible display.

The communication unit 140 may communicate with an external device. In particular, the communication unit 140 may include various communication chips or circuits for performing wireless communication, such as a Wi-Fi chip 141 , a Bluetooth chip 142 , an NFC chip 143 , and a mobile communication chip 144 . At this time, the Wi-Fi chip 141, the Bluetooth chip 142, and the NFC chip 143 perform communication in a LAN method, a WiFi method, a Bluetooth method, and an NFC method, respectively. In the case of using the Wi-Fi chip 141 or the Bluetooth chip 142, various types of connection information such as an SSID and a session key are first transmitted and received, and then various types of information can be transmitted and received by using this to connect communication. The mobile communication chip 144 refers to a chip that performs communication according to various communication standards such as 3rd Generation (3G), 3rd Generation Partnership Project (3GPP), and Long Term Evolution (LTE). On the other hand, the communication unit 140 is a USB (universal serial bus), HDMI (high definition multimedia interface), RS-232 (recommended standard232), power line communication, wired communication including at least one of POTS (plain old telephone service), etc. It goes without saying that communication may be performed in this way. In particular, the communication unit may transmit/receive various data for determining a user state by performing communication with external devices. Alternatively, when the user state analysis is performed in the external server, the communication unit 140 may transmit the collected learning data to the external server and receive the analysis result of the learning data.

The memory 150 may store, for example, a command or data for controlling at least one component of the coding robot 100 . According to an embodiment, the memory 150 may store software and/or a program. The program may include, for example, a kernel, middle-ware, an application programming interface (API) and/or an application program (or "application"). At least a part of the kernel, middleware, or API may be referred to as an operating system (OS). The kernel may, for example, control or manage system resources used to execute operations or functions implemented in other programs. In addition, the kernel may provide an interface capable of controlling or managing system resources when accessing individual components of the coding robot 100 from middleware, API, or application programs.

The middleware may, for example, play an intermediary role so that an API or an application program can communicate with the kernel to exchange data. Also, the middleware may process one or more work requests received from the application program according to priority. For example, the middleware may give priority to use the system resource of the coding robot 100 to at least one of the application programs, and process the one or more work requests. API is an interface for an application program to control functions provided by the kernel or middleware. For example, at least one interface or function (eg, command) for file control, window control, image processing, or character control. may include

Also, the memory 150 may include at least one of an internal memory and an external memory. Built-in memory includes, for example, volatile memory (such as DRAM, SRAM, or SDRAM), non-volatile memory (such as one time programmable ROM (OTPROM), PROM, EPROM, EEPROM, mask ROM, flash ROM, flash memory) , a hard drive, or a solid state drive (SSD), the external memory may include a flash drive, for example, a compact flash (CF), a secure digital (SD), It may include Micro-SD, Mini-SD, xD (extreme digital), MMC (multi-media card), memory stick, etc. The external memory may be functionally or physically connected to the coding robot 100 through various interfaces. can

The sensing unit 160 is configured to obtain information about the surroundings of the coding robot 100 . As described above, the sensing unit 160 may include various sensors, and may acquire information sensed by the sensors as user data. . The sensing unit 160 may include various sensors such as an infrared sensor, a pressure sensor, an illuminance sensor, a humidity sensor, and a gyro sensor. For example, when the sensing unit 160 includes an infrared sensor, data on the presence or absence of the user may be acquired as user data or data on the user's body temperature may be acquired through the infrared sensor. Alternatively, when the sensing unit 160 includes a pressure sensor, data on physical contact with the user may be acquired as user data. Alternatively, when the coding robot 100 includes an illuminance sensor, a humidity sensor, and the like, information on humidity and illuminance related to the user's surrounding environment may be acquired as user data.

The processor 130 may control the overall operation of the coding robot 100 by using various programs stored in the memory 150 .

The processor 130 may be composed of a RAM 131 , a ROM 132 , a graphic processing unit 133 , a main CPU 134 , and first to n interfaces 135-1 to 135-n and a bus 136 . can In this case, the RAM 131 , the ROM 132 , the graphic processing unit 133 , the main CPU 134 , the first to n interfaces 135 - 1 to 135 -n, etc. may be connected to each other through the bus 136 .

The RAM 131 stores OS and application programs. Specifically, when the coding robot 100 is booted, the OS may be loaded into the RAM 131 , and various application data selected by the user may be loaded into the RAM 131 .

The ROM 132 stores an instruction set for system booting, and the like. When a turn-on command is input and power is supplied, the main CPU 134 copies the OS stored in the memory 150 to the RAM 131 according to the command stored in the ROM 132, and executes the OS. to boot the system. When booting is completed, the main CPU 134 copies various application programs stored in the memory 150 to the RAM 131 , and executes the application programs copied to the RAM 131 to perform various operations.

The graphic processing unit 133 generates a screen including various objects, such as a graphical user interface (GUI) item, an image, text, and the like, by using a calculation unit (not shown) and a rendering unit (not shown). Here, the calculation unit may be configured to calculate attribute values such as coordinate values, shape, size, color, etc. of each object to be displayed according to the layout of the screen using the control command received from the input unit 140 . In addition, the rendering unit may be configured to generate screens of various layouts including objects based on the attribute values calculated by the operation unit. The screen generated by the rendering unit may be displayed in the display area of the display 130 .

The main CPU 134 accesses the memory 150 and performs booting using the OS stored in the memory 150 . In addition, the main CPU 134 performs various operations using various programs, contents, data, etc. stored in the memory 150 .

The first to n-th interfaces 135-1 to 135-n are connected to the various components described above. One of the first to n interfaces 135 - 1 to 135 - n may be a network interface connected to an external device through a network.

Hereinafter, various embodiments according to the present invention will be described with reference to FIGS. 3 to 5 .

3 is a flowchart illustrating a control method of a coding robot according to an embodiment of the present invention.

In step S110, the coding robot 100 may obtain information of the recognition target.

In an embodiment, the information of the recognition target may be acquired by the above-described input unit 110 or sensing unit 160 .

In step S120 , the coding robot 100 may acquire control data based on the acquired information of the subject to be recognized.

In step S130 , the coding robot 100 may determine whether the control data matches preset control information.

As an embodiment, the coding robot 100 may store control information matched with the control data in advance in the memory 150 . For example, when the control data is an angry emotion, the memory 150 may store operation information of the coding robot 100 corresponding to the angry emotion as a control command.

In step S140 , the coding robot 100 may perform an operation corresponding to a preset control command when the control data matches preset control information.

In step S150, the coding robot 100, when the control data does not match the preset control information, may analyze the control data to match any one of the preset control commands.

That is, even if there is no control command matched to the control data acquired based on the information of the person to be recognized, the coding robot 100 analyzes the control data and matches it with at least one control command among the pre-stored control commands. have.

Meanwhile, in the present invention, the control data may refer to at least one of emotion data and voice data of a subject to be recognized.

On the other hand, according to various embodiments of the present invention, when the control data is the emotion data of the subject to be recognized, the coding robot 100 may obtain the control data through various methods.

Specifically, the coding robot 100 may obtain a photographed image of the face of the recognition target. Thereafter, the coding robot 100 may acquire model variables based on the face photographed image. Thereafter, the coding robot 100 may determine whether the recognition target matches the pre-registered user's face based on the acquired model variables. Thereafter, when the recognition target matches the pre-registered user, the coding robot 100 may acquire a feature value corresponding to the photographed face image based on the photographed face image. Thereafter, the coding robot 100 may determine a face part including the eyes, nose, and mouth of the recognition target based on the acquired feature values. Thereafter, the coding robot 100 may determine the emotion of the subject to be recognized based on the determined face part.

In one embodiment, the coding robot 100 may recognize the user of the coding robot based on libraries such as OpenCV, Tensorflow, and Dlib, and determine the emotion of the recognition target.

At this time, OpenCV is a programming library for real-time computer vision, Robomi can handle cameras and process camera images, and TensorFlow is an open source software library for data flow programming for various tasks. It is also used in machine learning applications such as networks and can be used for face recognition and object recognition. Dlib is a general-purpose cross-platform software library written in the programming language C++, and the coding robot 100 is one of Dlib's facial landmarks (eyes, nose, mouth). back area) recognition function to perform emotion recognition.

Specifically, the coding robot 100 may perform face recognition based on the various libraries described above. To this end, the coding robot 100 may include a pre-written coding block.

Specifically, the face storage block may store the recognized face as an image file using OpenCV.

In addition, the face model generation block can create model variables using the Tensorflow library based on the stored face photos. The generated model variables can be used in the face recognition block.

Also, the face recognition block can calculate how much the face matches the registered face by using the model variable.

In addition, the facial emotion recognition block uses Dlib to indicate the vertices of the face as dots, as shown in FIG. 4 . Using the position values of these points, emotions can be inferred by calculating the size of the eyes, the angle of the eyebrows, and the size of the mouth. In addition, since the facial emotion recognition model can calculate the positions of the eyes, nose, and mouth, it goes without saying that an image can be overlaid on the positions of the eyes, nose, and mouth.

According to an embodiment, the coding robot 100 may determine the facial expression of the recognition target by using the following code.

[Algorithm 1]

def detect_face_elem(o_frame):

global detector

global predictor

if detector is None or predictor is None:

if "dlib" not in sys.modules:

import dlib

detector = dlib.get_frontal_face_detector()

predictor = dlib.shape_predictor('shape_predictor_68_face_landmarks.dat')

frame = copy. deepcopy(o_frame)

gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)

line_brow_x = []

line_brow_y = []

faces = detector(gray)

for face in faces:

#200107

x1 = face.left()

y1 = face.top()

x2 = face.right()

y2 = face.bottom()

cv2.rectangle(frame, (x1, y1), (x2, y2), (0, 255, 0), 1)

landmarks = predictor(gray, face)

## "Jaw"

for n in range(0, 17):

x = landmarks.part(n).x

y = landmarks.part(n).y

cv2.circle(frame, (x, y), 1, (19, 199, 109), -1)

## "Nose"

for n in range(27, 35):

x = landmarks.part(n).x

y = landmarks.part(n).y

cv2.circle(frame, (x, y), 1, (180, 42, 220), -1)

#cv2.putText(frame, str(n), (x, y), cv2.FONT_HERSHEY_SIMPLEX, 0.35, (0, 255, 255), 2, 4)

#31-34, 27-33

## "Left Eye"

for n in range(42, 48):

x = landmarks.part(n).x

y = landmarks.part(n).y

cv2.circle(frame, (x, y), 1, (163, 38, 32), -1)

## "Right Eye"

for n in range(36, 42):

x = landmarks.part(n).x

y = landmarks.part(n).y

cv2.circle(frame, (x, y), 1, (158, 163, 32), -1)

#cv2.putText(frame, str(n), (x, y), cv2.FONT_HERSHEY_SIMPLEX, 0.4, (0, 0, 255), 2, 4)

## "Left Eyebrow"

for n in range(22, 27):

x = landmarks.part(n).x

y = landmarks.part(n).y

cv2.circle(frame, (x, y), 1, (168, 100, 168), -1)

## "Right Eyebrow"

for n in range(17, 22):

x = landmarks.part(n).x

y = landmarks.part(n).y

cv2.circle(frame, (x, y), 1, (230, 159, 23), -1)

## "Mouth"

for n in range(48, 68):

x = landmarks.part(n).x

y = landmarks.part(n).y

cv2.circle(frame, (x, y), 1, (79, 76, 240), -1)

#cv2.putText(frame, str(n), (x, y), cv2.FONT_HERSHEY_SIMPLEX, 0.35, (0, 0, 255), 2, 4)

face_width = x2 - x1

face_height = y2 - y1

# Analysis of positional relationship of arbitrary n points as the basis for facial recognition

mouth_width = (landmarks.part(54).x - landmarks.part(48).x) / face_width # The amount of mouth open

mouth_height = (landmarks.part(66).y - landmarks.part(62).y) / face_width # The amount of mouth open

# calc center pos

face_x = face_width / 2 + x1

face_y = face_height / 2 + y1

face_pos = [face_x, face_y, face_width, face_height, x1, y1]

mouth_x = (landmarks.part(54).x - landmarks.part(48).x) / 2 + landmarks.part(48).x

mouth_y = (landmarks.part(66).y - landmarks.part(62).y) / 2 + landmarks.part(62).y

mouth_pos = [mouth_x, mouth_y, landmarks.part(54).x - landmarks.part(48).x, landmarks.part(66).y - landmarks.part(62).y, landmarks.part(48). x, landmarks.part(50).y]

nose_x = (landmarks.part(34).x - landmarks.part(31).x) / 2 + landmarks.part(31).x

nose_y = (landmarks.part(33).y - landmarks.part(27).y) / 2 + landmarks.part(27).y

nose_pos = [mouth_x, mouth_y, landmarks.part(34).x - landmarks.part(31).x, landmarks.part(33).y - landmarks.part(27).y, landmarks.part(31). x, landmarks.part(27).y]

eye_x1 = (landmarks.part(39).x - landmarks.part(36).x) / 2 + landmarks.part(36).x

eye_y1 = (landmarks.part(41).y - landmarks.part(37).y) / 2 + landmarks.part(37).y

eye_pos1 = [eye_x1, eye_y1, landmarks.part(39).x - landmarks.part(36).x, landmarks.part(41).y - landmarks.part(37).y, landmarks.part(36). x, landmarks.part(37).y]

eye_x2 = (landmarks.part(45).x - landmarks.part(42).x) / 2 + landmarks.part(42).x

eye_y2 = (landmarks.part(47).y - landmarks.part(43).y) / 2 + landmarks.part(43).y

eye_pos2 = [eye_x2, eye_y2, landmarks.part(45).x - landmarks.part(42).x, landmarks.part(47).y - landmarks.part(43).y, landmarks.part(42). x, landmarks.part(43).y]

# Analyze the degree of choosing eyebrows and the degree of frown with the 10 characteristics above the eyebrows

brow_sum = 0 # sum of heights

frown_sum = 0 # sum of both eyebrow distances

for j in range(17, 21):

brow_sum += (landmarks.part(j).y - y1) + (landmarks.part(j + 5).y - y1)

frown_sum += landmarks.part(j + 5).x - landmarks.part(j).x

line_brow_x.append(landmarks.part(j).x)

line_brow_y.append(landmarks.part(j).y)

tempx = np.array(line_brow_x)

tempy = np.array(line_brow_y)

z1 = np.polyfit(tempx, tempy, 1) # first-order straight line synthesis

brow_k = -round(z1[0], 3) # The slope of the sum curve and the slope of the actual eyebrow are opposite

eye_sum = (landmarks.part(41).y - landmarks.part(37).y + landmarks.part(40).y - landmarks.part(38).y +

landmarks.part(47).y - landmarks.part(43).y + landmarks.part(46).y - landmarks.part(44).y)

eye_height = (eye_sum / 4) / face_width

return frame, face_pos, eye_pos1, eye_pos2, nose_pos, mouth_pos, mouth_width, mouth_height, eye_height, brow_k

return frame, [], [], [], [], [], -1, -1, -1, -1

In addition to the above-described various coding blocks, the coding robot 100 may include various coding blocks. For example, the object recognition block stores coding instructions that can recognize about 32 objects and can be operated based on Tensorflow. Alternatively, the sign recognition block may be composed of a coding block for a user to directly learn a sign and a block for recognizing a sign based on the learned data. Alternatively, the lane recognition block is a coding block for recognizing two consecutive lines on the screen, measuring the distance between the line and the car, and moving to the middle of the lane if it is biased to one side to prevent departure. Alternatively, it goes without saying that a number recognition block for recognizing numbers 0 to 9 and a letter recognition block for recognizing English alphabets may be applied according to various embodiments of the present invention.

Thereafter, the coding robot 100 may acquire control data based on the determined emotion.

On the other hand, the coding robot 100 may be controlled according to a control command corresponding to the emotion determined by the above-described embodiment.

As an embodiment, the coding robot 100 may acquire coordinate information on a two-dimensional vector space corresponding to the determined emotion. Thereafter, the coding robot 100 may obtain a control command corresponding to coordinate information on a vector space.

For example, the 2D vector space may be the 2D space shown in FIG. 5 . In this case, of course, the two-dimensional space may be displayed on the original emotion model of J. Russel. The horizontal axis of J. Russel's prototype emotion model indicates whether emotions are positive or negative, and the vertical axis indicates activity. That is, the emotion of the recognition target determined by the coding robot 100 may be obtained as one piece of coordinate information on the circular emotion model of J. Russel shown in FIG. 5 .

The coding robot 100 may store a control command corresponding to each coordinate information on the circular emotion model of J. Russel. According to an embodiment, if the obtained coordinate information exists in the first quadrant, the coding robot 100 performs a first control command, and if it exists in the second quadrant, the coding robot 100 performs a second control command and, if present in the third quadrant, the coding robot 100 may execute the third control command, and if present in the fourth quadrant, the coding robot 100 may perform the fourth control command.

In one embodiment, the first to fourth control commands may be commands related to movement of the coding robot 100 .

On the other hand, according to another embodiment of the present invention, the coding robot 100 may obtain information about the change in emotion of the recognition subject based on the photographed image of the subject's face.

Thereafter, the coding robot 100 may obtain first coordinate information corresponding to the first emotion and second coordinate information corresponding to the second emotion when the emotion of the subject to be recognized is changed from the first emotion to the second emotion. have.

Thereafter, the coding robot 100 may obtain a movement command corresponding to a vector value corresponding to the first coordinate and the second coordinate as a control command.

For example, as shown in FIG. 5 , when the emotion of the recognition target is changed from the first coordinates 510 to the second coordinates 520 , the coding robot 100 uses the first coordinates 510 as a starting point and The second coordinate 520 may be moved in a vector direction as an end point.

According to an embodiment, the initial movement speed of the coding robot 100 is proportional to the distance from the origin to the first coordinate 510 , and the movement acceleration of the coding robot 100 is the second coordinate from the first coordinate 510 . It may be proportional to the distance to 520 . For example, when the first coordinate 510 is (a,b) and the second coordinate is (c,d), the initial movement speed V and the movement acceleration A may be expressed by the following equations.

In this case, k and l may be arbitrary positive constants.

Meanwhile, according to another embodiment of the present invention, when the control data is the voice data of the recognition target, the coding robot 100 may acquire the control data through various methods.

Specifically, the coding robot 100 may generate a motion control block including a motion speed control command and a motion repetition number control command for the coding robot based on voice data.

Thereafter, the coding robot 100 may acquire at least one of motion speed information and motion repetition number information based on the voice data.

Thereafter, the coding robot 100 may acquire a control command corresponding to at least one of the motion speed and the number of repetitions of the coding robot based on at least one of the motion speed information and the motion repetition number information.

At this time, when the voice data matches the preset control information, the coding robot 100 controls the motion speed or the number of repetitions of the coding robot based on the motion speed information or motion repetition number information corresponding to the preset control information. It can be obtained by command.

For example, if the voice data is "moving quickly to the right" and there is a pre-stored control command corresponding to it (for example, moving at a speed of 3 steps to the right), the coding robot 100 responds to the pre-stored control command. can be controlled accordingly.

However, when the voice data does not match the preset control information, the coding robot 100 may generate syllable spacing information based on the voice data and generate motion speed information in response to the syllable spacing information.

For example, if the voice data is "moving quickly to the right" and there is no preset control information corresponding to "quickly", the coding robot 100 generates syllable spacing information based on the voice data, Motion speed information may be generated in response to the syllable interval information, and a movement speed control command corresponding to the generated motion speed information may be generated.

Specifically, if the motion speed information is not found in the voice data, the coding robot 100 generates syllable interval information based on the voice data, generates motion speed information in response to the syllable interval information, and generates motion speed information It is possible to generate a movement speed control statement corresponding to .

Specifically, the coding robot 100 calculates the first time difference between the end time of one syllable and the start time of the next syllable of any one syllable among the syllables included in the voice data, and the average time difference of the plurality of first time differences can be generated as syllable spacing information.

Thereafter, the coding robot 100 may calculate the time difference ratio of the average time difference indicated by the syllable interval information compared to the reference time difference, and apply the time difference ratio to the basic movement speed to generate the motion speed information.

For example, the coding robot 100 calculates the time difference ratio as "120%" when the average time difference is "12 msec", the reference time difference is "10 msec", and the basic movement speed is "1x speed", and the basic movement By applying the time difference ratio "120%" to the speed "1x speed", it is possible to generate the operating speed information at "1.2x speed".

Meanwhile, when the voice data does not match the preset control information, the coding robot 100 may generate signal strength information based on the voice data and generate motion repetition number information in response to the signal strength information.

Specifically, when the motion repetition number information is not included in the voice data, the coding robot 100 generates signal strength information based on the voice signal, generates motion repetition number information in response to the signal strength information, and generates the generated An operation repetition number control statement corresponding to the operation repetition number information may be generated.

For example, the coding robot 100 generates signal strength information based on the voice signal if the motion repetition number information is not retrieved from the voice data, generates motion repetition number information in response to the signal strength information, and generates the generated motion An operation repetition number control statement corresponding to the repetition number information may be generated.

Specifically, the coding robot 100 may generate the maximum signal strength among the signal strengths according to time of the voice signal as signal strength information.

Thereafter, the coding robot 100 may calculate the signal strength ratio of the maximum signal strength indicated by the signal strength information to the reference signal strength, and apply the signal strength ratio to the basic motion repetition number to generate the motion repetition number information.

For example, the coding robot 100 calculates the signal strength ratio as "200%" when the maximum signal strength is "200DB", the reference signal strength is "100DB", and the basic operation repetition number is "1 time", , by applying a signal strength ratio of “200%” to the basic number of repetitions of motion “1 time”, information on the number of repetitions of motion may be generated as “2 times”. In this case, the operation repetition number information may be rounded to generate a natural number.

Based on the generated motion speed information or the number of motion repetitions, the motion speed or motion repetition number of the coding robot may be acquired as a control command.

Meanwhile, according to another embodiment of the present invention, when the control command is a control command for moving the coding robot, the coding robot 100 may recognize an object around the coding robot. Thereafter, when the recognized object matches the pre-stored sign, the coding robot 100 may acquire control information corresponding to the pre-stored sign. The coding robot 100 may recognize a lane around the coding robot. The coding robot 100 moves along the recognized lane, but may move corresponding to the control information.

In this case, the coding robot 100 may obtain information on two consecutive line segments based on an image input to the coding robot 100 , and may obtain a lane area based on the obtained line segment information.

Thereafter, the coding robot 100 may obtain distance information between the obtained line segment and the coding robot. Thereafter, the coding robot 100 may control the coding robot based on lane area and distance information.

For this purpose, it goes without saying that at least one of the above-described object recognition block, sign recognition block, lane recognition block, number recognition block, and letter recognition block may be used.

On the other hand, the components of the present invention may be implemented as a program (or application) to be executed in combination with a computer, which is hardware, and stored in a medium. Components of the present invention may be implemented as software programming or software components, and similarly, embodiments may include various algorithms implemented as data structures, processes, routines, or combinations of other programming constructs, including C, C++ , Java, assembler, etc. may be implemented in a programming or scripting language. Functional aspects may be implemented in an algorithm running on one or more processors.

As mentioned above, although embodiments of the present invention have been described with reference to the accompanying drawings, those skilled in the art to which the present invention pertains can realize that the present invention can be embodied in other specific forms without changing its technical spirit or essential features. you will be able to understand Therefore, it should be understood that the embodiments described above are illustrative in all respects and not restrictive.

100: coding robot

Claims (7)

In the control method of the coding robot,
obtaining, by the coding robot, information of a subject to be recognized;
obtaining, by the coding robot, control data based on the acquired information of the subject to be recognized;
determining, by the coding robot, whether the control data matches preset control information;
performing, by the coding robot, an operation corresponding to the preset control command when the control data matches preset control information; and
Including, by the coding robot, when the control data does not match the preset control information, analyzing the control data and matching any one of the preset control commands;
The control data is at least one of emotional data and voice data of the subject to be recognized,
When the control command is a control command for moving the coding robot,
The control method is
recognizing, by the coding robot, an object around the coding robot;
obtaining, by the coding robot, control information corresponding to the pre-stored sign when the recognized object matches a pre-stored sign;
recognizing, by the coding robot, a lane around the coding robot; and
moving the coding robot along the recognized lane, moving in response to the control information; including,
Recognizing the lane comprises:
obtaining information on two consecutive line segments based on the image input to the coding robot, and obtaining a lane area based on the obtained line segment information;
obtaining distance information between the obtained line segment and the coding robot; and
controlling the coding robot based on the lane area and the distance information; A control method comprising a. According to claim 1,
When the control data is the emotional data of the recognition target,
Obtaining the control data includes:
obtaining, by the coding robot, a photographed image of the face of the recognition target;
obtaining, by the coding robot, a model variable based on the face photographed image;
determining, by the coding robot, whether the recognition target matches the face of a pre-registered user based on the acquired model variables;
obtaining, by the coding robot, a feature value corresponding to the photographed face image based on the photographed face image, when the recognition target matches the previously registered user;
determining, by the coding robot, a face part including the eyes, nose, and mouth of the recognition target based on the acquired feature values;
determining, by the coding robot, the emotion of the person to be recognized based on the determined face part; and
A control method comprising a; obtaining, by the coding robot, the control data based on the determined emotion. 3. The method of claim 2,
The step of performing an operation corresponding to the control command includes:
obtaining, by the coding robot, coordinate information on a two-dimensional vector space corresponding to the determined emotion;
obtaining a control command corresponding to coordinate information on the vector space; and
performing an operation corresponding to the control command; A control method comprising a. 3. The method of claim 2,
The step of performing an operation corresponding to the control command includes:
obtaining, by the coding robot, emotion change information of the subject to be recognized based on a photographed image of the subject's face;
obtaining, by the coding robot, first coordinate information corresponding to the first emotion and second coordinate information corresponding to the second emotion when the emotion of the person to be recognized is changed from a first emotion to a second emotion; and
and obtaining, by the coding robot, a movement command corresponding to a vector value corresponding to the first coordinate and the second coordinate as the control command. delete According to claim 1,
When the control data is the voice data of the recognition target, the control method,
generating, by the coding robot, a motion control block including a motion speed control command and a motion repetition number control command for the coding robot based on the voice data;
obtaining, by the coding robot, at least one of motion speed information and motion repetition number information based on the voice data; and
obtaining, by the coding robot, a control command corresponding to at least one of the operation speed and the number of repetitions of the coding robot based on at least one of the operation speed information and the operation repetition number information; A control method comprising a. 7. The method of claim 6,
The step of obtaining a control command corresponding to at least one of the operation speed and the number of repetitions of the coding robot comprises:
When the voice data matches the preset control information, the coding robot controls the motion speed or the number of repetitions of the coding robot based on the motion speed information or motion repetition number information corresponding to the preset control information. obtaining with a command;
generating, by the coding robot, syllable spacing information based on the voice data when the voice data does not match preset control information, and generating the motion speed information in response to the syllable spacing information;
generating, by the coding robot, signal strength information based on the voice data when the voice data does not match preset control information, and generating the operation repetition number information in response to the signal strength information; and
Acquiring the operation speed or the number of repetitions of the coding robot as the control command based on the generated operation speed information or the number of repetitions.

Images