KR101306608B1 - Method for providing 3-dimensional display by using smart terminals and robot devices, and 3-dimensional display system using the same - Google Patents

Abstract

The present invention relates to a technology for providing a three-dimensional image by using a smart terminal and a robot device, in particular after taking a photograph of the face of a specific person in a smart terminal (for example, smart phone, smart pad) and converts it into a three-dimensional image robot The present invention relates to a method and system for providing a device and utilizing the three-dimensional display as a face screen in an operation process. According to an embodiment of the present invention, a method of providing a 3D display using a smart terminal connected to a network and a robot apparatus includes: a first step of extracting, by a smart terminal, a face image from a photo image; Generating a three-dimensional face image by using the extracted face image in the smart terminal; And a third step of providing the smart device to an external robot device so that the generated 3D image can be used as a face screen of the robot device.

Description

Method of providing 3D image using smart terminal and robot device and 3D image providing system using the same {METHOD FOR PROVIDING 3-DIMENSIONAL DISPLAY BY USING SMART TERMINALS AND ROBOT DEVICES, AND 3-DIMENSIONAL DISPLAY SYSTEM USING THE SAME}

The present invention relates to a technology for providing a three-dimensional image by using a smart terminal and a robot device, in particular after taking a photograph of the face of a specific person in a smart terminal (for example, smart phone, smart pad) and converts it into a three-dimensional image robot The present invention relates to a method and system for providing a device and utilizing the three-dimensional display as a face screen in an operation process.

In general, smart terminals (eg, smart phones, smart pads) provided schedule management based on voice calls, but as communication technologies have advanced, data communication including the Internet has become possible. In addition, as the technology of electronic devices has evolved, cameras are embedded in most smart terminals, and smart terminals can gradually function as small computers. Such a smart terminal that can perform various tasks is called a smart terminal.

In other words, the smart terminal has become an electronic device capable of performing various functions such as a computer in a hand for basic telephone functions. Such a smart terminal may perform a function of a phone, and additionally, communication with at least one other wireless medium is possible, and a variety of other services are possible through a camera, such as taking a picture and a video.

On the other hand, with the recent development of robot technology, robot devices have been used in various fields. These robotic devices take a variety of forms, from expensive robotic devices used in high-tech industries to children's toy robots.

Children's toys or learning robots are also available. These robotic devices are low-cost robotic devices that make sounds or movements of the robots, such as walking or arm movements. In another form, the robots have functions such as playing pictures or videos. have.

On the other hand, efforts have been made to link such play or learning robotic devices with smart terminals. This necessity is because not only increases the utilization of the smart terminal, but also increases the role of the robot device for play or learning. In addition, because of the increased role of the robot device can be achieved in various forms of education.

Therefore, various functions for linking these types of electronic devices and smart terminals are further required.

In the present invention, a photograph taken by a smart terminal is converted into a 3D image through software and provided to the robot device, and the robot device provides a technology for appropriately utilizing such a 3D image during operation.

In addition, the present invention transmits a three-dimensional image taken by the smart terminal to the robot device or an electronic device having a monitor and a speaker and a communication device using the smart terminal, and detects the movement of the smart terminal to provide a three-dimensional image to the movement. It provides a method and a system.

Method for providing a three-dimensional image by using a smart terminal and a robot device according to the present invention for achieving the above object, to provide a three-dimensional display using the smart terminal 100 and the robot device 200. A first step of the smart terminal extracting a face image from the photo image; Generating a 3D face image by using the extracted face image in the smart terminal; And a third step of providing the external 3D image to the robot apparatus so that the smart terminal can use the generated 3D image as the face screen of the robot apparatus.

At this time, the first step in the three-dimensional image providing method of the present invention, the step of photographing the face of the object using the camera unit of the smart terminal; And extracting a face image from the photographed photo image. In addition, the second step may be configured to apply the CCA transformation algorithm to the extracted face image to generate a three-dimensional face image.

In addition, the three-dimensional image providing method according to the invention, the fourth step of detecting the motion of the smart terminal through the motion detection unit of the smart terminal; A fifth step of regenerating the 3D image in response to the motion by the smart terminal; And a sixth step of providing the reproduced three-dimensional image to the external robot device so that the smart terminal can use the reproduced three-dimensional image as the face screen of the robot device.

In addition, the three-dimensional image providing method according to the present invention comprises a seventh step of receiving the text data from the smart terminal from the user; An eighth step of the smart terminal obtaining phoneme data corresponding to the text data from the memory; A ninth step of the smart terminal generating motion information of a mouth of the 3D image in response to the phoneme data; The smart terminal may transmit the mouth movement information of the 3D image to the robot device to provide the robot device to change the face screen in response to the mouth movement information on the face screen.

In addition, the three-dimensional image providing method according to the present invention, the smart terminal is an eleventh step of receiving a voice signal from the user; A twelfth step of the smart terminal recognizing the text from the voice signal; A thirteenth step of obtaining phoneme data corresponding to the recognized text from the memory; A fourteenth step of the smart terminal generating motion information of a mouth of the 3D image in response to the phoneme data; The smart terminal may transmit the mouth movement information of the 3D image to the robot device to provide the robot apparatus to change the face screen in response to the mouth movement information on the face screen.

The system for providing a 3D image using the smart terminal and the robot device according to the present invention for achieving the above object, the three-dimensional display using a smart terminal 100 and the robot device 200 connected by a network A computer readable recording medium installed in a smart terminal for providing, a computer readable recording medium recording a smart terminal 3D image providing program for performing a 3D image providing method according to the present invention; A robot device communication unit 210 for performing data communication with the smart terminal 100 on which the program is executed, an image providing unit 230 for displaying a 3D face image provided from the smart terminal through the robot device communication unit, and And a robot device 200 including a robot device controller 220 for controlling the output of the 3D image received from the smart terminal to the image providing unit 230.

In the three-dimensional image providing system of the present invention, the smart terminal three-dimensional image providing program recorded on the recording medium is a seventh step in which the smart terminal is provided with the text data from the user, and the smart terminal to store the phoneme data corresponding to the text data An eighth step of acquiring the information from the mobile terminal; and a ninth step of the smart terminal generating mouth motion information of the 3D image in response to the phoneme data; and the smart terminal transmitting the mouth motion information of the 3D image to the robot device. Is further configured to perform a tenth step of providing the user to change the face screen in response to the mouth movement information on the face screen, and the image providing unit 230 of the robot device 200 is provided from the smart terminal through the robot device communication unit. It may be configured to deform the image of the mouth portion in the three-dimensional face image corresponding to the provided mouth movement information. .

In addition, the smart terminal three-dimensional image providing program recorded on the recording medium is the eleventh step of receiving a voice signal from the smart terminal, the twelfth step of recognizing the text from the voice signal from the smart terminal, and the recognized text A thirteenth step of acquiring corresponding phoneme data from the memory; a fourteenth step of the smart terminal generating motion information of the mouth of the 3D image in response to the phoneme data; and a smart terminal of the mouth motion information of the 3D image by the robot. It is configured to perform the fifteenth step of transmitting to the device so that the robot device can change the face screen in response to the mouth movement information on the face screen, the image providing unit 230 of the robot device 200, the robot device Display the image of the mouth part in the 3D face image corresponding to the mouth motion information provided from the smart terminal through the communication unit It can be configured to.

In the present invention, by taking a picture by using a smart terminal, and converts the photographed picture to three-dimensional information and then provide the converted data to the electronic device equipped with a robot device or a monitor, a speaker and a communication device 3 produced in the smart terminal It can provide a dimensional image. In addition, the smart terminal can detect the movement of the smart terminal to control the movement of the produced 3D image. In addition, it is possible to transmit the voice or text together, so that the 3D image can provide the voice together.

1 is a conceptual diagram of a system for providing a three-dimensional image from the smart terminal to the electronic device in accordance with the present invention.
2 is a functional internal block diagram of a smart terminal according to an embodiment of the present invention.
3 is a functional block diagram of a robot device to which the present invention is applied.
4 is a control flowchart for providing a three-dimensional image from the smart terminal to the robot device according to an embodiment of the present invention.
FIG. 5A is an exemplary diagram illustrating a picture taken in a screen providing unit 170 of a smart terminal, a 3D image, and a user's selection list icon according to an embodiment of the present invention.
5B is a diagram illustrating a process of generating a 3D image from a captured image.
6 is a control flowchart of an image display mode when providing a 3D image from a smart terminal to a robot device according to an embodiment of the present invention.
7 is an exemplary view for explaining a process of detecting the movement of the three-dimensional image according to the movement of the smart terminal according to the present invention.
8 is a control flowchart for receiving and displaying three-dimensional data in a robot device according to the present invention, and for receiving and providing a voice and sound signal to a user.
9 is an exemplary view for explaining the movement of the three-dimensional image data when providing audio data using audio or text in addition to the three-dimensional image in accordance with the present invention.
10 is a conceptual diagram for explaining the overall operation in the case of transmitting a three-dimensional image and voice from the smart terminal to the robot device according to the present invention.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will now be described with reference to the accompanying drawings. It should be noted that the drawings of the present invention attached hereto are provided for the purpose of helping understanding of the present invention, and the present invention is not limited to the shape or the arrangement exemplified in the drawings of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS The above and other features and advantages of the present invention will be more apparent from the following description taken in conjunction with the accompanying drawings, in which: FIG.

1 is a conceptual diagram of a system for providing a 3D image from the smart terminal to the electronic device according to the present invention. Hereinafter, the configuration and operation of the system to which the present invention is applied will be described with reference to FIG. 1.

The system according to the present invention includes an electronic device 200 for communication between the smart terminal 100 and the smart terminal 100. Here, the electronic device is an electronic device having a communication unit, a monitor, and a speaker, which will be referred to as a robot device for convenience of description in the following description. In addition, the smart terminal 100 will be referred to as a "smart terminal" for convenience of description.

Smart terminal 100 is basically capable of voice or data communication through the base station 11 to communicate with the wireless network (10). In addition, the smart terminal 100 can take a picture according to the present invention, can generate a three-dimensional image using the photographed picture. At this time, only the face is extracted from the captured picture. In this case, a method of extracting a face from the smart terminal 100 may use a 'Haar Face Detecting Algorithm'. In the present invention, the bottom detection algorithm is illustrated as an example, and other algorithms may be used.

As such, when the face is extracted from the picture, the smart terminal 100 converts the extracted face data into three-dimensional face data. The transformation into 3D face data may use, for example, a Collaborative Construction Algorithm (CCA) transformation algorithm. In addition, other algorithms may be used, but in consideration of the operation speed of the smart terminal 100, it is preferable to use an algorithm that can be processed at a high speed in the smart terminal 100.

Through this process, the three-dimensional face data generated by the smart terminal 100 is transmitted to the robot device 200 through a predetermined communication method according to the present invention. The communication method may be any of wireless and wired methods. For example, when using wireless communication, one of various methods, such as Wi-Fi or Bluetooth, which is widely used, may be used. In the case of a wired line, various methods such as short range communication or cable communication for direct connection may be used.

As such, when the data generated by the smart terminal 100 is transmitted to the robot device 200, the robot device 200 displays and displays the received data. When the text or voice is input in the smart terminal 100, the text may be converted into a pre-stored voice format or the user's voice may be converted as it is or according to a specific format to the robot apparatus 200. The robot device 200 then receives this data and provides it to the user through the speaker.

In addition, the smart terminal 100 may recognize the motion of the user moving the smart terminal 100 and convert the motion information thereof into a motion image of the face to provide the robot device 200. Then, the robot device 200 receives the motion image information of the face provided from the smart terminal 100 and displays the movement of the face through a display device such as a monitor. Details thereof will be described in more detail with reference to the accompanying drawings.

2 is a functional internal block diagram of a smart terminal according to an embodiment of the present invention. Hereinafter, a block configuration and operation of a smart terminal according to the present invention will be described with reference to FIG. 2.

The smart terminal according to the present invention includes a wireless unit 110 having a first communication unit 111 and a second communication unit 112. The first communication unit 111 is a communication unit for transmitting and receiving voice or data to the wireless network 10 through the base station 11 when operating as a mobile terminal, the second communication unit 112 communicates with the robot device according to the present invention Communication unit for In FIG. 2, for convenience of description, it is assumed that the second communication unit 112 performs wireless communication.

The modem unit 120 encodes data to provide an electrical voice signal received from the control unit 130 or the speaker SPK to the first communication unit 111 or the second communication unit 112 included in the wireless unit 110. It modulates and demodulates and decodes the received data and provides the same to the controller 130 or the microphone (MIC).

The control unit 130 is a configuration for controlling the overall operation of the smart terminal 100, in particular, the control of the photographing to be provided to the robot device according to the present invention and extract the face from the photographed picture, and the extracted face data 3 Perform control to dimensional data. In addition, the controller 130 controls the modem unit 120 and the wireless unit 110 to convert the 3D image to move in response to the movement when the movement is detected and to provide the same to the robot apparatus.

In addition, when a voice is input, the controller 130 controls the modem unit 120 and the wireless unit 110 to convert the voice in a predetermined manner or to transmit the voice to the robot apparatus as it is. At this time, the shape of the mouth of the three-dimensional image is also converted to move the image information so as to transmit the robot device.

In addition, when the text is input, the controller 130 processes the text into voice data having a predetermined form, and controls the modem unit 120 and the wireless unit 110 to transmit the text to the robot device. At this time, the shape of the mouth of the 3D image is also converted to move the image information so as to move to the robot device.

The above operations may be performed by the controller 130 of the smart terminal or may be performed by the robot device 200. However, only motion information should be provided by the smart terminal 100. In the following description, it is assumed that the robot apparatus 200 merely assumes a form for receiving and processing data for convenience of description. However, it should be noted that some of the operations performed in the smart terminal may be configured to be performed in the robot device 200.

The motion detection unit 140 may be configured using, for example, a geomagnetic sensor or a rotation detection sensor, and detects the motion form of the smart terminal 100 when the user tilts or rotates the smart terminal 100. In accordance with this, a motion detection signal is generated and provided to the controller 130.

The memory unit 150 stores programs for controlling the smart terminal and various data stored by the user. In particular, the memory unit 150 controls to acquire a 3D image from a photograph taken according to the present invention and transmit the same to a robot device. It stores various data of the smart terminal for.

The camera unit 160 includes a camera lens and a photosensitive sensor for extracting photo data from light emitted through the lens, a configuration for converting information extracted from the sensor into photo or image information, a motor for driving the lens, and the like. can do. Since the configuration of the camera unit 160 may use well-known methods, a detailed description thereof will be omitted.

The screen providing unit 170 is a part for displaying images and various icons photographed by a smart terminal, and may be configured in various forms such as an LCD. In addition, the touch recognition unit 180 checks whether an icon provided by the screen providing unit 170 is selected and provides it to the controller 130 when a user's selection is made.

In addition, the smart terminal may include a connection connector for connecting with various keys or other devices, but will not be described further since it is not related to the present invention.

3 is a functional block diagram of a robot device 200 to which the technical idea of the present invention is applied. Hereinafter, a functional block and its operation of the robot apparatus 200 will be described with reference to FIG. 3.

The robot device 200 is provided with a communication unit 210 for communicating with the smart terminal 100, a controller 220 for controlling an operation of the robot device, an image providing unit 230 for providing an image, and a sound for providing an image. The sound providing unit 240 is included.

The communication unit 210 may select either a wireless or wired method as a device for performing data communication with the smart terminal 100. For example, when using wireless communication, the mobile communication network may be used directly, or one of various methods such as Wi-Fi or Brutus, which are widely used, may be used. In the case of a wired line, various methods such as short range communication or cable communication for direct connection may be used.

The communication unit 210 provides the control unit 220 when data is received from the smart terminal 100 in any manner.

The control unit 220 controls to be displayed to the user by outputting the three-dimensional image to the image providing unit 230 in the case of the image data based on the information received from the communication unit 210, and in the case of voice data sound providing unit 240 Control to let the user listen. In addition, when the communication with the smart terminal 100 is requested, the control unit 220 controls the communication unit 210 to set a predetermined channel, and controls the transmission and reception of data through the set channel.

The image providing unit 230 may be configured in various forms such as a monitor, LCD, or LED, and may be sufficient to display a three-dimensional image. The image providing unit 230 displays the image data provided from the control unit 220 for the user to see. In addition, the sound providing unit 240 may be configured as a speaker, and converts the voice or sound data output from the control unit 220 into an audible sound and outputs it.

4 is a control flowchart for providing a 3D image from the smart terminal to the robot device according to an embodiment of the present invention. Hereinafter, a process for providing 3D video and audio from a smart terminal to a robot device according to an embodiment of the present invention will be described in detail with reference to FIG. 4.

The controller 130 maintains the standby state in step S400. Here, the standby state refers to a state in which the user requests the operation of one of the functions of the smart terminal or waits for a wireless call. When the controller 130 proceeds to step S402, it is checked whether the user wants to establish communication with the robot device 200. If the user wants to communicate with the robot device 200, the controller 130 proceeds to step S404, otherwise it continues to maintain the standby state of step S400. 4 is a control flowchart for explaining a process for communicating with the robot apparatus 200 according to the present invention, and therefore, the request or call reception of another user will not be considered.

When the communication setting with the robot device 200 is required and the process proceeds to step S404, the controller 130 controls the second communication unit 112 of the wireless unit 110 to establish a channel with the robot device 200. The channel setting process may be set after step 410 described later. That is, it should be noted that the channel setting process may be moved to another step in the program production step in consideration of the battery usage time of the smart terminal, or the viewpoint may be changed according to the user's selection.

Thereafter, the controller 130 proceeds to step S406 and takes a picture of the image desired by the user through the camera unit 160, extracts face data from the captured picture, and converts it to 3D data. As described above, a method of extracting face data from a picture taken by a user may use a face detection algorithm, and converting the extracted face data into 3D data may use a CCA conversion algorithm.

After that, the controller 130 proceeds to step S408 and displays the captured image, the 3D image, and the selection icon by controlling the screen providing unit 170. This will be described with reference to [FIG. 5A].

5A is an exemplary diagram illustrating a photographed picture, a 3D image, and a user's selection list icon on the screen providing unit 170 of the smart terminal 100 according to an embodiment of the present invention.

As shown in FIG. 5A, the screen providing unit 170 of the smart terminal displays an area for displaying a picture 500 taken through a camera, and an area for image data 510 processed into a 3D image from the captured picture. And icon areas 520, 530, and 540 for the user to select. In addition, when all of the information cannot be displayed at a time, a scroll bar may be provided on the left or right side to scroll and display the screen. Meanwhile, the face picture utilized in the present invention is not limited to the picture taken by the smartphone 100 through the camera, and the picture provided from the outside via the Internet (eg, celebrities, cartoon characters, animals) is also stored in the memory. It should be construed as available.

Then, the process of generating the image data 510 processed into a 3D image from the photographed picture 500 may be configured as a 3D image through the process as shown in FIG. 5B.

5B is a diagram illustrating a process of generating a 3D image from a captured image. As described above, methods for processing 3D data from face data of a photograph taken or provided from the outside are already known techniques and thus will not be described in detail herein.

Referring again to FIG. 5A, FIG. 5A also illustrates whether the user should re-photograph after reviewing the processed data based on the photographed information and the photographed information, or whether the photographed image is generated. Icons 520, 530, and 540 for determining whether to transmit the 3D image to the robot apparatus 200 or cancel the communication itself with the robot apparatus 200 are displayed together. Therefore, in the following description, it is assumed that one of the above icons is selected.

The controller 130 checks whether the user selects one of these icons while performing operation S408. That is, the controller 130 checks whether the icon 530 for transmitting the 3D image to the robot device 200 is selected in step S410. If the user selects an icon 530 for transmitting the 3D image to the robot device 200, the controller 130 proceeds to step S420.

On the other hand, if the user checks in step S410 that the user does not select the icon 530 for transmitting the 3D image to the robot device 200, the control unit 130 proceeds to step S412 and the cancel icon in the screen providing unit 170. Check if 540 has been selected. When the cancel icon 540 is selected in the test result screen providing unit 170 of step S412, the controller 130 ends all processes therefor. If the channel is set at this time, the set channel is also released.

In addition, if the user does not select cancel as a result of the check in step S412, the controller 130 proceeds to step S414 to check whether the re-shoot icon 520 is selected in the screen providing unit 170. If the re-photographing of the test result of step S414 is selected, the control unit 130 proceeds to step S406, otherwise the control unit 130 proceeds to step S408.

Then, the case proceeds from step S410 to step S420 will be described first. In step S420, the controller 130 transmits the 3D image through the channel set in step S420. If step S404 has not been performed before, step S410 may be configured to set a channel with the robot apparatus 200. In this way, after transmitting the 3D image through the channel set in step S420, the controller 130 proceeds to step S430 to control the display mode. In the display mode, three-dimensional data may be displayed, and the voice of the user may be directly or processed to be transmitted to the robot apparatus 200, or the voice signal may be generated and transmitted using the text input by the user. The image of the mouth can also be configured to be processed to transmit in response to voice. In addition, when the user performs an operation such as rotating or tilting the smart terminal, motion information may be collected and processed into image data and then provided to the robot device 200. More details on this will be described with reference to FIG. 6 to be described later.

As such, when the display mode ends, the process of providing a 3D image and an audio or audio signal to the robot device 200 according to the present invention ends.

On the other hand, if re-shooting is selected in step S414, the controller 130 proceeds to step S406 and repeats the process after step 406 described above.

6 is a control flowchart of an image display mode when providing a 3D image from the smart terminal to the robot device 200 according to an embodiment of the present invention. Hereinafter, the image display mode according to the present invention will be described in detail with reference to FIG. 6.

Step S600 is a state of maintaining the provided image after providing a three-dimensional image from the smart terminal to the robot device 200. In this state of maintaining the image, the control unit 130 proceeds to step S602 to check whether a text or voice is input. If a text or voice is input, the controller 130 proceeds to step S604, and if not, proceeds to step S610.

First, the case proceeds to step S604. When proceeding to step S604 may be two cases. This is the case when a voice is input and a character is input. First, when a character is input, the controller 130 generates voice information to be transmitted by reading voice information corresponding to the character from the memory unit 150 in step S604. In this case, the voice information may be sound information such as goat crying or baby crying or the like. In addition, the controller 130 generates movement of the lips or face muscles in the 3D image information in response to the voice information generated in step S604. Thereafter, the controller 130 proceeds to step S606 to transmit the converted voice information and the 3D motion information to the robot device 200.

Next, voice may be input. When a voice is input, the voice may be transmitted as it is, or may be processed into another voice according to a special technique. If the voice is transmitted as it is, the control unit 130 modulates the electric voice signal input from the microphone (MIC) by controlling the modem unit 120. In this case, the control unit 130 in step S604 three-dimensional image The movement of the lips or facial muscles of the data is generated corresponding to the voice information. Thereafter, the controller 130 transmits the voice information and the motion information generated in operation S606 to the robot apparatus 200 through the second communication unit 112.

On the other hand, if the voice is not transmitted, but is processed and transmitted, the controller 130 reprocesses the voice signal using a predetermined voice conversion algorithm from the electric voice signal input from the microphone MIC. Here, the algorithm for speech conversion may have a variety of techniques, for example, there may be a variety of methods such as converting the voice at a high speed of 1.5 times faster or slower at a speed of 1/2. In the present invention, there is no particular limitation on the method of voice conversion.

After reprocessing the voice signal as described above, the controller 130 controls the modem unit 120 to modulate the reprocessed voice signal. In this case, the controller 130 generates movement of the lips or facial muscles of the 3D image data corresponding to the voice information in step S604. Thereafter, the controller 130 transmits the voice information and the motion information generated in operation S606 to the robot apparatus 200 through the second communication unit 112.

On the other hand, when proceeding to step S610, the controller 130 checks whether the movement is detected. That is, it is to check whether the rotation or tilt information is received from the motion detection unit 140. When the rotation or tilt information is received as described above, the control unit 130 proceeds to step S612 to generate three-dimensional data according to the movement. Next, the motion of the smart terminal will be described with reference to FIG. 7.

7 is an exemplary view for explaining a process of detecting the movement of the 3D image according to the movement of the smart terminal according to the present invention.

As shown in FIG. 7, the smart terminal may be tilted forward or laid backward as shown by reference numeral 700. In addition, as shown by reference numeral 702, the smart terminal may be moved to the left or to the right. The motion detector 140 detects the motion and provides the motion to the controller 130. That is, the motion detection unit 140 detects the degree of inclination of the smart terminal and the left and right movements as shown by reference numeral 702 as shown by reference numeral 700 and provides it to the controller 130.

Referring back to FIG. 6, the motion detection in step S610 is such that the motion of the smart terminal is detected. Then, the controller 130 generates 3D data for the 3D face image in response to the detected motion in step S612. For example, when tilting the smart terminal forward, the head information is generated and the motion information of the 3D image data is generated so that the chin side is not visible. On the contrary, when the smart terminal is laid on the back side, the neck and the jaw come out in front and the forehead is moved backward to generate motion information of the 3D image data.

In addition, when the left side of the smart terminal is moved backward with respect to the plane, the controller 130 generates motion information of the 3D image data such that the left face is visible while the right face of the generated 3D face is hidden, and the smart is based on the plane. When the right side of the terminal is moved backward, the controller 130 generates motion information of the 3D image data such that the left face of the generated 3D face is covered and the right face is visible. On the contrary, the smart terminal may be moved forward or backward with respect to the plane. Even in this case, the controller 130 generates motion information of the corresponding video.

When the motion data of the 3D image is generated as described above, the controller 130 controls the modem 120 and the second communication unit 112 to transmit the information generated in step S614 to the robot apparatus 200.

On the other hand, if the process proceeds to step S620, the controller 130 checks whether an end is selected. That is, it is to check whether or not you want to end the process of communicating with the robot apparatus 200. If the end of the check result of step S620 is selected, the control unit 130 proceeds to step S622 to release the channel set with the robot device 200, and ends the routine according to the present invention.

FIG. 8 is a control flowchart for receiving and displaying three-dimensional data in the robot apparatus 200 according to the present invention, and receiving and providing a voice and sound signal to a user. Hereinafter, an operation made in the robot device 200 according to the present invention will be described with reference to FIG. 8.

The control unit 220 maintains the standby state in step S800. Here, the standby state means a state waiting for a user to input a specific command or from a smart terminal. When the controller 220 proceeds to step S802, it checks whether communication with the smart terminal is required. If communication of the test result of step S802 is required, the control unit 220 proceeds to step S804 to set a channel for communication with the smart terminal, and maintains the set channel. On the contrary, if communication with the smart terminal is not required in step S802, the process proceeds to step S800.

Thereafter, the controller 220 proceeds to step S806 to check whether data is received from the smart terminal. If data is received from the smart terminal, the process proceeds to step S808, and otherwise, step S804 is maintained.

In step S808, the control unit 220 processes the received data to provide the data received from the smart terminal through the communication unit 210 to the image providing unit 230 or the sound providing unit 240. This processing includes demodulation and decoding of video and audio, and includes processing for displaying a 3D video on the video providing unit 230.

Thereafter, the controller 220 proceeds to step S810 to check whether additional data is received. If additional test data is received in step S810, the process proceeds to step S812 to process the additional data and to display it in the image providing unit 230 or the sound providing unit 240. In addition, the additional data may be data or voice data on the tilt or rotation of the smart terminal described above.

That is, in the case of a 3D image, the image providing unit 230 is controlled to display a 3D image. In the case of voice or sound, the image providing unit 240 is controlled to output a voice or sound.

On the other hand, if no additional data is received, the controller 220 proceeds to step S820 and checks whether a channel release is requested. If the channel release is requested as a result of step S820, the process proceeds to step S822 to release the channel established with the smart terminal and terminates.

On the other hand, if the channel release is not requested as a result of step S820, the controller 220 proceeds to step S808 to maintain the data display state.

Then, the change of the 3D image according to the movement of the audio signal will be described with reference to [FIG. 9].

FIG. 9 is an exemplary view for explaining the movement of 3D face image data when voice data using voice or text is provided in addition to the 3D face image according to the present invention.

In FIG. 9, reference numeral 900 is an example of a 3D face image. When voice information using voice or text is transmitted in such a 3D face image, the shape of the lips indicated by reference numeral 902 may be changed. That is, each time a syllable is expressed, the shape of the lips may be changed according to the process of opening and closing the lips and the phonetic form of the syllable.

The deformed information may use previously stored information, may be stored in the memory 150 of the smart terminal, may be stored in the memory (not shown in FIG. 3) of the robot device 200, or may be stored in an external server. It may be stored in the memory of the device. In addition, when using not only the shape of the lips but also using a plurality of photographs or storing the pattern information of the facial muscles, the facial muscles may also be expressed according to syllables or syllables.

10 is a conceptual diagram for explaining the overall operation when transmitting a three-dimensional image and voice from the smart terminal to the robot device 200 according to the present invention.

The user 20 takes a picture in step S1011 using the smart terminal 100. Thereafter, the smart terminal 100 extracts a face from the taken photo and generates 3D face data using the extracted face data. The three-dimensional face data generated as described above is provided to the robot device 200, and the robot device 200 may be displayed as shown by reference numeral 300 through the image providing unit 230.

In addition, the user detects the movement of the smart terminal by the motion detection unit 140 in the smart terminal 100 by moving the smart terminal 100 as in step S1021, and the robot 3D motion image information according to the information or the movement thereof. It may be provided to the device 200. Then, the robot apparatus 200 receives the 3D motion image information and displays the motion of the image in the 3D by the image providing unit 230.

In addition, the user may input voice or text into the smart terminal 100 as in step S1031. When the voice or text is input, the smart terminal 100 processes the same in a preset manner, generates the motion information of the 3D image or the 3D information according to the motion, and provides the robot device 200 in response to the voice. Then, the robot apparatus 200 receives the displayed information and displays the three-dimensional motion in the image providing unit 230 and outputs the voice information received through the sound providing unit 240.

Through the above operation, the smart terminal 100 may provide the robot apparatus 200 and the 3D image and voice to the robot apparatus 200 and display the provided information on the robot apparatus 200.

10: wireless network
11: base station
100: smart terminal
200: robot device
110: wireless unit
111: first communication unit
112: second communication unit
120: modem unit
MIC: Microphone
SPK: Speaker
130:
140: motion detection unit
150:
160:
170: screen provider
180: touch recognition unit
210:
220:
230: video providing unit
240: sound providing unit

Claims (9)

As a method for providing a three-dimensional display using the smart terminal 100 and the robot device 200 connected via a network,
A first step of extracting, by the smart terminal, a face image from a photo image;
Generating a 3D face image using the extracted face image in the smart terminal;
A third step of providing the robot device to an external robot device so that the smart terminal can use the generated 3D image as a face screen of the robot device;
A seventh step of receiving, by the smart terminal, text data from a user;
An eighth step of the smart terminal obtaining phoneme data corresponding to the text data;
A ninth step of the smart terminal generating motion information of a mouth of the 3D image corresponding to the phoneme data;
A tenth step of providing, by the smart terminal, the mouth movement information of the 3D image to the robot apparatus so that the robot apparatus changes the face screen in response to the mouth movement information on the face screen;
Method for providing a 3D image using a smart terminal and a robot device comprising a.
The method according to claim 1,
In the first step,
Photographing a face of an object by using a camera unit of the smart terminal;
Extracting a face image from the photographed photo image:
Method for providing a three-dimensional image using a smart terminal and a robot device comprising a.
The method according to claim 1,
The second step is a method for providing a three-dimensional image using a smart terminal and a robot device, characterized in that for generating the three-dimensional face image by applying a Collaborative Construction Algorithm (CCA) transformation algorithm to the extracted face image. .
The method according to claim 1,
A fourth step of detecting a motion of the smart terminal through the motion detection unit of the smart terminal;
A fifth step of the smart terminal regenerating the 3D image in response to the motion;
A sixth step of providing the external robot device so that the smart terminal can use the reproduced 3D image as a face screen of the robot device;
Method for providing a 3D image using a smart terminal and a robot device further comprises.
delete The method of claim 4,
An eleventh step in which the smart terminal receives a voice signal from a user;
A twelfth step of the smart terminal recognizing text from the voice signal;
A thirteenth step of obtaining phoneme data corresponding to the recognized text;
A fourteenth step of the smart terminal generating motion information of a mouth of the 3D image in response to the phoneme data;
A fifteenth step of providing, by the smart terminal, the mouth movement information of the 3D image to the robot device so that the robot device can change the face screen in response to the mouth movement information on the face screen;
Method for providing a 3D image using a smart terminal and a robot device further comprises.
It is installed in the smart terminal to provide a three-dimensional display using the smart terminal 100 and the robot device 200 connected via a network, the smart terminal three-dimensional for performing the method according to any one of claims 1 to 4. A computer-readable recording medium recording an image providing program;
The robot device communication unit 210 for performing data communication with the smart terminal 100, the program is executed, and provides a display of the three-dimensional face image provided from the smart terminal through the robot device communication unit and through the robot device communication unit The image providing unit 230 transforms and displays the image of the mouth portion in the 3D face image corresponding to the mouth movement information provided from the smart terminal, and the 3D image received from the smart terminal to the image providing unit 230. Robot device 200 including a robot device control unit 220 for controlling the output;
System for providing a three-dimensional image using a smart terminal and a robot device configured to include.
delete The method of claim 7,
The smart terminal 3D image providing program recorded on the recording medium includes the eleventh step of receiving the voice signal from the smart terminal, the twelfth step of recognizing the text from the voice signal by the smart terminal, A thirteenth step of acquiring phoneme data corresponding to text; and a fourteenth step of generating, by the smart terminal, motion information of the mouth of the 3D image in response to the phoneme data; And performing a fifteenth step of transmitting mouth movement information to the robot apparatus so that the robot apparatus can change the face screen in response to the mouth movement information on the face screen.
The image providing unit 230 of the robot device 200 is characterized in that the deformation display of the image of the mouth portion in the three-dimensional face image corresponding to the mouth movement information provided from the smart terminal through the robot device communication unit System for providing a 3D image using a smart terminal and a robot device.

Images