KR102470023B1 - System for providing remote proxy exploring service using realtime live - Google Patents

Abstract

Provided is a system for providing a remote proxy visit service using a real-time live broadcast. The system comprises: a user terminal which inputs a country, an area, and a specialized field that require a proxy visit, and is matched with an expert who meets the conditions of the country, the area, and the specialized field; an expert terminal which is matched with the user terminal, is connected to the user terminal through a real-time live broadcast at the date, the time, and the place requested by the user terminal, and transmits a response to a request received from the user terminal; and a proxy visit service providing server including a building unit which receives expert data including the country, the area, and the specialized field from the expert terminal and builds an expert pool, a matching unit which matches the expert who meets the conditions of the country, the area, and the specialized field when the user terminal inputs the country, the area, and the specialized field that require the proxy visit, and a connection unit which connects the user terminal and the expert terminal through the real-time live broadcast at the date, the time, and the place requested by the user terminal. Therefore, the system enables the expert to conduct onsite inspections conveniently and without omissions.

Description

Remote proxy visiting service using real-time live {SYSTEM FOR PROVIDING REMOTE PROXY EXPLORING SERVICE USING REALTIME LIVE}

The present invention relates to a system for providing a remote proxy tour service using real-time live, and provides a platform capable of conducting a field tour in real-time live by hiring an agent who is an expert even without going directly to an overseas site.

As non-face-to-face contact has recently become mandatory due to COVID-19, those who wish to depart abroad must attach the corona test results according to the regulations of the destination country, and even if they are not confirmed, they must quarantine for 2 weeks in preparation for the 2-week incubation period. Likewise, those who want to enter Korea are also required to attach certificates or self-quarantine for two weeks in accordance with Korean laws, putting a brake on overseas business trips, and the involuntary unemployment rate continues to soar. Those in charge who need to participate in fairs or exhibitions held overseas, conduct market research, or conduct on-site inspections must leave the country with 2 weeks to spare. It is increasing.

At this time, a platform for acting as a role or online business trip was researched and developed. In this regard, prior art Korea Patent No. 10-2161102 (published on September 29, 2020) and Korea Patent Publication No. 2006-0124109 (2006 released on December 05, 2018), the mission registered in the user terminal, that is, according to the command, matches and mediates the supporter terminal, and after the mission is completed, the user terminal inputs the rating according to the performance of the role-playing mission, and the supporter terminal A configuration that stores mission history information and a configuration that allows users to input all conditions related to overseas business trips from a user terminal within an online one-stop business trip platform, and to reserve and pay for overseas air tickets, accommodations, and local transportation, respectively. is initiated

However, in the former case, the user handles the user's business on behalf of the user, and is similar to the civil law office management or employment contract, so the user does not actually handle the business. Even in the case of the latter, it does not provide an overseas business trip instead, but only provides a platform for online reservations for business trips. Due to COVID-19, the user has to personally handle things such as field trips or site visits, but in a situation where he cannot go directly, the configuration of the platform as if the user is actually in the field by hiring a local expert is not disclosed. Accordingly, there is a need for research and development of a platform that allows a user to visit a desired date, time, place, and country on behalf of the user and perform the user's request in real time live on site, such as an avatar.

According to an embodiment of the present invention, country, region, and field of expertise are input from a user terminal, and expert terminals satisfying conditions of the country, region, and field of expertise are matched, and the expert terminal and the user are matched at a time, date, and place desired by the user. Terminals are connected live in real time, experts perform the requests made by users of user terminals on behalf of the user so that they can do proxy visits, the captured video is saved and transmitted to the user terminal as an on-site report, and real-time object detection Information such as object identification, recognition, and price comparison is provided so that no part is missed by the expert, and when a 3D sensor that performs motion tracking is used in the user terminal, a specific object is transmitted to the expert terminal through a gesture so that it is convenient and complete without omission. It is possible to provide a remote agency visiting service providing system using real-time live, which allows field surveys to be conducted. However, the technical problem to be achieved by the present embodiment is not limited to the technical problem as described above, and other technical problems may exist.

As a technical means for achieving the above-described technical problem, an embodiment of the present invention inputs a country, region, and specialized field requiring proxy visitation, and matches a user with an expert who satisfies the conditions of the country, region, and specialized field. It is matched with the terminal and the user terminal, is connected to the user terminal in real time at the date, time and place requested by the user terminal, and transmits a response to the request received from the user terminal, and the expert terminal that transmits a response to the request received from the user terminal and the country, region and expert terminal Construction department that builds a pool of experts by receiving expert data including fields, and matching experts who meet the conditions of the country, region, and field of expertise by entering the country, region, and field of expertise that require a proxy visit from the user terminal It includes a matching unit, a proxy tour service providing server including a connection unit that connects between the user terminal and the expert terminal in real-time live at the date, time and place requested by the user terminal.

According to any one of the above-described problem solving means of the present invention, country, region, and specialized field are input from a user terminal, and an expert terminal that satisfies the conditions of the country, region, and specialized field is matched, and the time, date and time desired by the user are matched. The expert terminal and the user terminal are connected in real time to the place, and the expert performs the requests made by the user of the user terminal for proxy tours, and the recorded video is saved and transmitted to the user terminal as a field report. Real-time object detection provides information such as object identification, recognition, and price comparison so that experts do not miss any parts, and when a 3D sensor for motion tracking is used in a user terminal, a specific object is sent to an expert terminal through gestures. Send it so that you can conduct an on-site survey conveniently and without omission.

1 is a diagram for explaining a remote proxy visiting service providing system using real-time live according to an embodiment of the present invention.
Figure 2 is a block diagram for explaining the agency visit service providing server included in the system of Figure 1.
3 and 4 are diagrams for explaining an embodiment in which a remote proxy visiting service using real-time live according to an embodiment of the present invention is implemented.
5 is an operational flowchart for explaining a method for providing a remote proxy visiting service using real-time live according to an embodiment of the present invention.

Hereinafter, embodiments of the present invention will be described in detail so that those skilled in the art can easily practice the present invention with reference to the accompanying drawings. However, the present invention may be embodied in many different forms and is not limited to the embodiments described herein. And in order to clearly explain the present invention in the drawings, parts irrelevant to the description are omitted, and similar reference numerals are attached to similar parts throughout the specification.

Throughout the specification, when a part is said to be "connected" to another part, this includes not only the case where it is "directly connected" but also the case where it is "electrically connected" with another element interposed therebetween. . In addition, when a part "includes" a certain component, this means that it may further include other components, not excluding other components, unless otherwise stated, and one or more other characteristics. However, it should be understood that it does not preclude the possibility of existence or addition of numbers, steps, operations, components, parts, or combinations thereof.

As used throughout the specification, the terms "about", "substantially", etc., are used at or approximating that value when manufacturing and material tolerances inherent in the stated meaning are given, and do not convey an understanding of the present invention. Accurate or absolute figures are used to help prevent exploitation by unscrupulous infringers of the disclosed disclosure. The term "step of (doing)" or "step of" as used throughout the specification of the present invention does not mean "step for".

In this specification, a "unit" includes a unit realized by hardware, a unit realized by software, and a unit realized using both. Further, one unit may be realized using two or more hardware, and two or more units may be realized by one hardware. On the other hand, '~ unit' is not limited to software or hardware, and '~ unit' may be configured to be in an addressable storage medium or configured to reproduce one or more processors. Thus, as an example, '~unit' refers to components such as software components, object-oriented software components, class components, and task components, processes, functions, properties, and procedures. , subroutines, segments of program code, drivers, firmware, microcode, circuitry, data, databases, data structures, tables, arrays and variables. Functions provided within components and '~units' may be combined into smaller numbers of components and '~units' or further separated into additional components and '~units'. In addition, components and '~units' may be implemented to play one or more CPUs in a device or a secure multimedia card.

In this specification, some of the operations or functions described as being performed by a terminal, device, or device may be performed instead by a server connected to the terminal, device, or device. Likewise, some of the operations or functions described as being performed by the server may also be performed by a terminal, apparatus, or device connected to the server.

In this specification, some of the operations or functions described as mapping or matching with the terminal mean mapping or matching the terminal's unique number or personal identification information, which is the terminal's identifying data. can be interpreted as

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings.

1 is a diagram for explaining a remote proxy visiting service providing system using real-time live according to an embodiment of the present invention. Referring to FIG. 1 , a remote proxy visiting service providing system 1 using real-time live may include at least one user terminal 100, a proxy visiting service providing server 300, and at least one expert terminal 400. can However, since the remote proxy visiting service providing system 1 using real-time live of FIG. 1 is only one embodiment of the present invention, the present invention is not limitedly interpreted through FIG. 1 .

At this time, each component of FIG. 1 is generally connected through a network (Network, 200). For example, as shown in FIG. 1 , at least one user terminal 100 may be connected to the agency visiting service providing server 300 through the network 200 . In addition, the agency visiting service providing server 300 may be connected to at least one user terminal 100 and at least one expert terminal 400 through the network 200 . In addition, at least one expert terminal 400 may be connected to the agency visiting service providing server 300 through the network 200 .

Here, the network refers to a connection structure capable of exchanging information between nodes such as a plurality of terminals and servers, and examples of such networks include a local area network (LAN) and a wide area network (WAN: Wide Area Network), the Internet (WWW: World Wide Web), wired and wireless data communications networks, telephone networks, and wired and wireless television communications networks. Examples of wireless data communication networks include 3G, 4G, 5G, 3rd Generation Partnership Project (3GPP), 5th Generation Partnership Project (5GPP), Long Term Evolution (LTE), World Interoperability for Microwave Access (WIMAX), Wi-Fi , Internet (Internet), LAN (Local Area Network), Wireless LAN (Wireless Local Area Network), WAN (Wide Area Network), PAN (Personal Area Network), RF (Radio Frequency), Bluetooth (Bluetooth) network, NFC ( A Near-Field Communication (Near-Field Communication) network, a satellite broadcasting network, an analog broadcasting network, a Digital Multimedia Broadcasting (DMB) network, etc. are included, but not limited thereto.

In the following, the term at least one is defined as a term including singular and plural, and even if at least one term does not exist, each component may exist in singular or plural, and may mean singular or plural. It will be self-evident. In addition, the singular or plural number of each component may be changed according to embodiments.

At least one user terminal 100 may be a terminal of a requester requesting remote proxy visiting service using a web page, app page, program, or application related to a remote proxy visiting service using real-time live. At this time, the user terminal 100 may be a terminal that receives expert matching from the agency visiting service providing server 300 after entering the country, region, and specialized field. In addition, the user terminal 100 may be a terminal that is connected to the expert terminal 400 live in real time via the agency visiting service providing server 300 .

Here, at least one user terminal 100 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser. In this case, at least one user terminal 100 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one user terminal 100 is, for example, a wireless communication device that ensures portability and mobility, navigation, PCS (Personal Communication System), GSM (Global System for Mobile communications), PDC (Personal Digital Cellular), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet ) may include all types of handheld-based wireless communication devices such as terminals, smartphones, smart pads, tablet PCs, and the like.

Agent visiting service providing server 300 may be a server that provides a remote proxy visiting service web page, app page, program or application using real-time live. In addition, the agency visiting service providing server 300 may be a server that constructs a pool of experts by receiving a country, region, and field of expertise from at least one expert terminal 400 . In addition, the agency visiting service providing server 300 transmits an expert having a condition satisfying the conditions of the user terminal 100 to the user terminal 100 and matches the expert terminal 400 selected by the user terminal 100. can be a server In addition, the proxy visiting service providing server 300 may be a server that connects the user terminal 100 and the expert terminal 400 live in real time at the time, place, and date requested by the user. The agency visit service providing server 300 may be a server that settles a part of the payment amount or an amount corresponding to a preset percentage to the expert terminal 400 when the agency visit is completed.

Here, the agency visiting service providing server 300 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser.

At least one expert terminal 400 uses a web page, app page, program, or application related to a remote agency visit service using real-time live proxy visit data in a country, region, and specialized field to join the expert pool. It may be a terminal that uploads to the service providing server 300 . At this time, the expert terminal 400 may be a terminal that undergoes verification such as identity authentication or qualification authentication through a third trusted party at the request of the proxy visiting service providing server 300 . In addition, when the expert terminal 400 is matched with the user in the agency visiting service providing server 300, it is connected to the user terminal 100 live in real time and processes the office work requested by the user terminal 100 on behalf of the user. , it may be an expert's terminal that handles the work requested by the user in real time.

Here, at least one expert terminal 400 may be implemented as a computer capable of accessing a remote server or terminal through a network. Here, the computer may include, for example, a laptop, a desktop, a laptop, and the like equipped with a navigation system and a web browser. In this case, at least one expert terminal 400 may be implemented as a terminal capable of accessing a remote server or terminal through a network. At least one expert terminal 400 is, for example, a wireless communication device that ensures portability and mobility, and includes navigation, personal communication system (PCS), global system for mobile communications (GSM), personal digital cellular (PDC), PHS (Personal Handyphone System), PDA (Personal Digital Assistant), IMT (International Mobile Telecommunication)-2000, CDMA (Code Division Multiple Access)-2000, W-CDMA (W-Code Division Multiple Access), Wibro (Wireless Broadband Internet ) may include all types of handheld-based wireless communication devices such as terminals, smartphones, smart pads, tablet PCs, and the like.

2 is a block diagram illustrating a server for providing agency visiting service included in the system of FIG. 1, and FIGS. 3 and 4 are implementations of remote agency visiting service using real-time live according to an embodiment of the present invention. It is a drawing for explaining an embodiment.

Referring to FIG. 2, the agency visiting service providing server 300 includes a construction unit 310, a matching unit 320, a connection unit 330, an AI translation unit 340, an image identification unit 350, and an object confirmation unit. 360 and a de-identification unit 370 may be included.

The agency visiting service providing server 300 according to an embodiment of the present invention or another server (not shown) operating in conjunction with at least one user terminal 100 and at least one expert terminal 400 uses real-time live When transmitting a remote proxy visiting service application, program, app page, web page, etc., at least one user terminal 100 and at least one expert terminal 400 use real-time live remote proxy visiting service application, program, You can install or open app pages, web pages, etc. Also, the service program may be run in at least one user terminal 100 and at least one expert terminal 400 by using a script executed in a web browser. Here, the web browser is a program that allows users to use the web (WWW: World Wide Web) service, and means a program that receives and displays hypertext described in HTML (Hyper Text Mark-up Language). For example, Netscape , Explorer, Chrome, and the like. In addition, an application means an application on a terminal, and includes, for example, an app running on a mobile terminal (smart phone).

Referring to FIG. 2 , the construction unit 310 may construct an expert pool by receiving expert data including country, region, and specialized field from the expert terminal 400 . If a freelancer is registered in the form of an individual entrepreneur rather than an expert registered through the hiring process on the platform of the present invention, a process of verifying whether the expert is an expert may be further performed through a third trust organization. Recently, there are increasing cases of Chinese nationals being deported from the country due to identity laundering and forgery of academic background. In the case of locals, it is difficult to know whether graduation certificates are forged or not. can make it

Accordingly, the at least one expert terminal 400 may receive expert data including at least one type of expert's license, career, profile, working period and working conditions, and transmit the input to the construction unit 310 . At this time, block chain technology can be used to prevent counterfeiting of personal education or career, forgery of license, or illegal rental when building a human resource pool. At this time, the block chain may be based on the pseudocode of the chain code, and the block chain network is composed of a three-party structure of a construction unit 310, a job terminal (not shown), and an expert terminal 400. The construction unit 310 has all authority of the network as a full node, and the recruiting terminal and expert terminal 400 participate in the network with limited functions as light weighted nodes.

<Block Generation Procedure>

Block generation is performed when the expert terminal 400 participates in a blockchain network or changes the education and career of an expert. ① The expert terminal 400 requests block generation when participating in the blockchain network with the construction unit 310 and registering its own information in the blockchain and changing existing information. ② When receiving a registration request, the construction unit 310 starts a block generation procedure based on the information provided by the expert terminal 400 . At this time, in order to check whether the expert of the expert terminal 400 has existed, it is searched whether the corresponding expert is registered in the blockchain. ③ The construction unit 310 checks whether the expert is registered in the blockchain according to the request of the manager terminal (not shown) and delivers it to the manager. ④ The administrator requests information about the expert's education and career from the government to check whether the expert's information is forged. ⑤ The government checks whether it is permissible to provide education and career information to experts. ⑥ The expert terminal 400 transmits whether or not to agree to provide personal information so that the user's information can be stored in the blockchain. ⑦ The government delivers whether or not the expert information has been verified. ⑧ Information about experts is created and stored in blocks in the blockchain.

<Individual academic background and career verification procedure>

The process of providing education and experience is used when a professional is actually looking for a job, not from the position of a producer receiving a request. It can be used at the request of the terminal (not shown). First of all, ① When an applicant registered as an expert submits a resume through a company terminal, that is, a job offer terminal, he/she submits his or her ID registered in the blockchain. ② The company receives the applicant's resume and notifies whether or not it has been received. ③ The company requests the construction unit 310 to verify whether the applicant's academic background and career in the resume are forged. ④ The construction unit 310 confirms whether the applicant agrees to confirm career in the block chain. ⑤ The applicant sends a message authorizing the construction unit 310 to retrieve his or her data. ⑥ The construction unit 310 executes the chain code to check the internal data of the blockchain, and at this time, it searches the expert data using the unique ID of the expert (applicant). ⑦ The construction unit 310 transmits the data registered in the block chain to the company (recruiting company of the job terminal) so that it can be checked for forgery or alteration.

<Operation of Chaincode>

Chaincodes can be used for consent to provide personal information, search for internal data in blocks, creation and modification of blocks, and verification of career information. Since the expert's career and educational background information is included in personal information, it is necessary to confirm whether the expert agrees to the information. Subjects using that chaincode are used by governments that can help verify expert information. At this time, the method of obtaining consent to provide information from the expert is to receive confirmation of consent in the form of a message or e-mail. Block generation and verification can be performed only when experts agree to provide information.

The chain code used for expert search can be used after confirming consent to provide personal information. This is because it is the chaincode that actually does the work of getting the data inside the blockchain. The data search chaincode can perform search using two methods: QueryString and txType. The QueryString method is a method of performing a search using only the expert ID. The txType method is a method of performing a search by assigning an expert ID and additional conditions. At this time, if the corresponding chaincode is not used, it can be configured so that the information cannot be verified in the blockchain.

The background technologies used in the above configuration are blockchain, chain code, and hyperledger consensus algorithm. First of all, blockchain is used to simplify complex systems by utilizing blockchain or to synchronize data to prevent confusion. The difference between existing security methods and blockchain is that by providing the same information to all people involved in the network, even if a specific person forges data, it is not recognized because it is different from the information known to all members. That is, unlike the existing data hiding, it is a technology that prevents counterfeiting through disclosure.

Transactions in the blockchain can only be registered in a block if it is authorized using a hash function. In this process, a digital signature is used, which is used to provide a non-repudiation function. The method mainly used as a method for transaction processing is the method using chaincode. Chaincode is used not only to generate blocks, but also to provide functions that can provide various services using blockchain. Chaincodes can be classified into system chaincodes and user chaincodes.

Hyperledger Fabric uses a different transaction processing method than other blockchain platforms. In the case of other blockchain platforms, a method of verifying transaction details and linking with the chain is used. In contrast, Hyperledger Fabric operates through the Execute, Order, Validate, and Update state procedures. In the case of the existing Hyperledger Fabric, the PBFT (Practical Byzantine Fault Tolerance) method was used as a consensus method, but the decrease in transaction per second (tps) performance occurred as a problem, so the Kafka algorithm is currently used for the consensus procedure. At this time, SBFT (Simple BFT) may be used and applied. Of course, since a license, educational background, career, etc. may be verified or shared in various ways other than the above-described method, it is not limited thereto.

The matching unit 320 may match experts who satisfy the conditions of the country, region, and specialized field when the user terminal 100 inputs a country, region, and specialized field requiring agency visits. The user terminal 100 may enter a country, region, and field of expertise requiring agency visitation, and be matched with an expert who satisfies conditions of the country, region, and field of expertise. The expert terminal 400 is matched with the user terminal 100, is connected to the user terminal 100 live in real time at the date, time, and place requested by the user terminal 100, and receives transmission from the user terminal 100. You can send a response to a request.

The connection unit 330 may connect the user terminal 100 and the expert terminal 400 live in real time at a date, time, and place requested by the user terminal 100 . In this case, the connection unit 330 may further perform traffic smoothing. The basic condition for traffic smoothing is that the server end knows the encoding curve. In a single link, knowing its own encoding curve also means knowing the decoding curve, so there was no problem in traffic smoothing. However, in order to perform optimized traffic smoothing in multi-link, the encoding curve of the transcoder must be considered together when data is transmitted on the link for the first time, so the entire curve cannot be known in a situation where transcoding is not performed. Therefore, when performing traffic smoothing in multi-link, it is necessary to adjust the reference delay values within a stable range. The maximum delay (δn) from transmission start to arrival at the decoder is the maximum delay (δnup) in , δndown), respectively.

Here, when the maximum delay (δnup, δndown) in the uplink and downlink is determined as in Equation 9 below, if the amount of data generated in the uplink is large, the transmission time transmitted in the uplink is increased, If the amount of transmitted data is large in the downlink, the transmission time in the downlink is increased so that the level of the amount of data is appropriately divided.

On the other hand, a traffic smoothing method in a single link is applied to each of the multi-links. The traffic smoothing process in a single link is, first, a frame in which a large amount of data occurs by weighting as much as the size of frames in a Group of Pictures (GOP). This is a process in which a large transmission time is given to a frame and a small transmission time is given to a frame in which a small amount of data is generated. In this way, since the overall amount of data generation is constant, the transmission load of the entire network can be uniformed to increase stability. The second is the process of determining the actual transmission rate for each allocated frame-by-frame transmission time. In this process, a constraint condition is required that the transmission rate must be set to prevent decoding underflow by the decoder in units of slices within the frame. Through this process, traffic smoothing in a single link can be achieved.

At this time, since the transmitted video is immediately transmitted according to the communication cycle without stacking, the transmission time can be shortened compared to other programs that transmit the screen using arbitrary buffering, such as video data. Compared to other programs, there is a disconnection phenomenon, but better performance can be seen in the purpose of implementing real-time transmission through fast transmission.

When voice speech is input from the expert terminal 400, the AI interpreter/translate unit 340 converts the spoken sentence into speech to text (STT) and transmits it to the user terminal 100 using an artificial intelligence translation program. . At this time, the artificial intelligence interpretation and translation program may use publicly known technology. Recently, Hangul and Computer Group joined hands with a Chinese company to enter the AI interpretation and translation market in earnest. It announced that it would enter the business, signed a contract to establish Accuply.AI, a joint venture, and announced that it would expand its business to Fintech, Edutech, smart healthcare, and hardware solutions that incorporate AI technology. The real-time automatic interpretation and translation market is rapidly growing as it expands from smartphones to various devices. ), Naver's Papago, Google's Google Translator, Microsoft's Skype Translator, Systran's Systran Enterprise, and Hancom Interfree's Genie Talk.

The image identification unit 350 recognizes at least one object in the image input through the expert terminal 400, corrects image distortion of the at least one object, and extracts image feature data of the at least one object to obtain an image. You can label tags on . In this case, the expert terminal 400 may include a three-dimensional (3D) sensor. At this time, information on the same product or product may be tagged in the image through image-based search, and information may be applied through price comparison. For example, experts and users who search for objects or products using the image features of objects and then visit them live on behalf of others, such as tagging objects inserted as PPL in IPTV dramas or movies with metadata to inform them of information. It adds an additional feature that guides you to products that you all missed. Of course, since it is an image-based search, the results may not be accurate, but the location, time, and country where the expert is located are specified, and in the case of most conventions, exhibitions, or expositions, the location of each booth or what kind of object is exhibited in that booth. The one to be introduced is already registered in the exhibition pamphlet, so searching for it may not be a difficult task.

When the object identification unit 360 points to at least one object in the screen captured by the expert terminal 400 using the 3D sensor for motion recognition in the user terminal 100 and then receives a motion input along the contour line , the expert terminal 400 may output the motion-input object. For example, assume that product A is visible behind the expert and the user wants to obtain information about product A. At this time, the user may feel stuffy, thinking that it is better to go and pick it up himself when he explains to the expert which side of the product A is located and how it looks. Accordingly, when a user draws the outline of a desired item (object) by hand, the hand-drawn object is also displayed on the expert's terminal 400, so that the expert can easily and intuitively grasp what the user is requesting to show. is to allow

At this time, the user terminal 100 may include a 3D sensor for gesture recognition, and the 3D sensor for gesture recognition is a MEMS (Micro Electro Mechanical System, MEMS) based motion tracker. Alternatively, it may be an optical 3D sensor, but it is not limited to those listed, and any product, program, etc. that can recognize a user's motion and detect motion will be possible. For example, for movie shooting or sports motion analysis, precise motion capture is required, and a motion sensor composed of a gyro sensor, an acceleration sensor, and a geomagnetic sensor is used. ) is attached to the location to be measured (mainly the joint) to use a motion tracker that measures the user's movement. A motion tracker has the advantage of higher accuracy and no self-occlusion than an optical 3D sensor, but it has the inconvenience of attaching a sensor to the location to be measured. An optical 3D sensor is used.

Accordingly, in one embodiment of the present invention, for convenience, an optical 3D sensor may be used as a sensor outputting a depth image. The motion sensor measures the position by sensing the 3-dimensional acceleration and angular velocity of the position. Accordingly, while the 3D position and orientation of the point where the sensor is attached can be known, the optical 3D sensor can know the depth values of all areas seen from the viewpoint of the D sensor. The difference between the two methods is that the former can know the relative position (translation) and orientation (orientation) from the initial position, but has a disadvantage in that position errors accumulate as the driving time increases. However, it has the advantage of being able to measure the exact position because it has a fast measurement speed and no self-shielding. The latter can know the absolute position from the 3D sensor, but cannot know the orientation, and self-occlusion occurs due to the limit of the distance image expressed in 2.5D. However, from the user's point of view, motion can be recognized naturally without an additional sensor, so it has higher usability than the former. MEMS-based 3D sensors are mainly used in the field of precise motion capture.

A general motion recognition process is a process of acquiring a distance image from a 3D sensor, extracting a user's skeleton (characteristic points for recognizing a user's posture - generally using joint parts), and then recognizing the user's motion from the posture and behavior of the skeleton proceeds with Unlike general camera images, the distance image used at this time has a distance value from the camera for each pixel, and this distance value is expressed in color for visualization of the distance value. 3D sensing can be divided into an active method and a passive method depending on whether a specific pattern is projected. An infrared distance sensor, an ultrasonic sensor, etc., which are commonly encountered in the surroundings, are active 3D sensing techniques, and a stereo camera that obtains distance information using two or more cameras is a passive 3D sensing technique. The passive method has a simpler system configuration than the active method, but has a disadvantage in that stable data cannot be obtained because it is sensitive to the sensing environment. The active method is more robust to changes in the surrounding environment than the passive method and can obtain abundant distance data, but it is difficult to operate stably in an outdoor environment having higher energy than the projected wave.

3D sensors for motion recognition mainly use light wave-based optical sensing, and according to the sensing method, triangulation, time-of-flight measurement, and interferometry ( interferometry). The triangulation method is a method of obtaining distance data using the mutual distances of two or more coordinate systems and the angles of an object to be measured, and the time delay method is a method of obtaining distance data by measuring the return time of light waves projected on an object. way to obtain it. The interferometry method is similar in principle to the time delay measurement method, but it is a method of obtaining distance data by using an optical interferometer to measure the time difference between light reflected from an object and the phase difference with a reference wave. Since the interferometry method is mainly used for local surface measurement rather than motion recognition, an embodiment of the present invention uses 3D sensing based on triangulation and time delay measurement.

<Triangulation 3D sensing>

Triangulation-based 3D sensing is a method of calculating distance using a simple trigonometric function that calculates the length of the other side using the length of one side and both angles thereof. For example, if you know the distance between two points A and B with an arbitrary distance (baseline) and know the angle α to C, you can calculate D1, the distance between C and B. If is known in this way, D2, the distance between D and B, can also be easily calculated. If this is simply applied to a distance measurement technique using a laser pointer, when a system is configured with a laser pointer and a camera at a certain distance (baseline), the angle between the object surface and the camera changes according to the height of the object. This is expressed as the position of the laser pointer in the camera, and when the distance is close or far, the position of the laser pointer moves left and right in the camera image, and the distance is calculated based on this information.

Single Point Laser-based triangulation enables precise measurement because it acquires the distance of one point at a time, and it can collect the projected wave at one point, enabling sensing in outdoor environments, such as scanning large structures. It is used a lot in work. Single-laser-based triangulation is accurate because it can calculate the distance of only one point at a time, but it is very slow to construct a high-resolution distance image. To overcome this, if the single laser is replaced with a line laser, the distance of one line can be measured at a time. Point and line laser-based 3D sensing enables accurate and precise 3D sensing, but is insufficient to acquire distance images in real time. 3D sensing for motion recognition can use techniques using plane-shaped spatial patterns to acquire distance images at a higher speed, and temporal codes (temporal code) and spatial code.

Unlike dots and lines, a technique using a plane pattern must be able to distinguish different regions (points or lines) constituting the pattern. In other words, one pattern can be regarded as being made up of a set of lines, and the lines from 1 to n constituting the pattern must be distinguishable from each other. At this time, inserting information for distinguishing each line from other lines along the time axis is called time code, and inserting information around pixels is called spatial code. In the time code-based 3D sensing method, a pattern is projected using a projector, a pattern distorted according to the shape of an object is obtained with a camera, and a distance image is obtained based on a code value for each pixel. Time code-based 3D sensing can also be precisely sensed, but it is difficult to sense dynamic objects because it uses multiple patterns, and it is difficult to use outdoors, so it is mainly used for object scanning for modeling in indoor environments.

In addition, techniques for constructing time codes may use various techniques such as gray codes in addition to binary codes. Unlike time codes, spatial codes have the advantage of being able to perform 3D sensing with a single pattern, which is fast, but have a disadvantage in that their accuracy and precision are significantly lower than those of time codes. However, motion recognition requires a fast sensing speed rather than accurate sensing, so it is widely used. Unlike time codes, spatial codes use information from neighboring pixels because they must contain information that can be distinguished from other areas in a single pattern, which leads to a decrease in accuracy. In order to calculate the distance value of one pixel, information on surrounding pixels must also be acquired, so an object smaller than a set (area) of pixels cannot be sensed. Spatial codes are also being studied with various techniques for high speed and accuracy, and techniques that assign codes using colors, DeBruijn Sequence-based techniques, and M-arrays-based techniques can be used.

A method of performing 3D sensing by projecting patterns in the form of dots, lines, and planes as described above is called structured light. The structured light technique is a representative active triangulation method. Because it projects a pattern, it can perform 3D sensing even in an environment without texture, but it performs accurate sensing in an environment where the pattern is difficult to see, such as outdoors. Can not. Depending on the pattern coding method, 3D sensing with low speed but high precision can be performed using time codes, and if high-speed 3D sensing is desired, speed can be improved with low precision by using spatial codes. Compared to stereo or time-of-flight (ToF) techniques, precise and rich 3D data can be stably acquired, but it is difficult to miniaturize compared to ToF-based 3D sensors because it requires an arbitrary baseline. Spatial code-based structured light systems with speeds similar to , may generally have lower precision than ToF sensors.

Stereo vision is a method of acquiring images in two coordinate systems having arbitrary baselines like the human eyes, and calculating a distance by finding a correspondence between the two images. Like the structured light system, it calculates the distance based on triangulation, and in the structured light system, it is configured in the form of replacing the projector with a camera. Since the distance can be calculated by finding a correspondence point in the camera images, there is a disadvantage that it cannot be used in an environment without texture (an environment in which a correspondence point cannot be found) or in an environment with severe lighting changes. In order to overcome such disadvantages of stereo vision, a method of attempting matching in an environment lacking texture and a method of efficiently performing correspondence point matching requiring high computation time can be used. A distance may be calculated by obtaining , or a method of using active stereo vision by adding an arbitrary pattern to compensate for an environment lacking in texture may be used.

<3D sensing based on time delay measurement>

Time delay measurement (ToF) is a method of measuring distance by projecting light waves and measuring the return time. Direct method measures the time a laser pulse returns with a digital counter, or projects and receives pulses with a certain period. There is a method of indirectly measuring the time difference (In-Direct Method) using the phase difference of the received pulse. A ToF sensor for motion recognition mainly uses an indirect measurement method that utilizes a phase difference of a pulse to obtain a high-resolution distance image. At this time, in order to calculate the phase difference, a method of sequentially having different timings in one pixel and a method of having different timings in adjacent pixels are used. Of course, it is not limited to the above method, and various methods may be used.

The de-identification unit 370 may blur processing when an object in the screen photographed by the expert terminal 400 is a person. Alternatively, if a face is included in a subject in the photographing data, the face may be replaced with a fake face using Generative Adversarial Networks (GAN). Alternatively, mosaic processing may be applied, or only the eye area may be covered or blurred.

Hereinafter, an operation process according to the configuration of the agency visiting service providing server of FIG. 2 will be described in detail with reference to FIGS. 3 and 4 as examples. However, it will be apparent that the embodiment is only any one of various embodiments of the present invention, and is not limited thereto.

Services such as those of FIGS. 3A to 3C may be provided. 3D to 3V are videos (https://vimeo.com/590982122) taken while actually following a Korean expert conducting a market research to find a cooperative company at the request of a Chinese client for proxy visit. While carrying an action cam or expert terminal 400, the expert consults with the person desired by the user terminal 100, which is a client, lifts a desired product, receives a business card from a company that sells the product, or shows it in front of a camera. , and sometimes write contracts instead. Of course, since drafting a contract is an agency act in lieu of a legal act, it is premised on a relationship in which an agency contract must be drawn up between the mandatary and the mandator, the power of attorney is granted, and a power of attorney must be given. 4A to 4G are introduction screens of the Cheondaeri (application name) application according to an embodiment of the present invention, which is currently known in Google Play and App Store.

Matters not described for the method for providing remote proxy visiting service using real-time live of FIGS. 2 to 4 are the same as or described for the method for providing remote proxy visiting service using real-time live through FIG. 1 above. Since it can be easily inferred from the contents, the following description will be omitted.

5 is a diagram illustrating a process of transmitting and receiving data between components included in the system for providing a remote proxy visiting service using real-time live of FIG. 1 according to an embodiment of the present invention. Hereinafter, an example of a process of transmitting and receiving data between each component will be described through FIG. 5, but the present application is not limited to such an embodiment, and according to various embodiments described above, It is obvious to those skilled in the art that a process of transmitting and receiving data may be changed.

Referring to Figure 5, the agency visit service providing server,

The order between the above-described steps (S5100 to S5400) is only an example, and is not limited thereto. That is, the order of the above-described steps (S5100 to S5400) may be mutually changed, and some of the steps may be simultaneously executed or deleted.

Matters not described in the method for providing remote proxy visiting service using real-time live of FIG. 5 are the same as or described for the method for providing remote proxy visiting service using real-time live through FIGS. 1 to 4 above. Since it can be easily inferred from the contents, the following description will be omitted.

The method for providing a remote proxy visiting service using real-time live according to an embodiment described with reference to FIG. 5 is implemented in the form of a recording medium including instructions executable by a computer such as an application or program module executed by a computer. It can be. Computer readable media can be any available media that can be accessed by a computer and includes both volatile and nonvolatile media, removable and non-removable media. Also, computer readable media may include all computer storage media. Computer storage media includes both volatile and nonvolatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data.

The method for providing a remote proxy tour service using real-time live according to an embodiment of the present invention described above is based on an application installed in a terminal (this may include a program included in a platform or operating system basically installed in the terminal) It can also be executed by an application (i.e., a program) that a user directly installs in the master terminal through an application providing server such as an application store server, an application or a web server related to the corresponding service. In this sense, the method for providing a remote agency visiting service using real-time live according to an embodiment of the present invention described above is implemented as an application (ie, a program) that is basically installed in a terminal or directly installed by a user, and is implemented as a computer such as a terminal. can be recorded on a readable recording medium.

The above description of the present invention is for illustrative purposes, and those skilled in the art can understand that it can be easily modified into other specific forms without changing the technical spirit or essential features of the present invention. will be. Therefore, the embodiments described above should be understood as illustrative in all respects and not limiting. For example, each component described as a single type may be implemented in a distributed manner, and similarly, components described as distributed may be implemented in a combined form.

The scope of the present invention is indicated by the following claims rather than the detailed description above, and all changes or modifications derived from the meaning and scope of the claims and equivalent concepts should be construed as being included in the scope of the present invention. do.

Claims (7)

A user terminal that inputs a country, region, and specialized field requiring proxy visitation, and is matched with an expert who satisfies the conditions of the country, region, and specialized field; an expert terminal that matches the user terminal, connects to the user terminal live in real time at a date, time, and place requested by the user terminal, and transmits a response to a request received from the user terminal; and a construction unit that constructs a pool of experts by receiving expert data including country, region, and specialized field from the expert terminal; , Agency visiting including a matching unit that matches experts satisfying the conditions of the region and specialized field, and a connection unit that connects the user terminal and the expert terminal live in real time at the date, time, and place requested by the user terminal In a known remote agency visiting service providing system including a; service providing server,
The user terminal,
Including a 3D sensor for gesture recognition,
The 3D sensor for recognizing the motion is a MEMS (Micro Electro Mechanical System) based motion tracker or an optical 3D sensor,
The expert terminal is a terminal including a three-dimensional (3D) sensor,
The agency visit service providing server,
an AI interpretation/translation unit for converting the spoken sentence into STT (Speech to Text) when voice speech is input from the expert terminal and then transmitting the result to the user terminal using an artificial intelligence interpretation/translation program;
After recognizing at least one object in the image input through the expert terminal, correcting image distortion of the at least one object, extracting image feature data of the at least one object, and labeling the image with a tag, image identification wealth;
When the user terminal points to at least one object in a screen photographed by the expert terminal using the 3D sensor for motion recognition and then receives a motion drawing along an outline, the object to which the motion is input in the expert terminal an object confirmation unit that outputs; and
When the object in the screen captured by the expert terminal is a person, further comprising a de-identification unit for blur processing;
The connection unit of the proxy visiting service providing server,
If the generated amount of data is large in the uplink, the transmission time transmitted in the uplink is increased, and if the generated amount of data is large in the downlink, the transmission time in the downlink is increased through traffic smoothing related to the real-time live A system for providing remote proxy visiting service using real-time live, characterized in that it processes images. delete delete delete delete delete delete

Images