CN113542332A - Customer service video interaction method and equipment based on positioning labels - Google Patents
Customer service video interaction method and equipment based on positioning labels Download PDFInfo
- Publication number
- CN113542332A CN113542332A CN202010320286.7A CN202010320286A CN113542332A CN 113542332 A CN113542332 A CN 113542332A CN 202010320286 A CN202010320286 A CN 202010320286A CN 113542332 A CN113542332 A CN 113542332A
- Authority
- CN
- China
- Prior art keywords
- customer service
- information
- side equipment
- equipment
- video
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
- 230000003993 interaction Effects 0.000 title claims abstract description 68
- 238000000034 method Methods 0.000 title claims abstract description 53
- 230000002452 interceptive effect Effects 0.000 claims abstract description 54
- 238000004422 calculation algorithm Methods 0.000 claims abstract description 41
- 238000002372 labelling Methods 0.000 claims description 56
- 238000000605 extraction Methods 0.000 claims description 14
- 238000004590 computer program Methods 0.000 claims description 12
- 238000004891 communication Methods 0.000 claims description 7
- 230000004807 localization Effects 0.000 claims 1
- 230000000694 effects Effects 0.000 abstract description 15
- 238000010586 diagram Methods 0.000 description 12
- 230000006870 function Effects 0.000 description 7
- 230000008569 process Effects 0.000 description 6
- 238000012545 processing Methods 0.000 description 5
- 230000000007 visual effect Effects 0.000 description 5
- 238000005516 engineering process Methods 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000006855 networking Effects 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000018109 developmental process Effects 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000000977 initiatory effect Effects 0.000 description 1
- 238000004519 manufacturing process Methods 0.000 description 1
- 239000000463 material Substances 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 238000003860 storage Methods 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000033772 system development Effects 0.000 description 1
- 230000001052 transient effect Effects 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/133—Protocols for remote procedure calls [RPC]
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/50—Network services
- H04L67/51—Discovery or management thereof, e.g. service location protocol [SLP] or web services
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
- User Interface Of Digital Computer (AREA)
Abstract
The invention discloses a customer service video interaction method and device based on positioning labels. The method comprises the following steps: the customer service system equipment receives the video information transmitted by the terminal side equipment and forwards the video information to the customer service side equipment for display, wherein the customer service system equipment comprises an entity list, an identification algorithm and a marking information base; the customer service system equipment selects an identification algorithm according to the target entity, identifies whether the target entity exists in the frame-extracted image updated by the video information according to the identification algorithm, and if so, identifies the position of the target entity to obtain positioning information; and the customer service system equipment receives the interactive information and synchronizes the interactive information and the positioning information to the terminal side equipment and the customer service side equipment so as to realize the display of the marking information on the terminal side equipment and the customer service side equipment. By adopting the customer service video interaction method based on positioning annotation, the interaction mode effect of video information annotation can be obviously improved.
Description
[ technical field ] A method for producing a semiconductor device
The invention relates to the field of Internet application, in particular to a customer service video interaction method and device based on positioning labels.
[ background of the invention ]
With the development of the internet of vehicles, the vehicle intelligent capability is gradually improved, and the application of the Over the Air (OTA for short) technology further improves the improvement speed of related systems. However, the problem of the fault also presents recessiveness and complication, the previous customer service mode is gradually difficult to adapt to a new service scene, and the remote expert customer service system development in a video mode is a trend.
The video customer service system has the advantages that the video access capacity is increased on the basis of the existing voice and online customer service system, richer information sharing is increased compared with the original mode, information of customer service side equipment can be displayed to the customer service side equipment in real time, the customer service system can conveniently and quickly know the field situation, and a user can be timely helped to answer or solve the actual problem. The current interactive mode of the video customer service and the terminal side equipment user is an interactive mode of video information annotation besides a voice mode and a character mode, but the interactive mode of the video information annotation still has a plurality of problems due to objective reasons of complex interactive scenes, high requirements on equipment performance and the like, and the interactive mode of the video information annotation has poor effect.
[ summary of the invention ]
In view of this, embodiments of the present invention provide a method and an apparatus for customer service video interaction based on positioning annotation, so as to solve the problem that an interaction mode of video information annotation at present is poor in effect.
In a first aspect, an embodiment of the present invention provides a customer service video interaction method based on a positioning label, including the following steps executed by a customer service system device:
the method comprises the following steps that customer service system equipment receives video information transmitted by terminal side equipment and forwards the video information to the customer service side equipment for display, wherein the customer service system equipment comprises an entity list, an identification algorithm and a marking information base;
the customer service system equipment selects the identification algorithm according to a target entity, identifies whether the target entity exists in a frame-extracted image updated by the video information according to the identification algorithm, and if so, identifies the position of the target entity to obtain positioning information, wherein the target entity is selected through a specified instruction input by the customer service side equipment based on the entity list;
the customer service system equipment receives interaction information, synchronizes the interaction information and the positioning information to the terminal side equipment and the customer service side equipment, and displays marking information on the terminal side equipment and the customer service side equipment, wherein the interaction information is obtained through the customer service side equipment based on the marking information base.
The above-described aspect and any possible implementation manner further provide an implementation manner, before the selecting, by the customer service system device, the identification algorithm according to the target entity, further including:
and the customer service platform system performs image frame extraction according to the video information at a preset time interval to obtain a frame extraction image updated by the video information, wherein the preset time interval is 1/3 seconds.
In a second aspect, an embodiment of the present invention further provides a customer service video interaction method based on positioning annotation, including the following steps executed by a customer service side device:
the customer service side equipment receives the video information forwarded by the customer service system equipment;
the customer service side equipment selects a target entity by using an entity list according to an input specified instruction, wherein the customer service side equipment is in communication connection with customer service system equipment, and the customer service system equipment comprises the entity list, an identification algorithm and a labeling information base;
based on the labeling information base, the customer service side equipment acquires interactive information edited by customer service personnel;
the customer service side equipment sends the interaction information to the customer service system equipment;
the customer service side equipment receives the interactive information and the positioning information synchronously sent by the customer service system equipment, wherein the positioning information is obtained by the customer service system equipment selecting the identification algorithm according to the target entity and identifying the position of the target entity identified in the frame-drawing image updated by the video information according to the identification algorithm;
and the customer service side equipment realizes the display of the marking information based on the interactive information and the positioning information, wherein the interactive information is used for displaying the marking information of character types, and the positioning information is used for displaying the marking information of graphic types.
The foregoing aspect and any possible implementation manner further provide an implementation manner, where the customer service side device displays the annotation information based on the interaction information and the positioning information, and the implementation manner includes:
the customer service side equipment circles the target entity by adopting the positioning information and a preset labeling formula to realize the display of entity circled labeling information;
the customer service side equipment generates a click area on an area for circling the target entity based on the interactive information, the click area is used for displaying the interactive information and an information reply area after receiving a click instruction, wherein the interactive information comprises text information and information, the information is displayed in a hyperlink mode, and the information reply area is used for displaying or replying reply information sent by the terminal side equipment;
and the customer service side equipment realizes the display of the serial numbers of different target entities according to the positioning information based on the pre-edited serial numbers of different target entities.
The foregoing aspect and any possible implementation manner further provide an implementation manner, where the positioning information includes a central position (x, y) and a region pixel size (a, b), and the customer service-side device circles the target entity by using the positioning information and a preset labeling formula to realize display of labeled information encircled by the entity, including:
the customer service side equipment calculates to obtain a circle graphic by taking a pixel point set as F (Xn, Yn) according to the central position (x, y) and the area pixel size (a, b),is given by the formula
And setting the display position of the serial number display of the target entity to be (X + a/2, Y + b/2).
In a third aspect, an embodiment of the present invention further provides a method for customer service video interaction based on positioning annotation, including the following steps executed by a terminal-side device:
the terminal side equipment transmits video information to the customer service system equipment, wherein the customer service system equipment comprises an entity list, an identification algorithm and a labeling information base;
the terminal side equipment receives interactive information and positioning information synchronously sent by the customer service system equipment, wherein the interactive information is obtained through the customer service side equipment based on the labeling information base, the positioning information is obtained by selecting the identification algorithm according to a target entity by the customer service system equipment and identifying the position of the target entity identified in the frame-drawing image updated by the video information according to the identification algorithm, and the target entity is selected through a specified instruction input by the customer service side equipment based on the entity list;
and the terminal side equipment realizes the display of the marking information based on the interactive information and the positioning information, wherein the interactive information is used for displaying the marking information of character types, and the positioning information is used for displaying the marking information of graphic types.
The above-described aspect and any possible implementation manner further provide an implementation manner, where the terminal-side device sequentially displays the annotation information on a device display interface of the terminal-side device in a chronological order according to a guidance history acquisition instruction input by a user, where switching display of the annotation information on the chronological order is achieved through a pause instruction, a forward instruction, or a rewind instruction input by the user.
In a fourth aspect, an embodiment of the present invention further provides customer service system equipment, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor, when executing the computer program, implements the steps of the method for customer service video interaction based on positioning labels according to the first aspect.
In a fifth aspect, an embodiment of the present invention further provides customer service side equipment, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor, when executing the computer program, implements the steps of the customer service video interaction method based on positioning labels according to the second aspect.
In a sixth aspect, an embodiment of the present invention further provides a terminal-side device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, where the processor, when executing the computer program, implements the steps of the method for interacting customer service video based on positioning annotation according to the third aspect.
In the embodiment of the invention, most steps needing data processing are distributed to the customer service system equipment and the customer service side equipment for execution, the terminal side equipment only needs to transmit video information to the customer service system equipment and receive returned interaction information and positioning information to realize the display of the marking information, and the requirement on the equipment performance of the terminal side equipment is not high; in addition, the embodiment of the invention completes the positioning and labeling of the target entity on the basis of the real-time updated image identification, and can more accurately highlight the labeled target entity by combining the display of the labeling information of the character type and the image type, thereby improving the communication efficiency between the terminal side equipment user and the customer service. In the embodiment of the invention, the equipment performance of the terminal side equipment can be reduced, the marking display effect of the terminal side equipment can be improved, the positioning marking can be more accurately realized, and the interactive mode effect of video information marking is obviously improved.
[ description of the drawings ]
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings needed to be used in the embodiments will be briefly described below, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and it is obvious for those skilled in the art to obtain other drawings based on these drawings without inventive labor.
FIG. 1 is a schematic diagram of a prior art labeling method according to an embodiment of the present invention;
FIG. 2 is a schematic diagram of a prior art labeling method according to an embodiment of the present invention;
FIG. 3 is another schematic diagram of a prior art labeling method according to an embodiment of the present invention;
FIG. 4a is a schematic diagram illustrating the position relationship between the annotation and the actual object under the first viewing angle in accordance with an embodiment of the present invention;
FIG. 4b is a schematic diagram illustrating the position relationship between the annotation and the actual object under the second viewing angle in an embodiment of the invention;
FIG. 5 is a flowchart of a method for positioning annotation based customer service video interaction according to an embodiment of the present invention;
FIG. 6 is a system architecture diagram of a positioning annotation based customer service video interaction method according to an embodiment of the present invention;
FIG. 7 is a flowchart of a method for interaction of customer service video based on positioning labels according to an embodiment of the present invention;
FIG. 8 is a schematic diagram of an encircled label in an embodiment of the present invention.
[ detailed description ] embodiments
For better understanding of the technical solutions of the present invention, the following detailed descriptions of the embodiments of the present invention are provided with reference to the accompanying drawings.
It should be understood that the described embodiments are only some embodiments of the invention, and not all embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
The terminology used in the embodiments of the invention is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used in the examples of the present invention and the appended claims, the singular forms "a," "an," and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise.
It should be understood that the term "and/or" as used herein is merely a field that describes the same of an associated object, meaning that three relationships may exist, e.g., A and/or B, may indicate: a is present alone, and A and B are present, B is present alone. In addition, the character "/" herein generally indicates that the former and latter related objects are in an "or" relationship.
It should be understood that although the terms first, second, third, etc. may be used to describe preset ranges, etc. in embodiments of the present invention, these preset ranges should not be limited to these terms. These terms are only used to distinguish preset ranges from each other. For example, the first preset range may also be referred to as a second preset range, and similarly, the second preset range may also be referred to as the first preset range, without departing from the scope of the embodiments of the present invention.
The word "if" as used herein may be interpreted as "at … …" or "when … …" or "in response to a determination" or "in response to a detection", depending on the context. Similarly, the phrases "if determined" or "if detected (a stated condition or event)" may be interpreted as "when determined" or "in response to a determination" or "when detected (a stated condition or event)" or "in response to a detection (a stated condition or event)", depending on the context.
In the prior art, the video information annotation has the following three interaction modes:
the marking method comprises the following steps:
performing circle drawing after video screenshot: and the camera of the customer service side equipment sends the acquired video to a remote customer service instructor, and the customer service instructor marks the screenshot of the picture and sends the screenshot back to the customer service side equipment to cover the video.
The schematic diagram of the implementation is shown in fig. 1, wherein the picture in fig. 1 is a static picture.
And a second labeling method:
continuous dotting marking on the video: after the camera of the customer service side equipment sends the collected video to the remote customer service instructor, the customer service instructor continuously clicks on the interaction layer in real time, and real-time instruction on the client is realized through the data synchronization of the interaction layers at two ends.
The schematic diagram of the implementation is shown in fig. 2, and the annotation is implemented by dotting on the video.
And a third labeling method:
labeling by using an Augmented Reality (AR) technology: after the camera of the customer service side equipment sends the collected video to the remote customer service instructor, the customer service side equipment utilizes the AR marking tool to circle the first visual angle, the AR marking tool is calculated based on space, the circle is displayed in a virtual mode under a real coordinate system, the same visual angle is kept for observation, and the circle is corresponding to the target entity.
A schematic diagram of its implementation is shown in fig. 3.
The above three labeling methods all have a number of problems, specifically:
the main problems of the labeling method I are as follows:
1) in the video interaction process, functional interfaces such as pictures, videos and interactive text information need to be switched, and the operation is complicated.
2) Related interaction is fragmented, and a uniform service process cannot be formed for complex interaction requirements.
3) Various types of interactive data such as characters, videos, pictures and the like can appear in related interaction, and more new requirements are put forward for later service management.
The labeling method II has the main problems that:
1) the labeling method is a transient service method, and can not form memorable and instructive contents for complex instructive interaction.
2) The interactive method is difficult to form the association of interactive contents, and challenges are formed on the traceability and the service quality management of the service.
3) The dotting can affect the actual scene, and under the conditions of video interaction equipment, actual phase shifting and the like, interaction visual errors can occur, so that the situation of unsmooth communication of guidance requiring complex operation is caused.
The labeling method has the main problems that:
1) the method has higher requirement on customer service side equipment and needs special equipment.
2) For the object marking, under the simple tool, the method is difficult to realize the coincidence of the circle marking and the plane of the actual object, so that the marked target cannot be seen after the observation visual angle deviates. If accuracy is required, professional techniques are required for spatial labeling.
Fig. 4a shows a schematic diagram of the positional relationship of the enclosing annotation to the actual object in the case of the first viewing angle. By way of comparison, fig. 4b shows a further schematic illustration of the positional relationship of the enclosing annotation to the actual object in the case of the second viewing angle. As can be seen from fig. 4a and 4b, the position relationship between the annotation for circling and the plane of the actual object is still matched under different viewing angles (the plane where the annotation for circling is located after the viewing angle is converted and the plane near the actual object are still parallel), that is, the annotation for circling is adaptively adjusted along with the change of the viewing angle of the video, so that the annotation for circling can accurately circle the actual object under different viewing angles. It can be understood that the labeling in this way is very effective, but has high requirements for the terminal side device and the customer service side device, requires a dedicated device, requires a professional technique for spatial labeling, and has a poor interaction effect for the performance of the common devices in a daily scene.
In summary, it can be seen that the annotation methods in the prior art have various problems, which result in poor interactive mode effect of video information annotation.
In view of this, the embodiment of the present invention provides a customer service video interaction method based on positioning annotation.
FIG. 5 shows a flowchart of a method for customer service video interaction based on positioning labels in this embodiment. The customer service video interaction method based on the positioning label can be applied to a customer service video interaction system of the positioning label, and can be realized by adopting the customer service video interaction system of the positioning label when the customer service video interaction of the positioning label is carried out. The positioning and labeling customer service video interaction system can be applied to various different computer devices, wherein the computer devices include but are not limited to computers, smart phones, tablets, servers and other devices. As shown in fig. 5, the method for customer service video interaction based on positioning annotation includes the following steps:
s10: and the customer service system equipment receives the video information transmitted by the terminal side equipment and forwards the video information to the customer service side equipment for display, wherein the customer service system equipment comprises an entity list, an identification algorithm and a labeling information base.
The video information may specifically refer to streaming media information.
In an embodiment, the positioning annotation based customer service video interaction method is used in a positioning annotation based customer service video interaction system, and the system includes terminal side equipment, such as a mobile phone and a tablet of a vehicle owner. The system also comprises a customer service side, wherein the customer service side comprises customer service system equipment and a customer service side system, the customer service system equipment can be understood as cloud service platform equipment for providing data query and algorithm service, and the customer service side system can be understood as equipment such as a mobile phone, a computer or a tablet of a customer service technician. In the customer service video interaction system based on the positioning annotation, the terminal side equipment and the customer service side equipment realize the display of the annotation information through the customer service system equipment.
In one embodiment, the terminal device transmits the video information to the customer service system device, and the customer service system device receives the video information and then forwards the video information to the customer service side device to be displayed on the customer service side device.
It can be understood that the customer service system device serves as a cloud service platform, and service contents with high data processing and storage requirements are all implemented on the customer service system device. The customer service system equipment comprises an entity list, an identification algorithm and a labeling information base. Specifically, for a special scene of the car networking customer service, entities related to the car are clear and limited, a list of entities to be identified can be provided in an exhaustive manner, and a special identification algorithm and a labeling information base are customized by combining corresponding entities and environments. The entity list can be understood by combining the car networking customer service scene, and the entity list can specifically refer to a list of entities related to the car; the marking information corresponding to the entity is stored in the marking information base so as to specially mark a certain entity, thereby achieving a better marking effect; the recognition algorithm is a recognition algorithm specially designed in a scene related to the vehicle, has a recognition function with high accuracy and a position recognition function, and can be specifically realized through an image recognition system when the recognition algorithm is used for recognition.
S20: and the customer service side equipment selects a target entity by using the entity list according to the input specified instruction.
It will be appreciated that when a user (e.g., owner) has operational difficulties with certain entities, the operation will be performed through visual guidance by a customer service technician. First, an entity to be operated by a user, i.e., a target entity, needs to be determined. Specifically, the customer service and the user may communicate with the target entity in advance, and then the customer service technician may select the entity in an entity list provided by the customer service system device in order to implement functions such as labeling of the target entity, and select the target entity according to a specified instruction by inputting the specified instruction (e.g., clicking operation), so as to determine an object to be labeled next.
S30: the customer service system equipment selects an identification algorithm according to the target entity, identifies whether the target entity exists in the frame-extracted image updated by the video information according to the identification algorithm, and if so, identifies the position of the target entity to obtain the positioning information.
Further, before step S30, that is, before the step of the customer service system device selecting the identification algorithm according to the target entity, the method further includes:
and the customer service platform system performs image frame extraction according to the video information at a preset time interval to obtain a frame extraction image updated by the video information, wherein the preset time interval is 1/3 seconds.
In an embodiment, the customer service platform system performs image frame extraction on the video information every 1/3 seconds to obtain an image after each frame extraction, that is, a frame extraction image updated by the video information. It can be understood that the customer service platform system performs image recognition and other processing on the image which is updated recently, that is, the frame-extracted image updated by the video information.
The preset time interval can be set to be other time intervals, and the embodiment adopts 1/3-second time intervals, so that excessive identification calculation can be avoided under the condition of ensuring the identification timeliness.
In step S30, the customer service system device will use the recognition algorithm to identify the target entity from the decimated image of the video information update. It should be noted that, after the customer service system device performs image frame extraction according to the video information each time to obtain a frame extraction image updated by the video information, the customer service system device re-identifies the target entity, for example, after every 1/3 seconds, the customer service system device identifies the target entity from the frame extraction image updated by the video information by using an identification algorithm. It is understood that the viewing angle of the target entity may change during the video capture process. The target entity is re-identified every 1/3 seconds, which can achieve the effect of tracking the target entity, and when the observation angle of the target entity changes, the display of highly accurate labeling information can be achieved based on the tracking effect. For example, when the target entity is indicated by an ellipse circle, the ellipse that encloses the target entity will change on the device display interface with time, and as the target entity "moves," the target entity is accurately enclosed and labeled at different observation angles.
S40: based on the labeling information base, the customer service side equipment acquires the interactive information edited by the customer service personnel.
In an embodiment, after the customer service technician selects an entity target related to the user communication in the entity list, the interaction information edited on the customer service side equipment is displayed as the labeled information on the terminal side equipment by referring to the labeled information base according to the service requirement. The interactive information includes text information, data information and other character information for interaction. Specifically, the data information can be displayed in a hyperlink mode, and a user can jump to an interface of the detailed data information related to the target entity after clicking the hyperlink.
It can be understood that the customer service technician can quickly complete the text information and the data information which are desired to be labeled on the customer service side equipment based on the labeling information base.
Further, an entity operation tool is provided on the device display interface of the customer service side device, and is used for retrieving the entity and initiating the display requirement of the label information encircled by the entity.
The entity operation tool refers to a tool for operating an entity on a device display interface of the customer service side device, such as a search tool (including a text input field for searching, and searching is started by clicking a search button), and may further include an interactive information editing tool for quickly editing the interactive information. In addition, the entity operation tool can also initiate the display requirement of the annotation information encircled by the entity. Specifically, whether to display the entity circle on the terminal device may be set on the customer service side device, for example, if yes, when the label information is displayed on the terminal side, a graphic such as an ellipse is used to circle the target entity.
Further, the customer service side equipment is connected with the customer service system equipment, and the customer service side equipment utilizes the entity list and the labeled information base of the customer service system equipment to realize serial number editing and addition of text information and data information. It can be understood that the customer service side device can also number different target entities, and select appropriate text information and material information to be added in different target entities.
S50: and the customer service system equipment receives the interactive information and synchronizes the interactive information and the positioning information to the terminal side equipment and the customer service side equipment.
In an embodiment, the customer service system device receives the interactive information sent by the customer service side device, and synchronizes the positioning information to the terminal side device and the customer service side device together, so that the display of the labeled information can be realized on the terminal side device and the customer service side device, and a customer service technician can check whether the labeled interactive information is correct or not.
S60: and the terminal side equipment and the customer service side equipment realize the display of the marking information based on the interactive information and the positioning information, wherein the interactive information is used for displaying the marking information of the character type, and the positioning information is used for displaying the marking information of the graphic type.
It is understood that the annotation information may include annotation of text type, such as interactive information; graphical type annotations may also be included, such as ellipses circled for the target entity. In an embodiment, after the terminal-side device and the customer-side device obtain the interaction information and the positioning information synchronously, the labeling information can be displayed on the device display interfaces of the respective devices.
Further, the device display interface of the terminal side device comprises a video display layer and an annotation display layer, wherein the video display layer is used for displaying a video on the device display interface according to the input video information, the annotation display layer is used for displaying annotation information, and the video display layer and the annotation display layer are consistent in the display origin position and the display pixels. It is understood that the device has two display layers on the display interface of the device, one for displaying the native video and one for displaying the annotation information.
Further, in step S60, the displaying of the annotation information by the terminal-side device and the customer-side device based on the interaction information and the positioning information includes:
s61: and the terminal side equipment and the customer service side equipment adopt the positioning information and a preset labeling formula to circle the target entity, so that the display of the labeled information of entity circle is realized.
In an embodiment, under the condition that the positioning information of the target entity is known, the target entity may be circled by using a preset labeling formula, such as an ellipse generating formula, a circle generating formula, a rectangle generating formula, and the like, to generate a corresponding graph, so as to realize display of the labeling information circled by the entity.
S62: the terminal side equipment and the customer service side equipment generate a click area on an area for circling the target entity based on the interactive information, the click area is used for displaying the interactive information and an information reply area after receiving a click instruction, and the information reply area is used for the terminal side equipment to receive reply information input by a user.
It can be understood that the user can click on the area of the graphical circle and display the interactive information and the information reply area on the clicked device display interface. Furthermore, only circled annotation information is generally displayed on the annotation display layer, and a user can click to check interactive information related to a target entity, and can input characters in the information reply area for reply and communicate with customer service technicians. Similarly, the customer service technician can reply the reply message sent by the user through the message reply area.
S63: and the terminal side equipment and the customer service side equipment realize the display of the serial numbers of different target entities according to the positioning information based on the pre-edited serial numbers of the different target entities.
It can be understood that the embodiment of the invention supports the simultaneous labeling of a plurality of different target entities, and a user can view the plurality of target entities according to the sequence numbers displayed by the target entities.
Further, the positioning information specifically includes a center position (x, y) and an area pixel size (a, b).
In step S61, the terminal side device and the customer service side device perform circle drawing on the target entity by using the positioning information and a preset labeling formula, so as to realize display of labeled information of entity circle drawing, including:
the terminal side equipment calculates to obtain a circled graph by taking the pixel point set as F (Xn, Yn) according to the central position (x, y) and the area pixel size (a, b), and the formula is
The display position of the serial number display of the set target entity may be (X + a/2, Y + b/2), and the display position of the serial number display of the set target entity may be other positions, which is not limited herein. It will be appreciated that an ellipse circumscribing the target entity can be obtained using the above formula.
Further, the customer service video interaction method based on the positioning label further comprises the following steps:
the terminal side equipment acquires an instruction according to the guide history input by the user and sequentially displays the labeling information on an equipment display interface of the terminal side equipment according to the time sequence, wherein the labeling information is switched and displayed on the time sequence through a pause instruction, a forward instruction or a backward instruction input by the user.
In an embodiment, the customer service application of the terminal-side device may further include a history viewing function, and the annotation information may be sequentially displayed on the device display interface through a history acquisition instruction (for example, clicking a history viewing button) input by the user, and generally displayed from the earliest annotation information, and the user may switch to display the corresponding annotation information by clicking, pausing, forwarding, or rewinding buttons, or inputting a designated time point.
In one embodiment, a system architecture diagram of a location-annotation based customer service video interaction method is shown in fig. 6. As can be seen from fig. 6, the customer service system device receives the video information transmitted by the terminal-side device, and obtains the positioning information through recognition by a recognition algorithm. And the customer service side equipment realizes the control of the marking information according to the editing of the customer service technician and sends the marking information to the customer service system equipment. And finally, the customer service system device synchronizes the positioning information and the labeling information to the terminal side device and the customer service side device to realize the display of the labeling information.
In one embodiment, as shown in fig. 7, a flowchart of a method for customer service video interaction based on positioning labels is provided:
1) the terminal side equipment or the customer service side equipment can initiate the related video customer service capability.
2) And the video information acquired by the terminal side equipment is forwarded to a customer service side system through customer service platform equipment for display, meanwhile, the customer service platform equipment performs timed frame extraction on the video at proper time intervals, and the acquired latest image is used as the current image identification application reference.
3) The customer service side equipment selects a target entity from the entity list according to the service requirement, and the serial number, the text information, the data information and the like can be operated and edited based on the customer service side equipment to form interactive information which is transmitted to the customer service system equipment.
4) After the customer service platform equipment acquires the currently marked interaction information, firstly, the image recognition system is called, and an optimal recognition algorithm is determined according to the target entity.
5) The image recognition system adopts a recognition algorithm to recognize the current latest video frame extraction image obtained in the step 2), judges whether a target entity exists, and further recognizes the central coordinates (x, y) of the target entity and the area pixel size (a, b) of the target entity if the target entity exists.
6) The customer service platform device synchronizes the information such as positioning information (center coordinates, area pixel size) and the like with the text information, the data information and the like received by the customer service side device to the terminal side and the customer service side system for marking and displaying.
7) The marking display layers of the terminal side equipment and the customer service side equipment adopt the optimal ellipse for circle marking according to the positioning information and the interaction information transmitted by the customer service platform equipment and based on the center position (x, y) and the area pixel size (a, b) of feedback, and the specific algorithm is as follows:
the display position of the serial number display of the setting target entity may be (X + a/2, Y + b/2). Further, on the label display layer, the pixel points at the positions of the circled pixel points are displayed in a color with preset transparency (such as 70%) so as to highlight the label of the target entity. Specifically, the process outlined is shown in FIG. 8.
8) When step 2 refreshes the latest image, steps 4), 5), 6), 7) are re-executed.
The embodiment of the invention adopts an image recognition-based labeling technology, and can realize the tracking and labeling of the target object. In a common life scene, the terminal-side device is a general terminal device, and the capability of realizing image identification and annotation requires high-strength computing capability, so that the terminal-side device cannot be applied; in addition, the image recognition algorithm is mainly used for recognizing a specific object in a specific environment of a specific scene, is generally difficult to adapt universally in a complex scene, needs to be developed by matching of multiple algorithms, and increases the calculation cost by times. In the embodiment of the invention, innovative system architectures (terminal side equipment, customer service system equipment and customer service side equipment) are adopted, the problems are mainly realized by the customer service system equipment, and the equipment requirement on the terminal side equipment is not high. Therefore, the embodiment of the invention adopts the annotation technology of image recognition, can be applied in common life scenes, and improves the interactive mode effect of video information annotation.
In the embodiment of the invention, most steps needing data processing are distributed to the customer service system equipment and the customer service side equipment for execution, the terminal side equipment only needs to transmit video information to the customer service system equipment and receive returned interaction information and positioning information to realize the display of the marking information, and the requirement on the equipment performance of the terminal side equipment is not high; in addition, the embodiment of the invention completes the positioning and labeling of the target entity on the basis of the real-time updated image identification, and can more accurately highlight the labeled target entity by combining the display of the labeling information of the character type and the image type, thereby improving the communication efficiency of the client and the customer service of the terminal side equipment. In the embodiment of the invention, the equipment performance of the terminal side equipment can be reduced, the marking display effect of the terminal side equipment can be improved, the positioning marking can be more accurately realized, and the interactive mode effect of video information marking is obviously improved.
It should be understood that, the sequence numbers of the steps in the foregoing embodiments do not imply an execution sequence, and the execution sequence of each process should be determined by its function and inherent logic, and should not constitute any limitation to the implementation process of the embodiments of the present invention.
The embodiment of the invention provides a positioning annotation based customer service video interaction system, which comprises terminal side equipment, customer service system equipment and customer service side equipment, wherein the steps executed by the terminal side equipment, the customer service system equipment and the customer service side equipment are the steps executed by the terminal side equipment, the customer service system equipment and the customer service side equipment in the positioning annotation based customer service video interaction method in the embodiment, and are not described again.
In the embodiment of the invention, most steps needing data processing are distributed to the customer service system equipment and the customer service side equipment for execution, the terminal side equipment only needs to transmit video information to the customer service system equipment and receive returned interaction information and positioning information to realize the display of the marking information, and the requirement on the equipment performance of the terminal side equipment is not high; in addition, the embodiment of the invention completes the positioning and labeling of the target entity on the basis of the real-time updated image identification, and can more accurately highlight the labeled target entity by combining the display of the labeling information of the character type and the image type, thereby improving the communication efficiency of the client and the customer service of the terminal side equipment. In the embodiment of the invention, the equipment performance of the terminal side equipment can be reduced, the marking display effect of the terminal side equipment can be improved, the positioning marking can be more accurately realized, and the interactive mode effect of video information marking is obviously improved.
It will be apparent to those skilled in the art that, for convenience and brevity of description, only the above-mentioned division of the functional units and modules is illustrated, and in practical applications, the above-mentioned function distribution may be performed by different functional units and modules according to needs, that is, the internal structure of the apparatus is divided into different functional units or modules, so as to perform all or part of the functions described above.
The above examples are only intended to illustrate the technical solution of the present invention, but not to limit it; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; such modifications and substitutions do not substantially depart from the spirit and scope of the embodiments of the present invention, and are intended to be included within the scope of the present invention.
Claims (10)
1. A customer service video interaction method based on positioning labels is characterized by comprising the following steps:
the method comprises the following steps that customer service system equipment receives video information transmitted by terminal side equipment and forwards the video information to the customer service side equipment for display, wherein the customer service system equipment comprises an entity list, an identification algorithm and a marking information base;
the customer service system equipment selects the identification algorithm according to a target entity, identifies whether the target entity exists in a frame-extracted image updated by the video information according to the identification algorithm, and if so, identifies the position of the target entity to obtain positioning information, wherein the target entity is selected through a specified instruction input by the customer service side equipment based on the entity list;
the customer service system equipment receives interaction information, synchronizes the interaction information and the positioning information to the terminal side equipment and the customer service side equipment, and displays marking information on the terminal side equipment and the customer service side equipment, wherein the interaction information is obtained through the customer service side equipment based on the marking information base.
2. The method of claim 1, wherein prior to the customer service system device selecting the recognition algorithm based on the target entity, further comprising:
and the customer service platform system performs image frame extraction according to the video information at a preset time interval to obtain a frame extraction image updated by the video information, wherein the preset time interval is 1/3 seconds.
3. A customer service video interaction method based on positioning labels is characterized by comprising the following steps:
the customer service side equipment receives the video information forwarded by the customer service system equipment;
the customer service side equipment selects a target entity by using an entity list according to an input specified instruction, wherein the customer service side equipment is in communication connection with customer service system equipment, and the customer service system equipment comprises the entity list, an identification algorithm and a labeling information base;
based on the labeling information base, the customer service side equipment acquires interactive information edited by customer service personnel;
the customer service side equipment sends the interaction information to the customer service system equipment;
the customer service side equipment receives the interactive information and the positioning information synchronously sent by the customer service system equipment, wherein the positioning information is obtained by the customer service system equipment selecting the identification algorithm according to the target entity and identifying the position of the target entity identified in the frame-drawing image updated by the video information according to the identification algorithm;
and the customer service side equipment realizes the display of the marking information based on the interactive information and the positioning information, wherein the interactive information is used for displaying the marking information of character types, and the positioning information is used for displaying the marking information of graphic types.
4. The method of claim 3, wherein the customer service side device displays annotation information based on the interaction information and the positioning information, and the method comprises:
the customer service side equipment circles the target entity by adopting the positioning information and a preset labeling formula to realize the display of entity circled labeling information;
the customer service side equipment generates a click area on an area for circling the target entity based on the interactive information, the click area is used for displaying the interactive information and an information reply area after receiving a click instruction, wherein the interactive information comprises text information and information, the information is displayed in a hyperlink mode, and the information reply area is used for displaying or replying reply information sent by the terminal side equipment;
and the customer service side equipment realizes the display of the serial numbers of different target entities according to the positioning information based on the pre-edited serial numbers of different target entities.
5. The method according to claim 4, wherein the positioning information includes a center position (x, y) and a region pixel size (a, b), and the customer service-side device circles the target entity by using the positioning information and a preset labeling formula to realize the display of labeled information of entity circle, including:
and the customer service side equipment calculates to obtain a circle graph by taking the pixel point set as F (Xn, Yn) according to the central position (x, y) and the area pixel size (a, b), wherein the formula is
Wherein the display position of the sequence number display of the target entity is set as (A)X+a/2，Y+b/2)。
6. A customer service video interaction method based on positioning labels is characterized by comprising the following steps:
the terminal side equipment transmits video information to the customer service system equipment, wherein the customer service system equipment comprises an entity list, an identification algorithm and a labeling information base;
the terminal side equipment receives interactive information and positioning information synchronously sent by the customer service system equipment, wherein the interactive information is obtained through the customer service side equipment based on the labeling information base, the positioning information is obtained by selecting the identification algorithm according to a target entity by the customer service system equipment and identifying the position of the target entity identified in the frame-drawing image updated by the video information according to the identification algorithm, and the target entity is selected through a specified instruction input by the customer service side equipment based on the entity list;
and the terminal side equipment realizes the display of the marking information based on the interactive information and the positioning information, wherein the interactive information is used for displaying the marking information of character types, and the positioning information is used for displaying the marking information of graphic types.
7. The method of claim 6, further comprising:
the terminal side equipment acquires an instruction according to a guide history input by a user, and sequentially displays the labeling information on an equipment display interface of the terminal side equipment according to a time sequence, wherein the labeling information is switched and displayed on the time sequence through a pause instruction, a forward instruction or a backward instruction input by the user.
8. Customer service system device comprising a memory, a processor and a computer program stored in said memory and executable on said processor, characterized in that said processor, when executing said computer program, implements the steps of the location annotation based customer service video interaction method according to claim 1 or 2.
9. Customer service side device comprising a memory, a processor and a computer program stored in said memory and executable on said processor, characterized in that said processor when executing said computer program realizes the steps of the method for customer service video interaction based on localization annotation according to any of claims 3-5.
10. Terminal-side device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that the processor when executing the computer program realizes the steps of the location annotation based customer service video interaction method according to claim 6 or 7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010320286.7A CN113542332B (en) | 2020-04-22 | 2020-04-22 | Customer service video interaction method and equipment based on positioning labels |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010320286.7A CN113542332B (en) | 2020-04-22 | 2020-04-22 | Customer service video interaction method and equipment based on positioning labels |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113542332A true CN113542332A (en) | 2021-10-22 |
| CN113542332B CN113542332B (en) | 2023-04-07 |
Family
ID=78094025
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010320286.7A Active CN113542332B (en) | 2020-04-22 | 2020-04-22 | Customer service video interaction method and equipment based on positioning labels |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113542332B (en) |
Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090092374A1 (en) * | 2007-10-07 | 2009-04-09 | Kulas Charles J | Digital Network-Based Video Tagging System |
| CN106339094A (en) * | 2016-09-05 | 2017-01-18 | 山东万腾电子科技有限公司 | Interactive remote expert cooperation maintenance system and method based on augmented reality technology |
| CN107333087A (en) * | 2017-06-27 | 2017-11-07 | 京东方科技集团股份有限公司 | A kind of information sharing method and device based on video session |
| CN108170273A (en) * | 2017-12-28 | 2018-06-15 | 南京华讯方舟通信设备有限公司 | A kind of expert's remote guide system and implementation method based on hololens glasses |
| WO2018107676A1 (en) * | 2016-12-15 | 2018-06-21 | 深圳市云鹏正曜科技发展有限公司 | Information interaction method, and data collection method and device |
| WO2019080873A1 (en) * | 2017-10-27 | 2019-05-02 | 腾讯科技（深圳）有限公司 | Method for generating annotations and related apparatus |
| WO2019128787A1 (en) * | 2017-12-26 | 2019-07-04 | 阿里巴巴集团控股有限公司 | Network video live broadcast method and apparatus, and electronic device |
| CN110266992A (en) * | 2019-06-24 | 2019-09-20 | 苏芯物联技术(南京)有限公司 | A kind of long-distance video interactive system and method based on augmented reality |
| CN110443294A (en) * | 2019-07-25 | 2019-11-12 | 丰图科技(深圳)有限公司 | Video labeling method, device, server, user terminal and storage medium |
| CN110647636A (en) * | 2019-09-05 | 2020-01-03 | 深圳追一科技有限公司 | Interaction method, interaction device, terminal equipment and storage medium |
| CN110717994A (en) * | 2019-10-21 | 2020-01-21 | 联想(北京)有限公司 | Method for realizing remote video interaction and related equipment |
-
2020
- 2020-04-22 CN CN202010320286.7A patent/CN113542332B/en active Active
Patent Citations (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20090092374A1 (en) * | 2007-10-07 | 2009-04-09 | Kulas Charles J | Digital Network-Based Video Tagging System |
| CN106339094A (en) * | 2016-09-05 | 2017-01-18 | 山东万腾电子科技有限公司 | Interactive remote expert cooperation maintenance system and method based on augmented reality technology |
| WO2018107676A1 (en) * | 2016-12-15 | 2018-06-21 | 深圳市云鹏正曜科技发展有限公司 | Information interaction method, and data collection method and device |
| CN107333087A (en) * | 2017-06-27 | 2017-11-07 | 京东方科技集团股份有限公司 | A kind of information sharing method and device based on video session |
| WO2019080873A1 (en) * | 2017-10-27 | 2019-05-02 | 腾讯科技（深圳）有限公司 | Method for generating annotations and related apparatus |
| WO2019128787A1 (en) * | 2017-12-26 | 2019-07-04 | 阿里巴巴集团控股有限公司 | Network video live broadcast method and apparatus, and electronic device |
| CN108170273A (en) * | 2017-12-28 | 2018-06-15 | 南京华讯方舟通信设备有限公司 | A kind of expert's remote guide system and implementation method based on hololens glasses |
| CN110266992A (en) * | 2019-06-24 | 2019-09-20 | 苏芯物联技术(南京)有限公司 | A kind of long-distance video interactive system and method based on augmented reality |
| CN110443294A (en) * | 2019-07-25 | 2019-11-12 | 丰图科技(深圳)有限公司 | Video labeling method, device, server, user terminal and storage medium |
| CN110647636A (en) * | 2019-09-05 | 2020-01-03 | 深圳追一科技有限公司 | Interaction method, interaction device, terminal equipment and storage medium |
| CN110717994A (en) * | 2019-10-21 | 2020-01-21 | 联想(北京)有限公司 | Method for realizing remote video interaction and related equipment |
Non-Patent Citations (1)
| Title |
|---|
| 张环等: "基于分层语义的体育视频标注及索引研究", 《计算机应用与软件》 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113542332B (en) | 2023-04-07 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20210120221A1 (en) | Augmented reality-based remote guidance method and apparatus, terminal, and storage medium | |
| US9256986B2 (en) | Automated guidance when taking a photograph, using virtual objects overlaid on an image | |
| US9639988B2 (en) | Information processing apparatus and computer program product for processing a virtual object | |
| US10677596B2 (en) | Image processing device, image processing method, and program | |
| EP2733629B1 (en) | System for associating tag information with images supporting image feature search | |
| EP3550479A1 (en) | Augmented-reality-based offline interaction method and apparatus | |
| US8554250B2 (en) | Remote object recognition | |
| US20150046299A1 (en) | Inventory Assessment with Mobile Devices | |
| US20120124509A1 (en) | Information processor, processing method and program | |
| US20190333478A1 (en) | Adaptive fiducials for image match recognition and tracking | |
| US20150070347A1 (en) | Computer-vision based augmented reality system | |
| JP6230113B2 (en) | Video instruction synchronization method, system, terminal, and program for synchronously superimposing instruction images on captured moving images | |
| Anagnostopoulos et al. | Gaze-Informed location-based services | |
| US20190122435A1 (en) | Generating time-delayed augmented reality content | |
| CN108492349B (en) | Processing method, device and equipment for writing strokes and storage medium | |
| CN110111241A (en) | Method and apparatus for generating dynamic image | |
| JP2024502516A (en) | Data annotation methods, apparatus, systems, devices and storage media | |
| US20130127822A1 (en) | Object data search systems and methods | |
| CN112270242B (en) | Track display method and device, readable medium and electronic equipment | |
| CN114140536A (en) | Pose data processing method and device, electronic equipment and storage medium | |
| CN113542332B (en) | Customer service video interaction method and equipment based on positioning labels | |
| JP6146869B2 (en) | Video instruction display method, system, terminal, and program for superimposing instruction image on photographing moving image synchronously | |
| Goose et al. | Paris: fusing vision-based location tracking with standards-based 3d visualization and speech interaction on a PDA | |
| EP3863235B1 (en) | Method and apparatus for processing data | |
| TWM542190U (en) | Augmented reality cloud and smart service searching system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |