WO2020259510A1

WO2020259510A1 - Method and apparatus for detecting information embedding region, electronic device, and storage medium

Info

Publication number: WO2020259510A1
Application number: PCT/CN2020/097782
Authority: WO
Inventors: 生辉; 黄东波
Original assignee: 腾讯科技（深圳）有限公司
Priority date: 2019-06-28
Filing date: 2020-06-23
Publication date: 2020-12-30
Also published as: CN112153483B; CN112153483A; US12073621B2; US20210406549A1

Abstract

Provided in embodiments of the present application are a method and apparatus for detecting an information embedding region, an electronic device, and a computer storage medium. The method comprises: obtaining a video to be embedded, and segmenting the video to be embedded so as to obtain video segments; obtaining a target frame in the video segments, and identifying and segmenting an object in the target frame so as to obtain annotation information corresponding to the object; determining a target object according to the annotation information, and clustering the target object so as to obtain a plurality of candidate regions to be embedded; and determining a target candidate region to be embedded from among the candidate regions to be embedded, and performing a maximum rectangular search on the target candidate region to be embedded so as to obtain a target region to be embedded.

Description

Detection method, device, electronic equipment and storage medium of information implantation area

Cross references to related applications

This application is based on a Chinese patent application with application number 201910578322.7 and an application date of June 28, 2019, and claims the priority of the Chinese patent application. The entire content of the Chinese patent application is hereby incorporated into this application by reference.

Technical field

This application relates to the field of computer technology, and in particular to a detection method for an information implantation area, a detection device for an information implantation area, electronic equipment and a storage medium.

Background technique

With the gradual maturity of electronic information, electronic media advertising has gradually become the main form of advertising. Taking video advertising as an example, video advertising can be divided into two forms: Video-In and Video-Out. Video-In is embedded advertising. , Is a form of soft advertising, where screens or physical advertisements are implanted on the desktop, wall, photo frame, and billboard in the video; Video-Out is a scene pop-up advertisement based on the car, face, Understand goals and scenarios, and display pop-up ads related to video content.

In related technologies, for video advertisements in the form of Video-In, it is usually necessary for professional designers to manually retrieve the video placement advertisement spots, which consumes a lot of manpower and time.

Summary of the invention

The embodiments of the present application provide a detection method for an information implantation area, a detection device for an information implantation area, electronic equipment, and a storage medium, which can improve the detection efficiency of video implantation advertising spaces.

The embodiment of the application provides a method for detecting an information implantation area, including:

Acquiring a video to be implanted, and segmenting the video to be implanted to obtain video segments;

Acquiring a target frame in the video segment, and identifying and segmenting an object in the target frame, so as to acquire label information corresponding to the object;

Determining a target object according to the annotation information, and clustering the target object to obtain multiple candidate regions to be implanted;

The target candidate to be implanted area is determined from the candidate to be implanted area, and the largest rectangle search is performed on the target candidate to be implanted area to obtain the target to be implanted area.

An embodiment of the application also provides a detection device for an information implantation area, including:

The shot segmentation module is configured to obtain a video to be implanted, and segment the video to be implanted to obtain video segments;

An object labeling module, configured to obtain a target frame in the video segment, and to identify and segment the object in the target frame to obtain label information corresponding to the object;

A clustering module, configured to determine a target object according to the annotation information, and cluster the target object to obtain multiple candidate regions to be implanted;

The area search module is configured to determine a target candidate to be implanted area from the candidate to be implanted area, and perform a largest rectangular search on the target candidate to be implanted area to obtain the target to be implanted area.

An embodiment of the present application also provides an electronic device, including: one or more processors; a storage device, configured to store one or more programs, when the one or more programs are used by the one or more processors When executed, the method for detecting the above-mentioned information implantation area provided in the embodiment of the present application is realized.

The embodiment of the present application also provides a computer storage medium, the computer storage medium stores a computer program, and the computer program is used to execute the information implantation area detection method provided by the embodiment of the present application.

The application of the method, device, electronic equipment, and storage medium for determining the image state based on pathological images provided by the embodiments of this application has at least the following beneficial technical effects:

First, the acquired video to be implanted is segmented to obtain video segments, and then the target frame is determined from each video segment, and objects in the target frame are identified and segmented to obtain all objects in the target frame Corresponding labeling information; then determine the target object according to the labeling information, cluster the target objects to obtain multiple candidate implanted regions; finally determine the target candidate implanted region from the candidate implanted regions, and compare the target candidates The area to be implanted is searched for the largest rectangle to obtain the target area to be implanted. In this way, it can automatically detect whether there is an information implantation area in the video, avoid manual screening and labeling, and reduce labor costs; at the same time, it can greatly reduce the detection time of the information implantation area, and improve the efficiency and accuracy of video advertising implantation .

Description of the drawings

The drawings here are incorporated into the specification and constitute a part of the specification, show embodiments that conform to the application, and are used together with the specification to explain the principle of the application. Obviously, the drawings in the following description are only some embodiments of the application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative work. In the attached picture:

FIG. 1 is a schematic diagram of a system architecture of a method for detecting an information implantation area provided by an embodiment of the application;

Figures 2A-2C are effect diagrams of desktop scene implantation in related technologies;

3 is a flowchart of a method for detecting an information implantation area provided by an embodiment of the application;

FIG. 4 is a schematic diagram of a process of segmenting a video to be implanted according to an embodiment of the application;

FIG. 5 is the object labeling information output by the instance segmentation model provided by the embodiment of this application;

FIG. 6 is a schematic flowchart of a mean shift processing for an object provided by an embodiment of the application;

FIG. 7 is a schematic diagram of the process of searching for the largest rectangle of the target candidate implanted area provided by an embodiment of the application;

FIG. 8A is a schematic diagram of a desktop structure before eliminating island noise areas according to an embodiment of the application;

FIG. 8B is a schematic diagram of the desktop structure after eliminating the island noise area according to an embodiment of the application;

FIG. 9 is a schematic diagram of the flow of advertisement placement provided by an embodiment of the application;

FIG. 10 is a block diagram of a detection device for an information implantation area provided by an embodiment of the application;

FIG. 11 is a schematic structural diagram of a computer system of an electronic device provided by an embodiment of the application.

Detailed ways

Example embodiments will now be described more fully with reference to the accompanying drawings. However, the example embodiments can be implemented in various forms, and should not be construed as being limited to the examples set forth herein; on the contrary, the provision of these embodiments makes this application more comprehensive and complete, and fully conveys the concept of the example embodiments To those skilled in the art.

Furthermore, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to give a sufficient understanding of the embodiments of the present application. However, those skilled in the art will realize that the technical solutions of the present application can be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. can be used. In other cases, well-known methods, devices, implementations or operations are not shown or described in detail to avoid obscuring various aspects of the present application.

The block diagrams shown in the drawings are merely functional entities, and do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in the form of software, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and/or processor devices and/or microcontroller devices. entity.

The flowchart shown in the drawings is only an exemplary description, and does not necessarily include all contents and operations/steps, nor does it have to be performed in the described order. For example, some operations/steps can be decomposed, and some operations/steps can be combined or partially combined, so the actual execution order may be changed according to actual conditions.

Before describing the embodiments of the present invention in further detail, the terms and terms involved in the embodiments of the present invention will be described. The terms and terms involved in the embodiments of the present invention are applicable to the following explanations.

Maximum rectangle search: By finding adjacent pixels in a specific area with the same pixel value, the rectangle area with the largest area can be determined from the specific area; for example, the maximum rectangle search can be achieved in the following way:

Use any pixel in the target area to be implanted as a reference point, and search for adjacent pixels with the same pixel value based on the pixel value of the reference point; if so, use the adjacent pixel as the reference point, repeat In the above steps, until all adjacent pixels with the same pixel value are obtained, any pixel is used as a vertex, a rectangle is formed according to the vertex and the adjacent pixel, the area of the rectangle is calculated, and the target rectangle with the largest area is selected, And take the area corresponding to the target rectangle as the target area to be implanted.

FIG. 1 shows a schematic diagram of an exemplary system architecture of a method for detecting an information implantation area provided by an embodiment of the present application.

As shown in FIG. 1, the system architecture 100 may include terminal devices (as shown in FIG. 1, one or more of the smart phone 101, the tablet computer 102, and the portable computer 103, of course, it may also be a desktop computer, etc.), a network 104 and server 105. The network 104 is used as a medium for providing a communication link between the terminal device and the server 105. The network 104 may include various connection types, such as wired communication links, wireless communication links, and so on.

It should be understood that the numbers of terminals, networks, and servers in FIG. 1 are merely illustrative. According to actual needs, there can be any number of terminals, networks and servers. For example, the server 105 may be a server cluster composed of multiple servers.

In some embodiments, the terminal device 101, which may also be the

terminal devices

102 and 103, sends a request for obtaining a video to be implanted to the server 105 via the network 104. The request for obtaining a video to be implanted includes the video number of the video to be implanted. There are multiple numbered videos stored in 105. By matching the obtained video number with all the video numbers, the video to be implanted corresponding to the video number in the video acquisition request to be implanted can be obtained. The server 105 can The matched video to be implanted is sent to the terminal device 101 (or terminal devices 102, 103) through the network 104. After receiving the video to be implanted, the terminal device performs shot segmentation on the video to be implanted to obtain video scores. Pieces; then obtain the target frame from each video segment, and identify and segment the objects in the target frame to obtain the label information corresponding to all the objects. The label information includes the classification information, confidence, and Mask and calibration frame, etc.; then the terminal device 101 determines the target object according to the labeling information, and clusters the target object to obtain multiple candidate regions to be implanted in the target object; finally, from the multiple candidate regions to be implanted Determine the target candidate to be implanted area, by performing the largest rectangular search on the target candidate to be implanted area, the target to be implanted area can be obtained, and the target to be implanted area is the information implantation area, which can be used for video advertising implantation. Into.

Applying the above-mentioned embodiments of this application, on the one hand, it can automatically detect whether there is an information implantation area in the video, avoid manual screening and labeling, and reduce labor costs; on the other hand, it can greatly reduce the detection time of the information implantation area, and improve the video Advertising implantation efficiency: by searching the largest rectangle in the target implantation area to obtain the target implantation area, the detection accuracy of the information implantation area can be improved.

It should be noted that, in some embodiments, the method for detecting the information implantation area provided in the embodiments of this application can be executed by the terminal device. Accordingly, the identification device of the information implantation area can be set in the terminal device; In some embodiments, the method for detecting the information implantation area provided in the embodiments of this application may also be implemented by the server.

In related technologies in this field, taking the video-in approach for advertising implantation as an example, in order to implant physical objects, 3D models, and print advertisements inside the video, implantation is generally performed on the desktop, wall, and photo frame. However, the retrieval of video placement advertisements in related technologies is fully implemented manually by professional designers and has not been automated. Figures 2A-2C show the effect of the desktop scene implantation. As shown in Figure 2A, it is the image in the original video. By detecting the advertising space in the desktop, the physical object is implanted into the advertising space, as shown in Figure 2B , You can also embed the 3D model and the poster into the advertising space at the same time, as shown in Figure 2C.

However, the manual retrieval of ad placement opportunities generally takes 1.5 times the length of the video. For video ad placement parties, this is very time-consuming and labor-intensive, which seriously affects ad placement. The efficiency of the advertising space, and there will be the defect of low accuracy of the advertising space, which further affects the effect of advertising placement.

In view of the problems in the related technology, the embodiment of this application first proposes a detection method of the information implantation area. The detection method of the information implantation area in the embodiment of the application can be used for video advertisement implantation, etc., as follows: The video advertisement implantation is taken as an example to elaborate the implementation details of the technical solution of the embodiment of the present application:

FIG. 3 schematically shows a flowchart of a method for detecting an information implantation area provided by an embodiment of the present application. The method for detecting an information implantation area may be executed by a terminal device, which may be the one shown in FIG. 1 Terminal Equipment. Referring to FIG. 3, the method for detecting the information implantation area at least includes steps S310 to S340, which are described in detail as follows:

In step S310, a video to be implanted is obtained, and the video to be implanted is segmented to obtain video segments.

In some embodiments, video advertisement implantation is a new technology system that uses computer vision technology to intelligently implant advertisements in videos that have already been produced (that is, videos to be implanted). Users can search for videos to be implanted online. Into the video, the video to be implanted can also be obtained from the video folder or video database of the terminal device 101. The video to be implanted can be a video file of any format, such as avi., mp4., rmvb., etc., this application The embodiment does not limit this.

In some embodiments, a request for acquiring a video to be implanted may be sent to the server 105 through the terminal device 101, and the request for acquiring a video to be implanted includes a video number of the video to be implanted. The video number has any number format, such as a number. The format number, English letters + numbers are numbered, etc.; after the server 105 parses the request for acquiring the implanted video to obtain the video number, it can match the video number with the video numbers of all videos stored in the server to Obtain the video to be implanted corresponding to the video number in the video to be implanted acquisition request; finally, the server 105 can return the matched video to be implanted to the terminal device 101, so that the terminal device 101 can make the advertising space in the video to be implanted Perform a search.

In some embodiments, after receiving the video to be implanted, the video to be implanted can be segmented to obtain video segments constituting the video to be implanted. The basic structure of a video is a hierarchical structure composed of frames, shots, scenes, and video programs. The frame is a static image, which is the smallest logical unit that composes the video. The sequence of consecutive frames in time is played continuously at equal intervals to form Dynamic video; a lens is a sequence of frames continuously shot by a camera from power on to power off, depicting an event or a part of a scene, without or with weak semantic information, emphasizing the similarity of the visual content of the frame; the scene is semantically related The continuous shots of the same subject can be shot with different angles and different techniques, or a combination of shots with the same theme and event, emphasizing semantic relevance; the video program contains a complete event or story as the highest level of video content Structure, which includes the composition relationship of the video and the summary, semantics and general description of the video. In order to effectively identify the information implantation area, the lens can be used as the processing unit, that is, each lens is regarded as a video segment. In practical applications, the to-be-implanted video can be segmented into multiple pieces. Shots, and then obtain the video segments that constitute the video to be implanted.

In some embodiments, the similarity algorithm can be used to identify the segmentation of the video to be implanted. FIG. 4 shows a schematic flow chart of segmenting the video to be implanted. As shown in FIG. 4, in step S401, The target feature of the image frame is extracted from the video; in step S402, similarity algorithm recognition is performed on the target feature in adjacent image frames, and the video to be implanted is segmented according to the recognition result to obtain video segments.

Here, during similarity algorithm identification, each pixel in two adjacent image frames can be compared for similarity. However, due to the huge number of pixels in the image, if the similarity of the pixels is compared one by one, it takes a lot of resources. The processing efficiency is low, so the target feature can be extracted from the video to be implanted. The target feature can be a multi-bit feature in the image frame contained in the video to be implanted. The similarity algorithm is performed on the target feature in the adjacent image frame Recognition to determine the boundary image frame of adjacent shots.

In some embodiments, performing similarity algorithm recognition on target features in adjacent image frames in step S402, and segmenting the to-be-implanted video according to the recognition result to obtain video segments may include:

Calculate the distance between the target features in adjacent image frames, and perform similarity algorithm recognition based on the distance; the distance can be Euclidean distance, cosine distance, etc., take Euclidean distance as an example to obtain target features in adjacent image frames After the Euclidean distance between them, the Euclidean distance can be compared with a preset distance threshold to determine the similarity between adjacent image frames. When the distance is less than the preset distance threshold, it is determined that the adjacent image frames belong to the same video segment; when the distance is greater than or equal to the preset distance threshold, it is determined that the adjacent image frames belong to different video segments.

In step S320, a target frame in the video segment is acquired, and an object in the target frame is identified and segmented to obtain label information corresponding to the object.

In some embodiments, when information is implanted, it can be implanted in typical and representative frames in the shot. For example, implanting information in key frames or representative frames can improve the efficiency of implantation. After the video to be implanted is divided, one or more target frames (which can be key frames of the video) can be determined from each video segment, and the objects in the target frame can be processed to obtain the target frame that can be used for advertising The implanted target area to be implanted.

In some embodiments, an instance segmentation model may be used to identify and segment the objects in the target frame, so as to classify the objects in the target frame and obtain label information corresponding to each object. The instance segmentation model can preprocess the input target frame, and perform convolution operation on the preprocessed target frame to extract features, and then obtain the feature image; then the feature image can be processed through the candidate region generation network, Acquire multiple candidate regions of interest, and classify and regress the multiple candidate regions of interest to obtain the target region of interest; then, the pixels of the target region of interest can be aligned with the pixels of the region in the target frame; Finally, operations such as classification, border regression, and mask generation can be performed on the target region of interest to obtain the label information corresponding to each object in the target frame.

In some embodiments, the instance segmentation model may be a Mask R-CNN model, of course, it may also be other machine learning models that can identify, segment and identify objects in the target frame. After the target frame is processed through the instance segmentation model, the label information corresponding to each object in the target frame can be obtained. The label information can include the classification information, confidence, mask, and calibration frame of the object. Figure 5 shows the instance segmentation. The object annotation information output by the model is shown in Figure 5. There are desktops and cups placed on the desktop in the input target frame. The desktop and the cup can be segmented through the instance segmentation model recognition segmentation, and use different color masks and The label box is marked, the dark area is the mask corresponding to the desktop, the dashed box A is the label box corresponding to the desktop, and the corresponding classification information is table with a confidence of 0.990; the light area is the mask corresponding to the cup, the dotted line Box B is the label box corresponding to the cup, the corresponding classification information is cup, and the confidence level is 0.933.

Before processing the target frame through the instance segmentation model, a large number of image frames can be collected as training samples to train the instance segmentation model. For example, one or more image frames in the video can be used as training samples, and the training samples can be input To the instance segmentation model to be trained, by comparing the label information output by the model with the label information corresponding to the training sample, it is judged whether the model has completed the training.

In step S330, a target object is determined according to the annotation information, and the target object is clustered to obtain multiple candidate regions to be implanted.

In some embodiments, after obtaining the label information of the object in the target frame, the target object can be obtained according to the label information. For example, if you want to implement advertisement implantation on the desktop, you can obtain the classification according to the classification information of the object in the label information. It is a table (desktop) object. For example, there are one or more desktops in the target frame, then the one or more desktops can be determined as the target object, and the target area to be implanted can be determined from the target object. Or select one of multiple desktops as the target object, and determine the target area to be implanted from the target object.

In actual implementation, in order to improve the effect of ad placement and save advertisers’ costs, ad placement can only be carried out in a more conspicuous location, so that a target object can be determined from multiple objects to be implanted, and The target object is clustered to obtain multiple candidate regions to be implanted.

In some embodiments, the target object for implantation can be determined from a plurality of objects to be implanted according to the area of the mask in the annotation information. In some embodiments, the mask with the largest area is selected. The object to be implanted is the target object, for example, the desktop with the largest mask area is selected. This desktop is the desktop occupying the main position in the target frame. If the advertisement is implanted on the desktop, the user will be able to pay attention when watching the video To this advertisement, the user reach rate of the advertisement is increased.

In some embodiments, before obtaining the target object according to the annotation information, the objects in the target frame can be filtered. In practical applications, the confidence level corresponding to each object can be compared with a preset confidence threshold. When the confidence is greater than the preset confidence threshold, the object is retained, and the object whose confidence is less than or equal to the preset confidence threshold is deleted. The preset confidence threshold can be set according to actual needs, for example, it can be set to 0.5, so that only all objects with confidence greater than 0.5 in the target frame can be retained.

In actual implementation, you can also filter the target frame, because the video to be implanted may contain multiple video segments, and each video segment may contain one or more target frames, but not every target frame There are target objects, so the target frame that does not contain the target object can be deleted. For example, when it is determined that the target object is the desktop, the target frame that does not contain the desktop can be deleted, and only the advertising space in the target frame that contains the desktop can be detected.

In some embodiments, after obtaining the target object, the pixels in the target object can be clustered to obtain multiple candidate regions to be implanted. In practical applications, the target object can be processed by mean shifting Clustering objects, FIG. 6 shows a schematic flow chart of the mean shift processing of the objects. As shown in FIG. 6, in step S601, any pixel in the target object is used as the target point, and the The target point is the center of the circle, and the target range is determined according to the preset radius; in step S602, the mean offset vector is determined according to the distance vector between the target point and any pixel in the target range, and the mean offset vector is determined according to the mean value. The offset vector moves the target point to the end point of the mean offset vector; in step S603, the end point is used as the target point, and steps S601-S602 are repeated until the position of the target point no longer changes In step S604, determine the pixel set according to the pixel points corresponding to the final target point and the pixel points within the preset radius; in step S605, obtain the distance between the pixel sets, and compare the distance with the preset A distance threshold is set for comparison, so as to determine the candidate region to be implanted according to the comparison result.

When determining the candidate region to be implanted according to the comparison result between the distance between the pixel sets and the preset distance threshold, the following method can be performed:

When the distance between the pixel sets is less than or equal to the preset distance threshold, merge the two pixel sets corresponding to the distance to form a candidate region to be implanted;

When the distance between the pixel sets is greater than the preset distance threshold, the two pixel sets corresponding to the distance are respectively used as candidate regions to be implanted.

By performing the mean shift processing on the target object, multiple connected regions in the target object can be obtained. These connected regions are candidate regions to be implanted. For example, the target object is a desktop, and the mean shift is performed for each pixel in the desktop. Through processing, multiple connected areas can be obtained, and these connected areas form a desktop. In actual implementation, all these connected areas can be used as candidate areas to be implanted in the desktop that can be used for advertisement implantation.

In step S340, a target candidate to be implanted area is determined from the candidate to be implanted area, and a largest rectangle search is performed on the target candidate to be implanted area to obtain the target to be implanted area.

In some embodiments, after a plurality of candidate regions to be implanted are determined, they can be screened to determine the target candidate regions to be implanted; for example, those that do not contain non-target objects can be selected from the plurality of candidate regions to be implanted Candidate areas to be implanted, such as mobile phones, tea cups, vases and other objects on the desktop. In order to ensure the effect of advertising placement, the candidate areas to be implanted that contain mobile phones, tea cups, vases and other objects can be discarded, and only the non-targets are kept The candidate implantation area of the object; then the area of the candidate implantation area that does not include the non-target object can be calculated, and the candidate implantation area with the largest area is selected as the target candidate implantation area. In this way, the core airspace of the target object with the optimal area and blank area can be obtained.

In some embodiments, because the shape of the target candidate to be implanted area may be irregular, if the advertisement is directly implanted in the target candidate to be implanted area, there may be missing information. Therefore, when the target is determined After the candidate area to be implanted, that is, the core airspace of the target object, the largest rectangular search for the target candidate area to be implanted is needed to determine the target area to be implanted for advertisement implantation.

FIG. 7 shows a schematic diagram of the process of searching for the largest rectangle of the target candidate to be implanted area. As shown in FIG. 7, the process of searching for the largest rectangle of the target candidate to be implanted includes:

In step S701, any pixel point in the target area to be implanted is used as a reference point, and the pixel value of the reference point is used to find whether there are adjacent pixels having the same pixel value.

For example, the pixel value of the pixel in the target area to be implanted is 0 or 1. If the pixel value of a certain pixel is 0, the pixel is used as the reference point to find the adjacent pixel with the pixel value of 0 If the pixel value of a certain pixel is 1, the pixel is used as the reference point to find a pixel with a pixel value of 1 adjacent to the pixel. If there is an adjacent pixel with a pixel value of 0.

In step S702, when there are adjacent pixels with the same pixel value, using the adjacent pixels as the reference point, step S701 is repeated until all adjacent pixels with the same pixel value are obtained.

For example, if there is a pixel point adjacent to the reference point that has the same pixel value as the reference point, the adjacent pixel point is used as the reference point and continues to expand outward to determine whether there is an adjacent pixel point that has the same pixel value. After multiple judgments, all adjacent pixels with the same pixel value can be obtained.

In step S703, use any one of the pixels as a vertex, and form a rectangle according to the vertex and the adjacent pixel.

Taking the pixel as the reference point for the first time as the vertex, a rectangle is formed according to the vertex and the neighboring pixels. It is worth noting that the pixels in the rectangle all have the same pixel value and do not include pixels with different pixel values. And the vertex can be a left vertex, a right vertex, etc., which is not specifically limited in the embodiment of the present application.

In step S704, the area of the rectangle is calculated, the target rectangle with the largest area is selected, and the area corresponding to the target rectangle is used as the target area to be implanted.

Multiple rectangles can be obtained through step S703. In order to obtain the optimal information implantation area, it can be determined according to the area of the rectangle. In practical applications, the area of each rectangle can be calculated, and the target rectangle with the largest area can be selected from it, and The area corresponding to the target rectangle is used as the target area to be implanted for advertisement implantation.

In some embodiments, there may be island noise regions in the target area to be implanted. These island noise regions are mainly formed by the light and shadow of the target object. In order to avoid the influence of these island noise regions on the maximum rectangle search result, Before the largest rectangle search is performed on the target region to be implanted, the mean filter processing can be performed on the target region to be implanted to obtain a uniform and smooth target region to be implanted.

Figure 8A shows a schematic diagram of the desktop structure before eliminating the island noise area, and Figure 8B shows a schematic diagram of the desktop structure after eliminating the island noise area. As shown in Figure 8A, there are some island noise areas in the core airspace of the desktop, as shown in the figure. After the average filtering process, these small black areas are eliminated, and the core airspace of the desktop becomes uniform and smooth, as shown in Figure 8B.

The method for detecting the information implantation area in the embodiment of the present application can be used to detect the area in the desktop that can be used for advertisement implantation, and can also be used to detect the advertisement implantation area in other desktop-like objects, for example, It can detect the advertising placement area for objects such as the cash register in the supermarket, the bench in the park, and the running belt of the treadmill.

Taking the placement of advertisements for a certain brand of drinks at the cashier of a supermarket as an example, Figure 9 shows a schematic diagram of the flow of advertisement placement, as shown in Figure 9:

In step S901, the video to be implanted is segmented to obtain video segments, and a target frame is obtained from the video segments.

Here, the advertisement placement scene of a certain brand of beverages corresponding to the supermarket checkout counter, the target frame can be the scene of supermarket checkout.

In step S902, the target frame is input to the instance segmentation model to obtain label information of all objects in the target frame.

In step S903, the target object is determined according to the classification information of the object and the area of the corresponding mask.

For example, in the target frame, the cashier counter corresponds to the largest mask area, so the cashier counter can be used as the target object.

In step S904, mean shift processing is performed on the target object to obtain multiple candidate regions to be implanted.

In step S905, a candidate region to be implanted that does not contain a non-target object with the largest area is obtained, and the candidate region to be implanted is used as the target candidate region to be implanted.

Here, there may be objects such as displays and advertising display boards on the cashier counter. In order to determine the core airspace in the cashier counter, multiple candidate areas to be implanted can be screened to obtain target candidate areas to be implanted.

In step S906, mean filtering is performed on the target candidate to be implanted region to obtain a uniform and smooth target candidate to be implanted region.

In step S907, a largest rectangle search is performed in the target candidate region to be implanted to obtain the target region to be implanted.

Here, the target area to be implanted is the airspace where advertisements can be implanted on the cashier counter. The target area to be implanted can be the airspace near the display on the cashier counter, or the corner of the cashier counter near the entrance. The embodiment does not specifically limit this.

In step S908, the advertisement is implanted into the target area to be implanted.

When an advertisement for a certain brand of drink is implanted, the advertisement can be a drink entity, or a 3D model containing a drink promotion poster, and so on.

The above embodiments describe the detection method of the information implantation area performed by the terminal device. Similarly, the detection method of the information implantation area may also be executed by a server, which may be a server dedicated to data processing. Accordingly, An instance segmentation model is set in the server. After receiving the video to be implanted from the server 105, the terminal device 101 can send the video to be implanted to the server for data processing. After the server receives the video to be implanted, the The video to be implanted is segmented to obtain video segments; then the target frame in the video segment is identified and segmented to obtain the label information of the object in the target frame; then the target object is determined according to the label information, and the target object Perform clustering to obtain multiple candidate implanted regions; finally, determine the target candidate implanted region from the candidate implanted regions, and perform the largest rectangular search for the target implanted region to obtain the target implanted region , And then send the target area to be implanted to the terminal device 101 to implement video advertisement implantation.

The technical solution of the embodiment of the present application uses the instance segmentation model to identify and segment the object in the target frame, and combines the color block clustering in the mask and the largest rectangle search to determine the target area to be implanted from the target object, thereby realizing automatic For the detection of embedded advertising space, the detection time is 1.5 times longer than the original manual video time. The detection method of the embodiment of the present application can compress the time to 0.2 times the video time length, which reduces labor costs on the one hand and increases In addition, in practical applications, the detection accuracy of the artificial intelligence-based information implantation area detection method in the embodiment of the application can reach 0.91, which greatly improves the accuracy of advertising space detection and avoids different manual screening The situation where the identified ad slots are different.

The following describes the device embodiments of the present application, which can be configured to execute the information implantation area detection method in the foregoing embodiments of the present application. For details that are not disclosed in the device embodiments of this application, please refer to the above-mentioned embodiment of the detection method of the information implantation area in this application.

Fig. 10 schematically shows a block diagram of a device for detecting an information implantation area provided by an embodiment of the present application.

Referring to FIG. 10, the information implantation area detection device 1000 provided by the embodiment of the present application includes: a shot segmentation module 1001, an object labeling module 1002, a clustering module 1003, and an area search module 1004.

Wherein, the shot segmentation module 1001 is configured to obtain a video to be implanted, and segment the video to be implanted to obtain video segments;

The object labeling module 1002 is configured to obtain a target frame in the video segment, and to identify and segment the object in the target frame to obtain label information corresponding to the object;

The clustering module 1003 is configured to determine a target object according to the labeling information, and cluster pixels in the target object to obtain multiple candidate regions to be implanted;

The region search module 1004 is configured to determine a target candidate region to be implanted from the candidate region to be implanted, and perform a largest rectangular search on the target region to be implanted to obtain the target region to be implanted.

In some embodiments, the detection device 1000 further includes:

A number sending module, configured to send a request for acquiring a video to be implanted to the server, where the request for acquiring a video to be implanted includes a video number with the implanted video;

The video receiving module is configured to receive the to-be-implanted video corresponding to the video number returned by the server in response to the to-be-implanted video acquisition request.

In some embodiments, the lens segmentation module 1001 includes:

A feature extraction unit configured to extract target features from the video to be implanted;

The similarity recognition unit is configured to perform similarity algorithm recognition on adjacent image frames, and segment the to-be-implanted video according to the recognition result to obtain the video segment.

In some embodiments, the object labeling module 1002 includes:

The model processing unit is configured to input the target frame into an instance segmentation model, and identify and segment the object in the target frame through the instance segmentation model to obtain the label information.

In some embodiments, the model processing unit is configured to: preprocess the target frame through the instance segmentation model, and perform feature extraction on the preprocessed target frame to obtain a feature image; A plurality of candidate regions of interest are determined on the feature image, and the plurality of candidate regions of interest are classified and regressed to obtain a target region of interest; an alignment operation is performed on the target region of interest to align the target The pixels in the frame are aligned with the pixels of the target region of interest; classification, frame regression, and mask generation are performed on the target region of interest to obtain the label information of the object.

In some embodiments, the annotation information includes classification information, confidence, mask, and calibration frame of the object.

In some embodiments, the clustering module 1003 includes:

The target object determining unit is configured to determine the target object according to the classification information in the annotation information and the area of the mask;

The clustering unit is configured to perform mean shift processing on the target object, so as to cluster pixels in the target object, and obtain the multiple candidate regions to be implanted.

In some embodiments, there are multiple video segments, and each video segment includes one or more target frames;

The clustering module 1003 may also be configured to: compare the confidence levels of the objects contained in each target frame with a preset confidence threshold, and reserve the confidence levels in each target frame to be greater than the preset confidence. Then, delete the target frame that does not contain the object to be implanted, and the classification information of the object to be implanted is the same as that of the target object.

In some embodiments, the clustering unit includes:

A range determining unit, configured to use any pixel in the target object as a target point, with the target point as the center of the circle, and determine the target range according to a preset radius;

The moving unit is configured to determine a mean offset vector according to the distance vector between the target point and any pixel in the target range, and move the target point to the mean offset according to the mean offset vector The end of the shift vector;

A repeating unit configured to use the end point as the target point, and repeat the above steps until the position of the target point no longer changes;

A pixel set determining unit, configured to determine a pixel set according to a pixel point corresponding to the final target point and a pixel point within the preset radius;

The comparison unit is configured to obtain the distance between the pixel sets and compare the distance with a preset distance threshold to determine the candidate region to be implanted according to the comparison result.

In some embodiments, the comparison unit is configured to: when the distance is less than or equal to the preset distance threshold, combine two sets of pixels corresponding to the distance to form the candidate region to be implanted When the distance is greater than the preset distance threshold, two pixel sets corresponding to the distance are respectively used as the candidate regions to be implanted.

In some embodiments, the region search module 1004 is configured to: obtain candidate regions for implantation that do not contain non-target objects in the candidate region to be implanted; calculate the candidate regions for implantation that do not contain non-target objects The area of the candidate to be implanted with the largest area is used as the target candidate to be implanted area.

In some embodiments, the detection device further includes: a filtering module configured to perform average filtering on the target region to be implanted to obtain a uniform and smooth target region to be implanted.

In some embodiments, the area search module 1004 is configured to: use any pixel in the target area to be implanted as a reference point, and search for whether there are phases with the same pixel value according to the pixel value of the reference point. Adjacent pixel; if it exists, use the adjacent pixel as the reference point and repeat the above steps until all adjacent pixels with the same pixel value are obtained; use any one of the pixels as the vertex, according to the The vertices and the adjacent pixel points form a rectangle; the area of the rectangle is calculated, a target rectangle with the largest area is selected, and the area corresponding to the target rectangle is used as the target area to be implanted.

FIG. 11 is a schematic structural diagram of a computer system of an electronic device provided in an embodiment of the application, and the electronic device is used to implement the method for detecting an information implantation area provided in an embodiment of the application.

It should be noted that the computer system 1100 of the electronic device shown in FIG. 11 is only an example, and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

As shown in FIG. 11, the computer system 1100 includes a central processing unit (CPU) 1101, which can be loaded into a random storage unit according to a program stored in a read-only memory (Read-Only Memory, ROM) 1102 or from a storage part 1108. Access to the program in the memory (Random Access Memory, RAM) 1103 to execute various appropriate actions and processing to implement the detection method of the information implantation area described in the foregoing embodiment. In RAM1103, various programs and data required for system operation are also stored. The CPU 1101, the ROM 1102, and the RAM 1103 are connected to each other through a bus 1104. An input/output (Input/Output, I/O) interface 1105 is also connected to the bus 1104.

The following components are connected to the I/O interface 1105: the input part 1106 including keyboard, mouse, etc.; including the output part 1107 such as cathode ray tube (Cathode Ray Tube, CRT), liquid crystal display (LCD), and speakers, etc. ; A storage part 1108 including a hard disk, etc.; and a communication part 1109 including a network interface card such as a local area network (LAN) card, a modem, and the like. The communication section 1109 performs communication processing via a network such as the Internet. The driver 1110 is also connected to the I/O interface 1105 as needed. A removable medium 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 1110 as needed, so that the computer program read from it is installed into the storage portion 1108 as needed.

According to the embodiments of the present application, the process described below with reference to the flowchart can be implemented as a computer software program. For example, the embodiments of the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program may be downloaded and installed from the network through the communication part 1109, and/or installed from the removable medium 1111. When the computer program is executed by the central processing unit (CPU) 1101, various functions defined in the system of the present application are executed.

It should be noted that the computer-readable medium shown in the embodiments of the present application may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. The computer-readable storage medium may be, for example, but not limited to, an electric, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections with one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable Erasable Programmable Read Only Memory (EPROM), flash memory, optical fiber, portable compact disk read-only memory (Compact Disc Read-Only Memory, CD-ROM), optical storage device, magnetic storage device, or any suitable of the above The combination. In the embodiments of the present application, the computer-readable storage medium may be any tangible medium that contains or stores a program, and the program may be used by or in combination with an instruction execution system, apparatus, or device. In the embodiment of the present application, a computer-readable signal medium may include a data signal propagated in a baseband or as a part of a carrier wave, and a computer-readable program code is carried therein. This propagated data signal can take many forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the foregoing. The computer-readable signal medium may also be any computer-readable medium other than the computer-readable storage medium. The computer-readable medium may send, propagate, or transmit the program for use by or in combination with the instruction execution system, apparatus, or device . The program code contained on the computer-readable medium can be transmitted by any suitable medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation of the system architecture, functions, and operations of the system, method, and computer program product according to various embodiments of the present application. In this regard, each block in the flowchart or block diagram may represent a module, program segment, or part of code, and the above-mentioned module, program segment, or part of code contains one or more for realizing the specified logical function Executable instructions. It should also be noted that, in some alternative implementations, the functions marked in the block may also occur in a different order from the order marked in the drawings. For example, two blocks shown in succession can actually be executed substantially in parallel, or they can sometimes be executed in the reverse order, depending on the functions involved. It should also be noted that each block in the block diagram or flowchart, and the combination of blocks in the block diagram or flowchart, can be implemented by a dedicated hardware-based system that performs the specified functions or operations, or can be It is realized by a combination of dedicated hardware and computer instructions.

The units involved in the embodiments described in the present application can be implemented in software or hardware, and the described units can also be provided in a processor. Among them, the names of these units do not constitute a limitation on the unit itself under certain circumstances.

The embodiments of the present application also provide a computer-readable medium. The computer-readable medium may be included in the electronic device described in the foregoing embodiment; or it may exist alone without being assembled into the electronic device. The above-mentioned computer-readable medium carries one or more programs, and when the above-mentioned one or more programs are executed by an electronic device, the electronic device realizes the method for detecting the information implantation area provided in the embodiments of the present application.

It should be noted that although several modules or units of the device used for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above may be embodied in one module or unit. Conversely, the features and functions of a module or unit described above can be further divided into multiple modules or units to be embodied.

Through the description of the foregoing embodiments, those skilled in the art can easily understand that the exemplary embodiments described herein can be implemented by software, or can be implemented by combining software with necessary hardware. Therefore, the technical solution according to the embodiments of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (can be a CD-ROM, U disk, mobile hard disk, etc.) or on the network , Including several instructions to make a computing device (which can be a personal computer, a server, a touch terminal, or a network device, etc.) execute the method according to the embodiment of the present application.

After considering the specification and practicing the invention disclosed herein, those skilled in the art will easily think of other embodiments of the present application. This application is intended to cover any variations, uses, or adaptive changes of this application. These variations, uses, or adaptive changes follow the general principles of this application and include common knowledge or customary technical means in the technical field not disclosed in this application. .

It should be understood that the present application is not limited to the precise structure that has been described above and shown in the drawings, and various modifications and changes can be made without departing from its scope. The scope of the application is only limited by the appended claims.

Industrial applicability

In this embodiment of the application, a video to be implanted is acquired, and the video to be implanted is segmented to obtain video segments; the target frame in the video segment is acquired, and the object in the target frame is identified and Segmentation to obtain label information corresponding to the object; determine a target object according to the label information, and cluster the target object to obtain multiple candidate regions to be implanted; from the candidate region to be implanted Determine the target candidate to be implanted area in, and perform a largest rectangle search on the target candidate to be implanted area to obtain the target to be implanted area. In this way, it can automatically detect whether there is an information implantation area in the video, avoid manual screening and labeling, and reduce labor costs; at the same time, it can greatly reduce the detection time of the information implantation area, and improve the efficiency and accuracy of video advertising implantation .

Claims

A method for detecting an information implantation area, the method is executed by a terminal device, and the method includes:

Obtaining a video to be implanted, segmenting the video to be implanted, to obtain video fragments;

Acquiring a target frame in the video segment, and identifying and segmenting an object in the target frame, so as to acquire label information corresponding to the object;

Determining a target object according to the annotation information, and clustering the target object to obtain multiple candidate regions to be implanted;

The target candidate to be implanted area is determined from the candidate to be implanted area, and the largest rectangle search is performed on the target candidate to be implanted area to obtain the target to be implanted area.
The method for detecting an information implantation area according to claim 1, wherein said obtaining a video to be implanted comprises:

Sending a request for obtaining a video to be implanted to the server, where the request for obtaining a video to be implanted includes the video number of the video to be implanted;

Receiving the video to be implanted corresponding to the video number returned by the server.
The method for detecting information implantation area according to claim 1, wherein said segmenting said video to be implanted to obtain video fragments comprises:

Extracting target features of image frames from the video to be implanted;

Perform similarity recognition on the target features in adjacent image frames, and segment the to-be-implanted video according to the recognition result to obtain the video segment.
The method for detecting an information implantation area according to claim 1, wherein the identifying and segmenting the object in the target frame to obtain the label information corresponding to the object comprises:

The target frame is input into an instance segmentation model, and the object in the target frame is identified and segmented through the instance segmentation model to obtain the label information corresponding to the object.
4. The method for detecting an information implantation area according to claim 4, wherein said identifying and segmenting objects in said target frame through said instance segmentation model comprises:

Preprocessing the target frame by using the instance segmentation model, and performing feature extraction on the preprocessed target frame to obtain a feature image;

Determining a plurality of candidate regions of interest on the characteristic image, and classifying and regressing the plurality of candidate regions of interest to obtain a target region of interest;

Performing an alignment operation on the target region of interest to align the pixels in the target frame with the pixels of the target region of interest;

Perform classification, frame regression, and mask generation on the target region of interest to obtain the label information of the object corresponding to the object.
The method for detecting an information implantation area according to any one of claims 1 to 5, wherein the labeling information includes classification information, confidence, mask, and calibration frame of the object.
The method for detecting an information implantation area according to claim 6, wherein the determining a target object according to the annotation information and clustering the target object to obtain a plurality of candidate areas to be implanted comprises:

Determining the target object according to the classification information in the annotation information and the area of the mask;

Perform mean shift processing on the target object to cluster the target object and obtain the multiple candidate regions to be implanted.
The method for detecting an information implantation area according to claim 7, wherein the number of said video segments is multiple, and each said video segment contains one or more target frames;

Before determining the target object according to the classification information in the annotation information and the area of the mask, the method further includes:

Comparing the confidence levels of the objects contained in each of the target frames with a preset confidence threshold, and keep the objects in each of the target frames whose confidence is greater than the preset confidence threshold;

Delete the target frame that does not contain the object to be implanted, and the classification information of the object to be implanted is the same as that of the target object.
7. The method for detecting an information implantation area according to claim 7, wherein the mean shift processing is performed on the target object to cluster the target object and obtain the multiple candidate regions to be implanted, include:

Taking any pixel in the target object as a target point, taking the target point as a center of a circle, and determining a target range according to a preset radius;

Determine the mean offset vector according to the distance vector between the target point and any pixel in the target range, and move the target point to the distance of the mean offset vector according to the mean offset vector An end point, and use the end point as the target point;

Repeat the above steps until the position of the target point no longer changes;

Determine a pixel set according to the pixel points corresponding to the target point whose position does not change and the pixel points within the preset radius;

The distance between the pixel sets is acquired, and the distance is compared with a preset distance threshold to determine the candidate region to be implanted according to the comparison result.
The method for detecting an information implantation area according to claim 9, wherein the comparing the distance with a preset distance threshold to determine the candidate area to be implanted according to the comparison result comprises:

When the distance is less than or equal to the preset distance threshold, combining two pixel sets corresponding to the distance to form the candidate region to be implanted;

When the distance is greater than the preset distance threshold, two pixel sets corresponding to the distance are respectively used as the candidate regions to be implanted.
The method for detecting an information implantation area according to claim 1, wherein determining a target candidate implantation area from the candidate implantation area includes:

Acquiring candidate regions for implantation that do not contain non-target objects in the candidate regions for implantation;

The area of the candidate region to be implanted that does not contain a non-target object is calculated, and the candidate region to be implanted with the largest area is used as the target candidate region to be implanted.
The method for detecting an information implantation area according to claim 1, wherein before the largest rectangle search is performed on the target area to be implanted, the method further comprises:

Mean filtering is performed on the target region to be implanted to obtain a uniform and smooth target region to be implanted.
The method for detecting an information implantation area according to claim 12, wherein, performing a largest rectangle search on the target area to be implanted to obtain the target area to be implanted comprises:

Taking any pixel in the target area to be implanted as a reference point, and searching for whether there are adjacent pixels with the same pixel value according to the pixel value of the reference point;

If it exists, use the adjacent pixel as the reference point and repeat the above steps until all adjacent pixels with the same pixel value are obtained;

Taking the any pixel as a vertex, and forming a rectangle according to the vertex and the adjacent pixel;

Calculate the area of the rectangle, select the target rectangle with the largest area, and use the area corresponding to the target rectangle as the target area to be implanted.
A detection device for information implantation area, said device comprising:

The shot segmentation module is configured to obtain a video to be implanted, and segment the video to be implanted to obtain video fragments;

An object labeling module, configured to obtain a target frame in the video segment, and to identify and segment the object in the target frame to obtain label information corresponding to the object;

A clustering module, configured to determine a target object according to the annotation information, and cluster the target object to obtain multiple candidate regions to be implanted;

The area search module is configured to determine the target candidate implantation area from the candidate implantation area, and perform a largest rectangle search on the target implantation area to obtain the target implantation area.
An electronic device including:

One or more processors;

A storage device configured to store one or more programs, and when the one or more programs are executed by the one or more processors, the information implantation area according to any one of claims 1 to 13 is implemented The detection method.
A computer storage medium storing a computer program, and the computer program is used to execute the method for detecting an information implantation area according to any one of claims 1 to 13.