KR102670709B1 - Virtual advertisement insertion system using area and object recognition of broadcast image and method thereof - Google Patents

Abstract

The present invention relates to a virtual advertisement insertion system and method for inserting virtual advertisements into broadcast images. More specifically, the present invention relates to a virtual advertisement insertion system and method for inserting virtual advertisements into broadcast images provided online and providing them to viewers, recognizing objects in the broadcast images, and , the virtual advertisement is overlaid and inserted into a similar area with a size corresponding to the size of the overlay area of the virtual advertisement to be inserted among the object and background, and the movement of the object is tracked to insert the virtual advertisement within the similar area according to the movement of the object. It relates to a system and method for inserting virtual advertisements using area and object recognition in broadcasting video, which inserts virtual advertisements by moving them.

Description

Virtual advertisement insertion system using area and object recognition of broadcast image and method thereof}

The present invention relates to a virtual advertisement insertion system and method for inserting virtual advertisements into broadcast images. More specifically, the present invention relates to a virtual advertisement insertion system and method for inserting virtual advertisements into broadcast images and providing them to viewers, recognizing objects in the broadcast images, and among the objects and background. The virtual advertisement is overlaid and inserted into a similar area with a size corresponding to the size of the overlay area of the virtual advertisement to be inserted, the movement of the object is tracked, and the insertion position of the virtual advertisement is moved within the similar area according to the movement of the object to insert the virtual advertisement. It relates to a system and method for inserting virtual advertisements using area and object recognition of broadcast video.

In general, advertising is divided into print media and radio media advertising, and is becoming more diverse, such as installation media advertising, transportation advertising, and Internet advertising.

In particular, television advertisements are video-based advertisements that are so effective that advertisers are willing to spend a lot of money on them.

Television advertisements cost a lot of money to produce advertisement videos, and then provide them in the middle or between broadcast programs of each broadcasting station through over-the-air broadcasting. Therefore, there are many restrictions on advertisement time, and advertisement production costs and There is a problem that advertising costs are high.

In order to minimize these advertising time constraints and increase television advertising broadcasting time, a television virtual advertising method has been developed that inserts virtual advertising (hereinafter referred to as “virtual advertising”) into some areas of the broadcast video during the broadcast of a random broadcast program. It is provided along with standard television advertisements.

The conventional virtual advertisement insertion system inserts a virtual advertisement into a portion of the video during the airing of a broadcast program, and inserts the virtual advertisement by designating a position to insert the advertisement in a pre-scheduled section from a broadcast manager who monitors it in real time, or in the case of a recorded broadcast program. The broadcast program video is inserted into the relevant location in advance before being transmitted.

As described above, the conventional virtual advertisement insertion system has a cumbersome and inconvenient problem because, in the case of real-time broadcasting programs such as live broadcasts, the broadcasting manager must continuously monitor the broadcast program video.

In addition, conventional virtual advertisements are overlaid and displayed at the inserted position for a preset advertising time, so there is a problem that the objects and background of the video can be obscured, reducing the viewer's enjoyment of the broadcast program or causing inconvenience. .

In addition, as the Internet, wireless communication, and mobile terminal technology develop rapidly, a ubiquitous environment in which communication can be performed by accessing information and communication networks such as the Internet at any time is established, the demand for Internet advertising is increasing, but conventional virtual advertising The insertion system has a problem in that it is not applied to Internet broadcast videos.

Republic of Korea Patent Publication No. 10-2000-0054304 (published on September 5, 2000)

Therefore, the purpose of the present invention is to insert a virtual advertisement into a broadcast video and provide it to viewers, recognize the object of the broadcast video, and place the virtual advertisement in a similar area with a size corresponding to the size of the overlay area of the virtual advertisement to be inserted among the object and background. A system and method for inserting a virtual advertisement using area and object recognition of a broadcast video that inserts a virtual advertisement by overlaying and inserting a virtual advertisement by tracking the movement of the object and moving the insertion position of the virtual advertisement within the similar area according to the movement of the object. It is provided.

A virtual advertisement insertion system using area and object recognition of broadcast video according to the present invention to achieve the above purpose: acquires broadcast programming information, advertisement programming condition information, and virtual advertisement, and according to the advertisement programming condition information, An advertisement programming server that determines a broadcast program in which to insert a virtual advertisement among broadcast programs in the broadcast programming information, generates advertisement programming information, and then transmits advertisement operation information including the advertisement programming information and the obtained virtual advertisement; and receiving the advertisement operation information from the advertisement programming server, separately storing advertisement programming information and virtual advertisements, and transmitting the broadcasting program of the broadcasting programming information input from at least one broadcasting company encoder according to the advertising programming of the advertising programming information. It is characterized by including a virtual advertisement server that overlays and inserts a virtual advertisement in the virtual advertisement insertion section of the advertisement programming information of the corresponding broadcast video among broadcast videos and then transmits the virtual advertisement to the user terminal.

The advertisement programming server includes an advertisement storage unit including an advertisement programming DB that stores broadcast programming information and advertisement programming information, and an advertisement DB that stores a plurality of virtual advertisements and advertisement programming condition information corresponding to each virtual advertisement; An advertising communication unit that connects to a wired and wireless data communication network and performs data communication with devices connected to the wired and wireless data communication network; An advertisement encoding unit that receives virtual advertisement and format information, encodes the virtual advertisement into a video format of the format information, and outputs it; And obtaining the broadcast programming information and virtual advertisements and storing them in the advertisement programming DB and advertisement DB of the advertisement storage unit, respectively, and determining a broadcast program in which to insert a virtual advertisement among the broadcast programs of the broadcast programming information to generate advertisement programming information. Afterwards, it characterized by comprising an advertisement programming control unit that transmits the advertisement operation information including the advertisement programming information and the virtual advertisement encoded in the advertisement encoding unit to the virtual advertisement server through the advertisement communication unit.

The advertisement programming control unit includes a broadcast programming information acquisition unit that acquires broadcast programming information from a broadcasting company through the advertising communication unit and stores it in the advertisement programming DB; an advertisement acquisition unit that receives virtual advertisements and advertisement programming condition information from one of an advertiser terminal unit and an administrator terminal unit through the advertisement communication unit and stores them in the advertisement DB; An advertisement conversion unit that outputs the virtual advertisement stored in the advertisement DB and format information corresponding to the broadcast video format of the broadcaster into which the virtual advertisement is to be inserted to the advertisement encoding unit, receives the converted virtual advertisement and stores it in the advertisement DB; and an advertisement operation information providing unit that generates advertisement operation information including the advertisement programming information and the converted virtual advertisement and transmits the advertisement operation information to the virtual advertisement server through the advertisement communication unit.

The virtual advertisement server includes a virtual advertisement storage unit including an advertisement schedule DB for storing advertisement schedule information and a virtual advertisement DB for storing virtual advertisements for a plurality of broadcast video formats; a server communication unit that connects to a wired or wireless data communication network and performs data communication with devices connected to the wired or wireless data communication network; And receiving the advertisement operation information from the advertisement programming server, separating the advertisement programming information and the virtual advertisement and storing them in each of the advertisement programming DB and virtual advertisement DB of the virtual advertisement storage unit, and creating the virtual advertisement according to the advertisement programming of the advertisement programming information. In a similar area of the broadcast video corresponding to the size of the overlay area of the virtual advertisement in the virtual advertisement insertion section of the advertisement programming information of the broadcast image among the broadcast images for the broadcast program of the broadcast programming information input from at least one broadcasting company encoder It is characterized by including a virtual advertisement control unit that overlays and inserts the advertisement and then transmits it to the user terminal.

The virtual advertisement control unit receives the advertisement operation information from the advertisement programming server, detects advertisement programming information and virtual advertisement from the advertisement operation information, and stores the advertisement in the advertisement programming DB and virtual advertisement DB of the virtual advertisement storage unit, respectively. Operational Information Acquisition Department; a broadcast reception processing unit that receives and outputs broadcast video from a broadcast encoder of a specific broadcasting station when a broadcast video request event for a broadcast program of a specific broadcasting station occurs; A similar area of the broadcast image corresponding to the size of the overlay area of the virtual advertisement in the virtual advertisement insertion section of the advertisement programming information of the broadcast image among the broadcast images for the broadcast program of the broadcast programming information of the broadcasting company output from the broadcast reception processing unit. an insertion area detection unit that detects; a virtual advertisement insertion unit that overlays and inserts the virtual advertisement into a similar area detected by the insertion area detection unit; and a video transmission unit that transmits the broadcast video in which the virtual advertisement is inserted to the corresponding user terminal.

The insertion area detection unit includes an object detection unit that applies an object detection artificial intelligence model and detects objects in the broadcast video of the virtual advertisement insertion section through the object detection artificial intelligence model; a similar area space detection unit that detects one or more similar areas of a certain size or more within the broadcast video of the virtual advertisement insertion section; And insertion position determination for determining a virtual advertisement insertion position among one or more similar areas of the detected object and background corresponding to the size of the overlay area of the virtual advertisement, and outputting the determined virtual advertisement insertion location information to the virtual advertisement insertion unit. It includes a unit, wherein the virtual advertisement insertion unit inserts the virtual advertisement into the similar area within the broadcast video corresponding to the virtual advertisement insertion location information.

The insertion area detection unit further includes an object location tracking unit that tracks the location of the detected object and outputs object location information to the virtual advertisement insertion unit, wherein the virtual advertisement insertion unit tracks the location of the detected objects and outputs object location information to the virtual advertisement insertion unit. The virtual advertisement is overlaid and inserted into the similar area so as not to overlap with other areas.

The insertion position determination unit is characterized in that, if an object having a similar area corresponding to the size of the overlay area of the virtual advertisement exists among the detected objects, the similar area of the object is determined as the virtual advertisement insertion target similar area.

If there is no similar area larger than the size of the overlay area of the virtual advertisement, the insertion position determination unit selects a similar area with the largest area, provides size information of the selected similar area to the virtual advertisement insertion unit, and provides the virtual advertisement insertion unit with size information of the selected similar area. When the size information of the similar area is input from the insertion position determination unit, the insertion unit adjusts the size of the overlay area (virtual advertisement) of the virtual advertisement to be smaller than the size of the similar area, and then overlays and inserts the virtual advertisement into the similar area. do.

The insertion position determination unit determines a similar area corresponding to the size of the virtual advertisement overlay area as the virtual advertisement insertion position, and determines a similar area closest to the object corresponding to the advertising content of the virtual advertisement.

The virtual advertisement insertion method using area and object recognition of broadcast video according to the present invention to achieve the above object is: an advertisement programming server acquires broadcast programming information, advertisement programming condition information, and virtual advertisement, and the advertisement programming conditions An advertisement programming process of determining a broadcast program in which to insert a virtual advertisement among the broadcast programs in the broadcast programming information according to the information, generating advertisement programming information, and then transmitting advertisement operation information including the advertisement programming information and the acquired virtual advertisement. ; And a virtual advertisement server receives the advertisement operation information from the advertisement programming server and stores the advertisement programming information and virtual advertisement separately, and the broadcast programming information input from at least one broadcasting company encoder according to the advertisement programming of the advertisement programming information. It is characterized by including a virtual advertisement providing process of overlaying and inserting a virtual advertisement in the virtual advertisement insertion section of the advertisement programming information of the broadcast video of the broadcast program and then transmitting it to the user terminal.

The advertisement programming process includes a broadcast programming information acquisition step in which the advertisement programming control unit of the advertisement programming server acquires broadcast programming information from a broadcaster through a broadcast programming information acquisition unit and stores it in an advertisement programming DB; An advertisement acquisition step in which the advertisement programming control unit receives a virtual advertisement from an advertiser terminal unit and a manager terminal unit through an advertisement acquisition unit and stores the virtual advertisement in an advertisement DB; The advertisement programming control unit outputs the virtual advertisement stored in the advertisement DB and the format information corresponding to the broadcast video format of the broadcaster to insert the virtual advertisement to the advertisement encoding unit through the advertisement conversion unit, thereby converting the virtual advertisement to the broadcast video format. An advertisement conversion step of storing the virtual advertisement converted into format in an advertisement DB; And an advertisement operation information providing step in which the advertisement programming control unit generates advertisement operation information including the advertisement programming information and the converted virtual advertisement through an advertisement operation information provision unit and then transmits the advertisement operation information including the advertisement programming information and the converted virtual advertisement to a virtual advertisement server through an advertisement communication unit. It is characterized by:

In the virtual advertisement provision process, the virtual advertisement control unit of the virtual advertisement server receives the advertisement operation information from the advertisement programming server through the advertisement operation information acquisition unit, detects advertisement programming information and virtual advertisement from the advertisement operation information, and generates the virtual advertisement. Obtaining advertising operation information stored in each of the advertising programming DB and virtual advertising DB of the storage unit; A broadcast reception processing step in which the virtual advertisement control unit receives and outputs a broadcast video from a broadcast encoder of a specific broadcasting station when a broadcast video request event for a broadcast program of a specific broadcasting station occurs; The overlay area of the virtual advertisement in the virtual advertisement insertion section of the advertisement programming information of the broadcast image among the broadcast images of the broadcast program of the broadcasting company's broadcast programming information that the virtual advertisement control unit outputs in real time from the broadcast reception processing unit through the insertion area detection unit. An insertion area detection step of detecting a similar area of the broadcast video corresponding to the size; A virtual advertisement insertion step in which the virtual advertisement control unit overlays and inserts the virtual advertisement into a similar area detected by the insertion area detection unit through a virtual advertisement insertion unit; and an advertisement insertion video transmission step in which the virtual advertisement control unit transmits the broadcast video with the virtual advertisement inserted to a corresponding user terminal through a video transmission unit.

The insertion area detection step includes: an object detection step in which the insertion area detection unit of the virtual advertisement control unit detects an object in the broadcast video of the virtual advertisement insertion section through an object detection unit applying an object detection artificial intelligence model; A similar area space detection step in which the insertion area detection unit detects one or more similar areas of a certain size or more within the broadcast video of the virtual advertisement insertion section through the similar area space detection unit; And the insertion area detection unit determines, through the insertion position determination unit, a similar area corresponding to the size of the overlay area of the virtual advertisement among one or more similar areas of the detected object and background as the virtual advertisement insertion location, and determines the virtual advertisement insertion location information. An insertion position determination step of outputting to a virtual advertisement insertion unit, wherein in the virtual advertisement insertion step, the virtual advertisement insertion unit inserts the virtual advertisement into the similar area in the broadcast video corresponding to the virtual advertisement insertion location information. Do it as

The insertion area detection step further includes an object position tracking step in which the insertion area detector tracks the position of the detected object through an object position tracking unit and outputs object location information to the virtual advertisement insertion unit, wherein the virtual advertisement is inserted. In the step, the virtual advertisement insertion unit overlays and inserts the virtual advertisement into the similar area so as not to overlap the objects corresponding to the positions of the detected objects.

In the insertion position determination step, if an object having a similar area corresponding to the size of the overlay area of the virtual advertisement exists among the detected objects, the insertion position determination unit determines the similar area of the object as the virtual advertisement insertion target similar area. Do it as

The present invention automatically detects objects and identifies similar areas in a section of video at a time corresponding to preset advertisement programming information of a real-time broadcast video, and avoids objects in the similar area corresponding to the size of the overlay area of the virtual advertisement to be inserted. Since it displays virtual advertisements, it has the effect of providing virtual advertisements without affecting the screen displayed in the broadcast video as much as possible.

In addition, the present invention inserts virtual advertisements by determining the virtual advertisement insertion area based on automatic recognition of similar areas and objects according to advertisement programming information, so that various virtual advertisements can be efficiently inserted into broadcast videos for multiple broadcasting companies provided through the Internet. It has the effect of being freely inserted and provided.

In addition, the present invention has the effect of improving the efficiency of advertising because it can display virtual advertising in response to objects corresponding to advertising content.

Figure 1 is a diagram showing the configuration of a communication system including a virtual advertisement insertion system using area and object recognition of broadcast video according to the present invention.
Figure 2 is a diagram showing the detailed configuration of the advertisement programming server of the virtual advertisement insertion system using area and object recognition of broadcast video according to the present invention.
Figure 3 is a diagram showing the detailed configuration of the virtual advertisement server of the virtual advertisement insertion system using area and object recognition of broadcast video according to the present invention.
Figure 4 is a diagram showing the detailed configuration of the insertion area detection unit of the virtual advertisement server in the virtual advertisement insertion system according to the present invention.
Figure 5 is a procedure diagram showing a method of inserting a virtual advertisement using area and object recognition of a broadcast video according to the present invention.
Figure 6 is a flowchart showing a method of detecting a virtual advertisement insertion area among the virtual advertisement insertion methods using area and object recognition of broadcast video according to the present invention.

Hereinafter, with reference to the attached drawings, the configuration and operation of the virtual advertisement insertion system using area and object recognition of broadcast video according to the present invention will be described in detail, and the virtual advertisement insertion area detection method and virtual advertisement insertion method in the system will be described. do.

Figure 1 is a diagram showing the configuration of a communication system including a virtual advertisement insertion system using area and object recognition of broadcast video according to the present invention.

The virtual advertisement insertion system 30 using area and object recognition of broadcast video according to the present invention includes a user terminal 10, a broadcaster encoder 20, an advertiser terminal 40, and an administrator through a wired or wireless data communication network 1. It is connected to the terminal unit 50, etc. to perform mutual data communication.

The wired and wireless data communication network (1) includes an Internet network including a WiFi network, a local area network (LAN), a mobile communication network including 3 Generation (3G), 4G, 5G, etc., and WiBro. It may be a data communication network in which one or more of the following networks are combined.

The user terminal unit 10 selects a broadcast program from a random broadcaster, receives and displays the corresponding broadcast video, and receives and displays the broadcast video with the virtual advertisement inserted through the virtual advertisement insertion system 30. The user terminal unit 100 may be a computer terminal 11 such as a desktop computer, personal computer, laptop, etc., or a mobile terminal 12 such as a smartphone or smart pad.

The broadcaster encoder 20 is configured for each broadcaster and transmits broadcast video according to the broadcast program of the broadcaster's broadcast programming information to the virtual advertisement insertion system 30 according to the Real Time Message Protocol (RTMP) through a wired or wireless data communication network. do. The broadcasting company encoder 20 will provide access information of the user terminal 10 that has selected its own broadcasting company and broadcasting program to the virtual advertisement insertion system 30.

The advertiser terminal unit 40 may provide virtual advertisement request information including advertisement programming condition information of the virtual advertisement to the virtual advertisement insertion system 30 through the wired and wireless data communication network 1. The advertisement programming condition information may include a broadcast program in which a virtual advertisement is desired to be inserted, information on the virtual advertisement insertion section of the broadcast program, advertisement content, etc. The advertisement content will include detailed information such as the type of virtual advertisement and product information according to the type.

The advertiser terminal unit 40 may provide the produced virtual advertisement to the virtual advertisement insertion system 30 through the wired and wireless data communication network 1.

The virtual advertisement insertion system 30 includes an advertisement programming server 100 and a virtual advertisement server 200, and acquires broadcast programming information, advertisement programming condition information, and virtual advertisement for at least one broadcasting station, and the advertisement programming conditions. The virtual advertisement is inserted into the broadcast video for the corresponding broadcast program of the broadcast programming information according to the information, and then the broadcast video for the broadcast program is provided to the user terminal unit 10 that requested the broadcast video. It would be desirable for the advertisement programming server 100 and the virtual advertisement server 200 to be connected through a local network of the wired and wireless data communication network 1. Additionally, the advertisement programming server 100 and the virtual advertisement server 200 may be configured as one device or may be configured as separate devices.

Specifically, the advertisement programming server 100 acquires broadcast programming information, advertisement programming condition information, and virtual advertisements, and determines a broadcast program in which to insert a virtual advertisement among the broadcast programs of the broadcast programming information based on the advertisement programming condition information. After generating the advertisement programming information, the advertisement operation information including the advertisement programming information and the obtained virtual advertisement is transmitted to the virtual advertisement server 200. The virtual advertisement may be obtained from the advertiser terminal 40 or directly through the system administrator's administrator terminal 50.

The virtual advertisement server 200 receives the advertisement operation information from the advertisement programming server 100, stores the advertisement programming information and the virtual advertisement separately, and distributes the virtual advertisement to at least one broadcaster according to the advertisement programming of the advertisement programming information. After overlaying and inserting the broadcast video into the virtual advertisement insertion section of the advertisement programming information of the broadcast video among the broadcast video for the broadcast program of the broadcast program information input from the encoder 20, the broadcast video is sent to the user terminal unit 10 that requested the broadcast video. send. Access information for the user terminal unit 10 may be input from the broadcasting company encoder 20, or may be obtained from a separate broadcasting company server (not shown) that provides a means for selecting broadcast images.

Figure 2 is a diagram showing the detailed configuration of the advertisement programming server of the virtual advertisement insertion system using area and object recognition of broadcast video according to the present invention. The detailed configuration and operation of the advertisement programming server 100 will be described with reference to FIG. 2.

The advertisement programming server 100 includes an advertisement storage unit 110, an advertisement communication unit 20, an advertisement encoding unit 130, and an advertisement programming control unit 140.

The advertisement storage unit 110 includes a program area for storing a control program for controlling the overall operation of the advertisement programming server 100 according to the present invention, a temporary area for temporarily storing data generated during execution of the control program, and the control program. It includes a data area that semi-permanently stores data required for the program and data generated during execution of the control program. According to the present invention, an advertisement programming DB 111 and an advertisement DB 112 will be configured in the data area.

The advertisement programming DB 111 stores broadcast programming information for each broadcasting company and advertisement programming information synchronized with the broadcast programming information for each broadcasting company, and stores virtual advertisement identification information for the advertisement programming information.

The advertisement DB 112 stores a plurality of virtual advertisements, virtual advertisement identification information for each virtual advertisement, and advertisement programming condition information. The virtual advertisement may include a provided virtual advertisement as received from the advertiser and a converted virtual advertisement converted into the broadcast video format of the broadcasting company's broadcast program to be provided. If the video format of the provided virtual advertisement and the format of the broadcast video are the same, the provided virtual advertisement will become a converted virtual advertisement.

The advertising communication unit 120 is connected to the wired or wireless data communication network 1 either wired or wirelessly, and includes a user terminal 10, a broadcaster encoder 20, and an advertiser terminal 40 connected to the wired and wireless data communication network 1. , devices such as the administrator terminal unit 50, the virtual advertisement server 200, etc., and the advertisement programming control unit 140 enable mutual data communication.

The advertisement encoding unit 130 encodes the virtual advertisement input from the advertisement programming control unit 140 into a format corresponding to the video format information (control data) input from the advertisement programming control unit 140, and then encodes the virtual advertisement input from the advertisement programming control unit 140. ) is provided.

The advertisement encoding unit 130 may be configured as a separate server.

The advertisement programming control unit 140 includes a broadcast programming information acquisition unit 141, an advertisement acquisition unit 142, an advertisement conversion unit 143, and an advertisement operation information provision unit 144, and an advertisement programming server according to the present invention ( 100) controls the overall operation.

The broadcast programming information acquisition unit 141 acquires broadcast programming information from the broadcasting company, the administrator terminal unit 50 of the system manager, etc. through the advertising communication unit 120 and stores it in the advertising programming DB 111.

The advertisement acquisition unit 142 receives advertisement programming condition information, virtual advertisements, and the virtual advertisement from one of the advertiser's advertiser terminal 40 and the system manager's manager terminal 50 through the advertisement communication unit 120. Virtual advertisement information is received and stored in the advertisement DB 112. The virtual advertisement information includes meta information, which is additional information about the virtual advertisement, and the meta information may include a homepage lending URL, information for collecting statistics on the number of viewers, etc.

The advertisement conversion unit 143 outputs format information about the format of the broadcast video of the broadcaster into which the virtual advertisement is to be inserted to the advertisement encoding unit 130 according to the virtual advertisement and advertisement programming information stored in the advertisement DB 112. , the virtual advertisement converted accordingly is input and stored in the advertisement DB 112.

The advertisement operation information provider 144 generates advertisement operation information including the advertisement programming information stored in the advertisement programming DB 111 and the converted virtual advertisement stored in the advertisement DB 112, and the virtual advertisement server 200 ) and send it to

The advertisement programming information and virtual advertisement transmitted by the advertisement operation information providing unit 144 may generate and transmit advertisement operation information including the entire advertisement programming information and virtual advertisements therefor, and advertisements added to the previous advertisement operation information. Only advertising operation information including programming information and virtual advertisements may be transmitted.

Figure 3 is a diagram showing the detailed configuration of the virtual advertisement server of the virtual advertisement insertion system using area and object recognition of broadcast video according to the present invention.

Referring to FIG. 3, the virtual advertisement server 200 includes a virtual advertisement storage unit 210, a server communication unit 220, and a virtual advertisement control unit 230.

The virtual advertisement storage unit 210 includes a program area for storing a control program that controls the overall operation of the virtual advertisement server 200 according to the present invention, a temporary area for temporarily storing data generated during execution of the control program, and the control program. It includes a data area that semi-permanently stores data required for execution and data generated during execution of the control program. According to the present invention, an advertisement programming DB 211 and a virtual advertisement DB 212 will be configured in the data area.

The advertisement programming DB 211 stores advertisement programming information.

The virtual advertisement DB 212 stores a plurality of virtual advertisement identification information and virtual advertisements corresponding to the virtual advertisement identification information.

The server communication unit 220 is connected to the wired or wireless data communication network 1 either wired or wirelessly, and includes a user terminal 10, a broadcaster encoder 20, and an advertiser terminal 40 connected to the wired and wireless data communication network 1. , devices such as the administrator terminal unit 50, the advertisement programming server 100, etc., and the virtual advertisement control unit enable mutual data communication.

The virtual advertisement control unit 230 includes an advertisement operation information acquisition unit 231, a broadcast reception processing unit 232, an insertion area detection unit 233, a virtual advertisement insertion unit 234, and an image transmission unit 235, according to the present invention. Controls the overall operation of the virtual advertising server 200 according to.

The advertisement operation information acquisition unit 231 receives the advertisement operation information from the advertisement programming server 100, detects advertisement programming information and virtual advertisements (including virtual advertisement identification information) from the advertisement operation information, and generates the virtual advertisement. It is stored in each of the advertisement programming DB 211 and the virtual advertisement DB 212 of the storage unit 200.

When a broadcast video request event for a broadcast program of a specific broadcaster occurs, the broadcast reception processing unit 232 receives the broadcast video from the broadcaster encoder 20 of the broadcaster through the server communication unit 220 and sends it to the insertion area detection unit 233. Print out.

The insertion area detection unit 233 refers to the broadcast video of the broadcast program received from the broadcast reception processing unit 232 with the advertisement programming information of the advertisement programming DB 211 and compares the virtual advertisement non-insertion section and the virtual advertisement in which the virtual advertisement is not overlaid. A virtual advertisement insertion section in which an advertisement is overlaid is distinguished, and in the virtual advertisement non-insertion section, the broadcast video is transmitted as is through the video transmission unit 235 to the user terminal unit 10 that requested the broadcast through the server communication unit 220. And, in the virtual advertisement insertion section where the virtual advertisement is inserted according to the advertisement programming information, a similar area of the broadcast image corresponding to the size of the overlay area of the virtual advertisement is detected, and then the broadcast image is inserted into the virtual advertisement insertion section along with the insertion area information. Output as (234).

The virtual advertisement insertion unit 234 overlays and inserts the virtual advertisement into the similar area detected by the insertion area detection unit 233.

The video transmission unit 235 transmits the broadcast video input from the broadcast reception processing unit 232 and the virtual advertisement insertion unit 234 to the requesting user terminal unit 10.

Figure 4 is a diagram showing the detailed configuration of the insertion area detection unit of the virtual advertisement server in the virtual advertisement insertion system according to the present invention. The detailed configuration and operation of the insertion area detection unit 233 will be described with reference to FIG. 4 .

The insertion area detection unit 233 includes an object detection unit 241, a similar area space detection unit 242, and an insertion position determination unit 244, and may further include an object position tracking unit 243 depending on the embodiment. .

The object detection unit 241 has an object detection artificial intelligence model learned for a large number of various objects, detects objects present in the broadcast video input through the object detection artificial intelligence model, and detects the broadcast video as a background and Classifies objects and outputs the type, location information, and background information of the detected object. The location information may be coordinate information of the bounding box of the detected object, and the background information may be information about a part excluding the object.

The similar area spatial detection unit 242 receives the type, location information, and background information about the detected object from the object detection unit 241, detects at least one similar area from the object and background of the input broadcast video, and detects the detected similar area. Similar area information for is output to the insertion position determination unit 244. The similar area is an area of a certain size or more with the same pattern, same color, etc., and may be an area on the background or object. Additionally, the similar area information may be coordinate information of a bounding box forming the similar area, and the shape of the bounding box may be a triangle, square, circle, polygon, etc., but a square shape is preferable. The similar area bounding box may be of any shape, but it is preferable to have a rectangular shape that includes the similar area as much as possible in order to insert a virtual advertisement.

The insertion position determination unit 244 receives similar area information about one or more similar areas detected from the similar area space detection unit 242 and receives the information included in the advertisement programming condition information of the advertisement operation information from the advertisement operation information acquisition unit 231. After receiving the virtual advertisement overlay area size (ratio) information, a similar area with a size corresponding to the size of the virtual advertisement overlay area among the similar areas of the similar area information is determined as the virtual advertisement insertion position, and the virtual advertisement insertion unit 234 determines the virtual advertisement insertion position. Outputs similar area information about similar areas.

In addition, if there is no similar area corresponding to the size of the virtual advertisement overlay area, the insertion position determination unit 244 selects the largest similar area among the similar areas and inserts the similar area information about the selected similar area into the virtual advertisement insertion unit. It will output as (234).

In addition, the insertion position determination unit 244 receives the type and location information of the object from the object detection unit 241, receives advertisement content included in the advertisement programming condition information from the advertisement operation information acquisition unit 231, and receives the advertisement content included in the advertisement programming condition information. If a similar area corresponding to the size of the virtual advertisement overlay area exists in the object corresponding to the content, the similar area is preferentially selected and output, and if there is no similar area corresponding to the size of the virtual advertisement overlay area in the object. , it may be configured to select and output a similar area closest to the object corresponding to the advertisement content information.

Additionally, the insertion position determination unit 244 may be configured to select and output a similar area closest to the set position when a preset position (e.g., bottom right, bottom left, center, etc. of a broadcast image) is determined.

The object location tracking unit 243 tracks the changing positions of each object detected by the object detection unit 241 and outputs the changed positions to the virtual advertisement insertion unit 234. Then, the virtual advertisement insertion unit 234 displays the virtual advertisement in the determined similar area, but when an object outside the similar area enters the similar area, if the size of the similar area is maintained, the virtual advertisement is continuously overlaid and inserted into the similar area. And, if the size is not maintained, it may be configured to request the insertion position determination unit 244 to reselect a similar area.

In addition, when the virtual advertisement insertion unit 243 determines the similar area in which the virtual advertisement will be inserted, the virtual advertisement is continuously inserted into the initially determined location of the similar area (it does not matter if it is not a similar area thereafter) regardless of the intrusion of an object. It may be configured to do so.

Figure 5 is a procedure diagram showing a method of inserting a virtual advertisement using area and object recognition of a broadcast video according to the present invention.

Referring to FIG. 5, the advertisement programming server 100 acquires programming information for each broadcaster and stores it in the advertisement programming DB 111 (S111).

The advertisement programming server 100 monitors whether virtual advertisements and advertisement programming condition information are obtained from either the advertiser terminal 40 or the manager terminal 50 (S113).

When the virtual advertisement and advertisement programming condition information are acquired, the advertisement programming server 100 organizes the virtual advertisement into a broadcast program for each broadcasting company in response to the broadcast programming information and advertisement programming condition information and generates advertisement programming information according to the programming of the virtual advertisement. Then, it is stored in the advertisement programming DB (111) (S115).

The advertisement programming server 100 converts the broadcast video to correspond to the video format of the broadcasting company that will provide the virtual advertisement and stores it in the advertisement DB 112 (S117).

When the virtual advertisement conversion according to the advertisement programming information and virtual advertisement programming is completed, the advertisement programming server 100 generates advertisement operation information including the advertisement programming information and the converted virtual advertisement (S119), and then the virtual advertisement server 200 ) is provided (S120).

When advertising operation information is received from the advertisement programming server 100, the virtual advertisement server 200 separates the advertisement programming information and virtual advertisement from the received advertisement operation information and stores them in the advertisement programming DB 211 and the virtual advertisement DB 212, respectively. Save (S121).

When a broadcast video request event occurs, the broadcaster encoder 20 loads the broadcast video requested by the broadcast video request event (S123) and transmits it to the virtual advertisement server 200 (S125). The broadcast video request event may be generated according to a broadcasting company's broadcast programming time, or may be generated by a request for the broadcast video from an arbitrary viewer, that is, a user. It would be desirable for the broadcast video to be transmitted to the virtual advertisement server 200 through Real Time Message Protocol (RTMP).

The virtual advertising server 200 monitors whether broadcast video is received from the broadcaster encoder 20 after receiving the advertising operation information (S127), and when the broadcast video is received, it refers to the advertising programming information and adds information to the received broadcast video. Determine whether a virtual advertisement exists (S129).

If a scheduled virtual advertisement exists, the virtual advertisement server 200 detects the advertisement insertion position while transmitting the broadcast image in real time through the wired and wireless data communication network 1, and determines a similar area for virtual advertisement insertion within the broadcast image (S131).

Once the similar area is determined, the virtual advertisement server 200 overlays and inserts the virtual advertisement into the determined similar area of the broadcast image in the virtual advertisement insertion section according to the advertisement programming information of the broadcast image (S133).

When a virtual advertisement is inserted, the virtual advertisement server 200 transmits the broadcast video with the virtual advertisement inserted to the user terminal units 10 (S135).

Figure 6 is a flowchart showing a method of detecting a virtual advertisement insertion area among the virtual advertisement insertion methods using area and object recognition of broadcast video according to the present invention.

Referring to FIG. 6, when the broadcast video to be transmitted to the user terminal unit 10 is received from the broadcaster encoder 20, the virtual advertisement control unit 230 of the virtual advertisement server 200 detects the virtual advertisement insertion position by referring to the advertisement programming information. Monitor whether an event, that is, a similar area detection event, occurs (S211).

In other words, the virtual advertisement control unit 230 monitors whether the section of the broadcast image being transmitted during transmission of the broadcast image is the starting point of the virtual advertisement insertion section of the advertisement programming information.

When a virtual advertisement insertion position detection event occurs, the virtual advertisement control unit 230 transmits the broadcast video of the virtual advertisement insertion section to the object detection artificial intelligence model of the object detection unit 241 in units of frame images or certain time units (e.g., in seconds). Apply it to detect objects in the broadcast video and determine the type and location of the detected object (S213).

When the type and location of the object are detected, the virtual advertisement control unit 230 tracks the location of each object through the object position tracking unit 243 (S215) and detects one or more similar areas in the object and background of consecutive frame images. (S217).

When a similar area is detected, the virtual advertisement control unit 230 checks whether a similar area with a size corresponding to the size of the virtual advertisement overlay area exists through the insertion position determination unit 244 (S219).

As a result of the inspection, if a similar area with a size corresponding to the size of the virtual advertisement overlay area exists, the virtual advertisement control unit 230 overlays and inserts the virtual advertisement into the similar area through the virtual advertisement insertion unit 234, but at least one When a random object enters the similar area according to the location tracking of the above object, a virtual advertisement is inserted into the similar area or another similar area by reflecting the object entering the similar area (S221).

Meanwhile, if the similar area is a similar area on the object, the virtual advertisement control unit 230 will overlay and display the virtual advertisement on the object according to the position movement of the object.

On the other hand, if there is no similar area with a size corresponding to the size of the virtual advertisement overlay area, the virtual advertisement control unit 230 selects the largest similar area among the similar areas detected through the insertion position determination unit 244, and selects the virtual advertisement overlay area. The size of the corresponding virtual advertisement is reduced through the advertisement insertion unit 234 to correspond to the selected similar area, and then overlaid and inserted into the similar area (S223).

Meanwhile, it is common knowledge in the technical field that the present invention is not limited to the above-described typical preferred embodiments, but can be implemented by various improvements, changes, substitutions, or additions without departing from the gist of the present invention. Anyone who has will be able to understand it easily. If the implementation by such improvement, change, substitution or addition falls within the scope of the appended patent claims below, the technical idea thereof should also be regarded as belonging to the present invention.

1: Wired and wireless data communication network 10: User terminal
20: Broadcasting company encoder 30: Virtual advertisement insertion system
40: Advertiser terminal 50: Administrator terminal
100: Advertisement programming server 110: Advertisement storage unit
111: Advertisement programming DB 112: Advertisement DB
120: Advertising communication department 130: Advertising encoding department
140: Advertisement programming control unit 141: Broadcast programming information acquisition unit
142: Advertisement acquisition unit 143: Advertisement conversion unit
144: Advertisement operation information provision unit 200: Virtual advertisement server
210: Virtual advertisement storage unit 211: Advertisement programming DB
212: Virtual advertisement DB (212) 220: Server communication department
230: Virtual advertisement control unit 231: Advertisement operation information acquisition unit
232: Broadcast reception processing unit 233: Insertion area detection unit
234: Virtual advertisement insertion unit 235: Video transmission unit
241: object detection unit 242: similar area space detection unit
243: object position tracking unit 244: insertion position determination unit

Claims (16)

After obtaining broadcast programming information, advertisement programming condition information, and virtual advertisements, determining a broadcast program in which to insert a virtual advertisement among the broadcast programs of the broadcast programming information according to the advertisement programming condition information, and generating advertisement programming information, the advertisement An advertisement programming server that transmits advertisement operation information including programming information and acquired virtual advertisements; and
The advertisement operation information is received from the advertisement programming server, the advertisement programming information and the virtual advertisement are stored separately, and the broadcast program of the broadcast programming information input from at least one broadcasting company encoder according to the advertisement programming of the advertisement programming information is stored separately. Includes a virtual advertisement server that overlays and inserts a virtual advertisement in the virtual advertisement insertion section of the advertisement programming information of the broadcast video and then transmits it to the user terminal,
The virtual advertising server is,
A virtual advertisement storage unit including an advertisement programming DB that stores advertisement programming information and a virtual advertisement DB that stores virtual advertisements for a plurality of broadcast video formats;
a server communication unit that connects to a wired or wireless data communication network and performs data communication with devices connected to the wired or wireless data communication network; and
The advertisement operation information is received from the advertisement programming server, the advertisement programming information and the virtual advertisement are separated, and stored in each of the advertisement programming DB and virtual advertisement DB of the virtual advertisement storage unit, and the virtual advertisement is generated according to the advertisement programming of the advertisement programming information. Overlay on a similar area of the broadcast video corresponding to the size of the overlay area of the virtual advertisement in the virtual advertisement insertion section of the advertisement programming information of the broadcast image among the broadcast images for the broadcast program of the broadcast programming information input from at least one broadcasting company encoder. It includes a virtual advertisement control unit that is inserted and transmitted to the user terminal,
The virtual advertisement control unit,
an advertisement operation information acquisition unit that receives the advertisement operation information from the advertisement programming server, detects advertisement programming information and virtual advertisements from the advertisement operation information, and stores them in each of the advertisement programming DB and virtual advertisement DB of the virtual advertisement storage unit;
a broadcast reception processing unit that receives and outputs broadcast video from a broadcast encoder of a specific broadcasting station when a broadcast video request event for a broadcast program of a specific broadcasting station occurs;
A similar area of the broadcast image corresponding to the size of the overlay area of the virtual advertisement in the virtual advertisement insertion section of the advertisement programming information of the broadcast image among the broadcast images for the broadcast program of the broadcast programming information of the broadcasting company output from the broadcast reception processing unit. an insertion area detection unit that detects;
a virtual advertisement insertion unit that overlays and inserts the virtual advertisement into a similar area detected by the insertion area detection unit; and
It includes a video transmission unit that transmits the broadcast video in which the virtual advertisement is inserted to the corresponding user terminal;
The insertion area detection unit,
An object detection unit that applies an object detection artificial intelligence model and detects objects in the broadcast video of the virtual advertisement insertion section through the object detection artificial intelligence model;
a similar area space detection unit that detects one or more similar areas of a certain size or more from objects and backgrounds in the broadcast video of the virtual advertisement insertion section; and
An insertion position determination unit that determines a similar area corresponding to the size of the overlay area of the virtual advertisement among the detected object and one or more similar areas of the background as the virtual advertisement insertion location, and outputs the determined virtual advertisement insertion location information to the virtual advertisement insertion unit. Including,
A virtual advertisement insertion system using area and object recognition of a broadcast image, wherein the virtual advertisement insertion unit inserts the virtual advertisement into the similar area in the broadcast image corresponding to the virtual advertisement insertion location information.
According to paragraph 1,
The advertisement programming server,
An advertisement storage unit including an advertisement programming DB that stores broadcast programming information and advertisement programming information, and an advertisement DB that stores a plurality of virtual advertisements and advertisement programming condition information corresponding to each virtual advertisement;
An advertising communication unit that connects to a wired and wireless data communication network and performs data communication with devices connected to the wired and wireless data communication network;
An advertisement encoding unit that receives virtual advertisement and format information, encodes the virtual advertisement into a video format of the format information, and outputs it; and
After obtaining the broadcast programming information and the virtual advertisement and storing them in the advertisement programming DB and advertisement DB of the advertisement storage unit, respectively, determining the broadcast program in which to insert the virtual advertisement among the broadcast programs in the broadcast programming information, and generating the advertisement programming information , an area and object of a broadcast video, comprising an advertisement programming control unit that transmits advertisement operation information including the advertisement programming information and a virtual advertisement encoded in the advertisement encoding unit to the virtual advertisement server through the advertisement communication unit. Virtual advertisement insertion system using recognition.
According to paragraph 2,
The advertisement programming control unit,
a broadcast programming information acquisition unit that acquires broadcast programming information from a broadcasting company through the advertising communication unit and stores it in the advertising programming DB;
an advertisement acquisition unit that receives virtual advertisements and advertisement programming condition information from one of an advertiser terminal unit and an administrator terminal unit through the advertisement communication unit and stores them in the advertisement DB;
An advertisement conversion unit that outputs the virtual advertisement stored in the advertisement DB and format information corresponding to the broadcast video format of the broadcaster into which the virtual advertisement is to be inserted to the advertisement encoding unit, receives the converted virtual advertisement and stores it in the advertisement DB; and
An advertisement operation information providing unit that generates advertisement operation information including the advertisement programming information and the converted virtual advertisement and transmits the advertisement operation information to the virtual advertisement server through the advertisement communication unit using area and object recognition of the broadcast video. Virtual ad insertion system.
delete delete delete According to paragraph 1,
The insertion area detection unit,
It further includes an object location tracking unit that tracks the location of the detected object and outputs object location information to the virtual advertisement insertion unit,
The virtual advertisement insertion part,
A virtual advertisement insertion system using area and object recognition of broadcast video, characterized in that the virtual advertisement is overlaid and inserted into the similar area in response to the positions of the detected objects so as not to overlap the objects.
In clause 7,
The insertion position determination unit,
If there is an object with a similar area corresponding to the size of the overlay area of the virtual advertisement among the detected objects, the similar area of the object is determined as a similar area to insert the virtual advertisement using area and object recognition of the broadcast video. Virtual ad insertion system.
According to clause 8,
The insertion position determination unit,
If there is no similar area larger than the size of the overlay area of the virtual advertisement, the similar area with the largest area is selected, and the size information of the selected similar area is provided to the virtual advertisement insertion unit,
The virtual advertisement insertion unit,
When the size information of the similar area is input from the insertion position determination unit, the size of the overlay area (virtual advertisement) of the virtual advertisement is adjusted to be smaller than the size of the similar area and then overlaid and inserted into the similar area. A virtual advertisement insertion system using area and object recognition.
According to clause 9,
The insertion position determination unit,
Area and object recognition of broadcast video, characterized in that a similar area corresponding to the size of the virtual advertisement overlay area is determined as the virtual advertisement insertion position, and the similar area closest to the object corresponding to the advertising content of the virtual advertisement is determined. Virtual advertisement insertion system used.
The advertisement programming server acquires broadcast programming information, advertisement programming condition information, and virtual advertisements, determines a broadcast program to insert a virtual advertisement among the broadcast programs of the broadcast programming information according to the advertisement programming condition information, and generates advertisement programming information. Afterwards, an advertisement programming process of transmitting advertisement operation information including the advertisement programming information and the obtained virtual advertisement; and
A virtual advertisement server receives the advertisement operation information from the advertisement programming server, stores advertisement programming information and virtual advertisements separately, and stores the broadcast programming information input from at least one broadcasting company encoder according to the advertisement programming of the advertisement programming information. It includes a virtual advertisement provision process of overlaying and inserting a virtual advertisement into the virtual advertisement insertion section of the advertisement programming information of the broadcast image among broadcast images for a broadcast program and then transmitting the virtual advertisement to the user terminal;
The process of providing virtual advertising is,
The virtual advertisement control unit of the virtual advertisement server receives the advertisement operation information from the advertisement programming server through the advertisement operation information acquisition unit, detects advertisement programming information and virtual advertisements from the advertisement operation information, and creates an advertisement programming DB of the virtual advertisement storage unit. Obtaining advertising operation information stored in each virtual advertising DB;
A broadcast reception processing step in which the virtual advertisement control unit receives and outputs a broadcast video from a broadcast encoder of a specific broadcasting station when a broadcast video request event for a broadcast program of a specific broadcasting station occurs;
The overlay area of the virtual advertisement in the virtual advertisement insertion section of the advertisement programming information of the broadcast image among the broadcast images of the broadcast program of the broadcasting company's broadcast programming information that the virtual advertisement control unit outputs in real time from the broadcast reception processing unit through the insertion area detection unit. An insertion area detection step of detecting a similar area of the broadcast video corresponding to the size;
A virtual advertisement insertion step in which the virtual advertisement control unit overlays and inserts the virtual advertisement into a similar area detected by the insertion area detection unit through a virtual advertisement insertion unit; and
An advertisement insertion video transmission step in which the virtual advertisement control unit transmits the broadcast video in which the virtual advertisement is inserted to a corresponding user terminal through a video transmission unit,
The insertion area detection step is,
An object detection step in which the insertion area detection unit of the virtual advertisement control unit detects an object in the broadcast video of the virtual advertisement insertion section through an object detection unit applying an object detection artificial intelligence model;
A similar area space detection step in which the insertion area detection unit detects one or more similar areas of a certain size or more from objects and backgrounds in the broadcast video of the virtual advertisement insertion section through the similar area space detection unit; and
The insertion area detection unit determines, through the insertion position determination unit, a similar area corresponding to the size of the overlay area of the virtual advertisement among one or more similar areas of the detected object and background as the virtual advertisement insertion location, and provides the determined virtual advertisement insertion location information. Including an insertion position decision step for outputting to the virtual advertisement insertion section,
In the virtual advertisement insertion step,
A virtual advertisement insertion method using area and object recognition of a broadcast image, characterized in that the virtual advertisement insertion unit inserts the virtual advertisement into the similar area in the broadcast image corresponding to the virtual advertisement insertion location information.
According to clause 11,
The advertising programming process is
A broadcast programming information acquisition step in which the advertisement programming control unit of the advertisement programming server acquires broadcast programming information from a broadcaster through a broadcast programming information acquisition unit and stores the broadcast programming information in an advertisement programming DB;
An advertisement acquisition step in which the advertisement programming control unit receives a virtual advertisement from an advertiser terminal unit and a manager terminal unit through an advertisement acquisition unit and stores the virtual advertisement in an advertisement DB;
The advertisement programming control unit outputs the virtual advertisement stored in the advertisement DB and the format information corresponding to the broadcast video format of the broadcaster to insert the virtual advertisement to the advertisement encoding unit through the advertisement conversion unit, thereby converting the virtual advertisement to the broadcast video format. An advertisement conversion step of storing the virtual advertisement converted into format in an advertisement DB; and
An advertisement operation information providing step wherein the advertisement programming control unit generates advertisement operation information including the advertisement programming information and the converted virtual advertisement through an advertisement operation information provision unit and then transmits the advertisement operation information including the advertisement programming information and the converted virtual advertisement to a virtual advertisement server through an advertisement communication unit. A virtual advertisement insertion method using area and object recognition in broadcast video, characterized by:
delete delete According to clause 11,
The insertion area detection step is,
It further includes an object location tracking step in which the insertion area detection unit tracks the location of the detected object through the object location tracking unit and outputs object location information to the virtual advertisement insertion unit,
In the virtual advertisement insertion step,
A virtual advertisement insertion method using area and object recognition of a broadcast video, characterized in that the virtual advertisement insertion unit overlays and inserts the virtual advertisement into the similar area so as not to overlap the objects corresponding to the positions of the detected objects.
According to clause 15,
In the insertion position determination step,
The area of the broadcast video, wherein the insertion position determination unit determines the similar area of the object as the virtual advertisement insertion target similarity area if there is an object having a similar area corresponding to the size of the overlay area of the virtual advertisement among the detected objects. and a virtual advertisement insertion method using object recognition.