WO2018122895A1

WO2018122895A1 - Image processing device, image processing method, image processing program, and image monitoring system

Info

Publication number: WO2018122895A1
Application number: PCT/JP2016/088620
Authority: WO
Inventors: 司深澤; 浩平岡原; 古木　一朗
Original assignee: 三菱電機株式会社
Priority date: 2016-12-26
Filing date: 2016-12-26
Publication date: 2018-07-05
Also published as: JP6289762B1; JPWO2018122895A1

Abstract

Provided is an image processing device (10) for superimposing and displaying additional information relating to an object (41), which is an object of interest present in a camera image, as a tag, said device including: an object detection unit (12); a tag information management unit (13) for managing a database in which the additional information is stored; a tag arrangement determination unit (14) for performing first processing for acquiring additional information relating to the detected object (41), second processing for setting a search region including the detected object (41), and third processing for determining, when a background image region (that is, a region other than a foreground image region composed of a region occupied by the object and a region occupied by a tag) in the search region has a displayable region in which a tag (51) can be displayed as additional information, the display position of the tag (51) in the displayable region; and a tag superimposing unit (15) for superimposing and displaying the tag (51) on a camera image.

Description

Video processing apparatus, video processing method, video processing program, and video monitoring system

The present invention relates to a video processing apparatus, a video processing method, a video processing program, and a video processing program for superimposing and displaying additional information related to an object as a target of interest present in a camera video as a tag on the camera video on a display device, and The present invention relates to a video monitoring system including the video processing device.

Conventionally, there is a system in which additional information is superimposed and displayed on a camera video (captured image) displayed on a display device based on video data acquired by camera shooting.

For example, Patent Document 1 proposes a system for displaying additional information such as subtitles on a head mounted display worn by a viewer watching the stage. This system detects a human figure as an object from an image existing in the direction of the user's line of sight, and displays additional information related to the human figure.

Also, Patent Document 2 proposes a device that causes additional information to follow an object whose position changes in the camera image when the shooting direction of the camera changes.

JP 2012-108793 A (for example, paragraphs 0055 to 0057) JP 2014-086988 A (for example, paragraphs 0035 to 0044)

However, in the above-described conventional technology, in the camera image (captured image) displayed on the display device, additional information of another object is displayed on top of a certain object or displayed on the additional information of a certain object. There is a problem that it may be difficult to monitor the object and the additional information because the additional information of the other objects are displayed in an overlapping manner.

The present invention has been made to solve the above-described problems, and can easily monitor an object and a tag as additional information related to the object in a camera image displayed on a display device. It is an object of the present invention to provide a video processing device, a video processing method, a video processing program, and a video monitoring system including the video processing device.

A video processing apparatus according to an aspect of the present invention is a video processing that causes a display device to superimpose and display additional information related to an object as a target of interest existing in a camera video acquired by camera shooting as a tag on the camera video. An apparatus for detecting an object, wherein a region occupied by the object and a region occupied by the tag in the camera video are set as a foreground image region, and a region other than the foreground image region in the camera video is set as a background image region. A detection unit; a tag information management unit that manages a database that stores the additional information; and a first process that acquires the additional information related to the object detected by the object detection unit from the tag information management unit. , Including the object detected by the object detection unit. When there is a displayable area that can display a tag as the additional information acquired in the first process in the background image area in the search area and the second process of setting the search area, A tag arrangement determining unit that performs a third process of determining a display position of the tag within the displayable region; and a tag superimposing unit that generates composite video data for displaying the tag superimposed on the camera video.

The video processing method according to another aspect of the present invention is a video in which additional information related to an object as a target of interest present in a camera video acquired by camera shooting is superimposed and displayed as a tag on the camera video on a display device. In the processing method, an area occupied by the object and an area occupied by the tag in the camera image are set as a foreground image area, and an area other than the foreground image area in the camera image is set as a background image area, and the object is detected. A search step including a detection step, a first process for acquiring the additional information related to the object detected by the detection step from a database managing the additional information, and a search area including the object detected by the detection step And a second process for setting the A third display that determines a display position of the tag within the displayable area when there is a displayable area that can display the tag as the additional information acquired in the first process in the scene area; An arrangement determining step for performing processing, and a superimposing step for generating composite video data for superimposing and displaying the tag on the camera video.

According to the present invention, in the camera image displayed on the display device, the tag as the additional information related to the object is arranged in the background image area, so that it is easy to monitor other objects and the additional information. is there.

It is a functional block diagram which shows schematic structure of the video processing apparatus which concerns on Embodiment 1 of this invention. 2 is a diagram illustrating a hardware configuration of a video processing apparatus according to Embodiment 1. FIG. It is a figure which shows an example of the tag information table stored in the tag information table DB part in Embodiment 1 in a table format. 6 is a diagram illustrating an example of object detection by an object detection unit according to Embodiment 1. FIG. (A)-(c) is a figure which shows the other example of the tag in Embodiment 1. FIG. 6 is a diagram illustrating an example of an object information list created by an object detection unit according to Embodiment 1 in a table format. FIG. 6 is a diagram for explaining a search area in the first embodiment. FIG. FIG. 10 is a diagram for explaining an enlarged search region in the first embodiment. 3 is a flowchart illustrating an operation of a video reception unit in the first embodiment. 4 is a flowchart illustrating an operation of a tag information management unit in the first embodiment. 3 is a flowchart illustrating an operation of an object detection unit according to the first embodiment. 6 is a flowchart showing an operation of a tag arrangement determining unit in the first embodiment. 6 is a flowchart showing the operation of the tag superposition unit in the first embodiment. It is a figure which shows an example of the tag information table stored in the tag information table DB part of the video processing apparatus which concerns on Embodiment 2. FIG. (A)-(c) is a figure for demonstrating the effect of the video processing apparatus which concerns on Embodiment 2. FIG. It is a figure which shows an example of the tag information table stored in the tag information table DB part of the video processing apparatus which concerns on Embodiment 3. FIG. (A)-(c) is a figure for demonstrating the effect of the video processing apparatus which concerns on Embodiment 3. FIG. 6 is a functional block diagram illustrating a schematic configuration of a modification of the video processing device according to Embodiments 1 to 3. FIG.

Hereinafter, a video processing device, a video processing method, a video processing program, and a video monitoring system according to embodiments of the present invention will be described with reference to the accompanying drawings. The video processing device according to the embodiment is a tag placement device that can place a tag as additional information related to an object (subject) as a target of interest in a camera video at an appropriate position on the camera video. . Also, the video processing method and the video processing program according to the embodiments are respectively a tag placement method and a tag placement program for placing a tag as additional information related to an object in a camera video at an appropriate position on the camera video. It is. The video monitoring system according to the embodiment is a system having one or more cameras, a video processing device, and one or more display devices. The following embodiments are merely examples, and various modifications can be made within the scope of the present invention.

<< 1 >> Embodiment 1
<< 1-1 >> Configuration FIG. 1 is a functional block diagram showing a schematic configuration of a video processing apparatus 10 according to Embodiment 1 of the present invention. The video processing apparatus 10 is an apparatus capable of executing the video processing method according to the first embodiment and the video processing program according to the first embodiment.

In FIG. 1, a video processing device 10 can communicate with a camera 20 as a photographing device for photographing a subject and a display device 30 that displays a video based on inputted video data on a screen 31 in a wired or wireless manner. It is connected. A network such as the Internet may be interposed between the video processing device 10 and the camera 20 and between the video processing device 10 and the display device 30. In addition, the camera 20, the video processing device 10, and the display device 30 can constitute a video monitoring system that monitors the inside of a facility such as a factory or a public area with a camera video. In addition, a plurality of cameras may be connected to the video processing device 10, and a plurality of display devices corresponding to the plurality of cameras may be connected. In addition, a plurality of cameras may be connected to the video processing device 10, and a plurality of videos corresponding to the plurality of cameras may be sequentially displayed on the display device 30.

The video processing device 10 uses the camera 31 on the screen 31 of the display device 30 to display additional information related to the object (subject) 41 as a target of interest that exists in the camera video corresponding to the video data D1 acquired by photographing with the camera 20. It can be displayed superimposed on the video as a tag 51. In the present application, an area occupied by one or more objects 41 and an area occupied by one or more tags 51 in the camera video on the screen 31 are referred to as a “foreground image area”, and other than the foreground image area in the camera video on the screen 31. The area is referred to as a “background image area”.

As shown in FIG. 1, a tag as an additional information management unit that manages a database (tag information table DB) of tag information as additional information stored in a video reception unit 11, an object detection unit 12, and a storage unit. And an information management unit 13. The video processing apparatus 10 may include a tag information table DB (DataBase) unit 17 as a storage unit that stores tag information. However, the tag information table DB unit 17 may be a storage device external to the video processing device 10. Further, the video processing apparatus 10 includes a tag placement determination unit 14 as an additional information placement determination unit, a tag superposition unit 15 as an additional information superposition unit, and a video output unit 16.

The video receiving unit 11 receives video data D1 as camera video generated by camera shooting, performs necessary processing on the video data D1, and outputs video data. When the video signal D1 is compression encoded data, the video reception unit 11 decodes the received compression encoded data as video data (uncompressed image data), for example, RAW image data D2. Is generated. The video receiving unit 11 provides the RAW image data D <b> 2 to the object detection unit 12 and the tag superimposing unit 15.

The tag information management unit 13 associates an object ID that is identification information (ID) that identifies an object with the tag information of the object identified by the object ID, and associates the associated object ID and tag information. It has a function of managing as a tag information table DB. The tag information management unit 13 has a function of providing tag information corresponding to the object ID specified based on the object detected by the object detection unit 12 to the tag arrangement determination unit 14.

In the example shown in FIG. 1, the tag information management unit 13 is shown as a part of the video processing device 10, but may be a part of an external device separated from the video processing device 10. When the tag information management unit 13 is a part of an external device, each of the video processing device 10 and the external device includes a communication unit for performing communication between them. The communication method (communication standard) and communication means (wired communication or wireless communication, dedicated line or via network) between the video processing device 10 and the external device are not particularly limited. The tag information table DB managed by the tag information management unit 13 is a DB in which an object ID and tag information can be added and deleted by a user operation or the like.

The object detection unit 12 detects an object 41 as a target of interest such as a person included in the RAW image data D2 received from the video reception unit 11, recognizes (identifies) the detected object, and determines tag placement. Information D3 related to the object 41 is provided to the unit 14. The object is, for example, a person walking in a public area, a product moving on a production line in a factory, or the like, but is not limited thereto.

The tag arrangement determination unit 14 receives information D3 related to the object 41 detected by the object detection unit 12, and acquires tag information D4 related to the detected object 41 from the tag information management unit 13 (first processing) )I do. Next, the tag arrangement determining unit 14 performs a process (second process) for setting a search area including the detected object 41 (for example, a rectangular area surrounding the object 41). Next, the tag arrangement determination unit 14 determines whether or not there is a displayable area in which the tag information acquired in the first process can be displayed as the tag 51 in the background image area in the set search area. When it is determined that there is a displayable area, a process (third process) for determining the display position of the tag 51 within the displayable area is performed.

The tag placement determination unit 14 performs a process of changing the search area (for example, a process of expanding the search area) (fourth process) when there is no displayable area of the tag 51 in the background image area in the search area. And after the fourth process, the third process is performed again.

The tag superimposing unit 15 generates composite video data D6 for superimposing and displaying the tag 51 based on the tag information D5 related to the object 41 on the camera video displayed on the screen 31 of the display device 30.

The video output unit 16 outputs video data D7 based on the composite video data D6 to the display device 30.

FIG. 2 is a diagram illustrating a hardware configuration of the video processing apparatus 10 according to the first embodiment. As shown in FIG. 2, the video processing apparatus 10 includes, as main hardware configurations, a main processor 101 as an information processing unit, a main memory 102 as an information storage unit, a camera interface 103, and an image processing unit. An image processing processor 104 as an information processing unit, an image processing memory 105 as an information processing unit for image processing, a display interface 106, an input interface 107, and a signal path 108 to which these components 101 to 107 are connected Is provided.

The camera 20 is connected to the camera interface 103 of the video processing apparatus 10 so as to be communicable by wire or wirelessly. The camera 20 and the camera interface 103 may be connected via a network such as an IP (Internet Protocol) network, or may be connected by a dedicated communication line such as a coaxial cable. Thus, the connection method between the camera 20 and the camera interface 103 is not particularly limited.

The camera interface 103 receives video data D1 as a camera video transmitted from the camera 20. The main processor 101 can execute a tag placement program for performing image processing for displaying a tag superimposed on the camera video received by the camera interface 103.

The main memory 102 is a storage device that is directly accessed by the main processor 101. The main memory 102 holds tag information associated with the object ID. The main memory 102 is, for example, a semiconductor memory device. In addition to the main memory 102, a mass storage device such as an HDD (Hard Disk Drive) may be provided. The main processor 101 reads out and executes the tag placement program stored in the main memory 102, thereby performing processing performed by the object detection unit 12 illustrated in FIG. 1, processing performed by the tag information management unit 13, and tag placement determination unit 14 The process performed by the tag superimposing unit 15 can be executed.

The image processor 104 is a dedicated processor specialized in image processing that can execute matrix operations and floating point operations at high speed. The image processing memory 105 is a storage device that is directly accessed by the image processing processor 104. However, when the main processor 101 performs the operation of the image processing processor 104 and the main memory 102 performs the function of the image processing memory 105, it is not necessary to include the image processing processor 104 and the image processing memory 105. .

The display interface 106 is connected to the display device 30 and provides the display device 30 with a video data signal D7 for displaying the tag 51 superimposed on the camera video.

FIG. 3 is a diagram showing an example of a tag information table stored in the tag information table DB unit 17 according to the first embodiment in a table format. As shown in FIG. 3, the tag information table 17a has an object ID and tag information associated with each other for each item number (item number). The object ID is an ID (identification information) given to the object detected and recognized by the object detection unit 12.

Tag information is additional information related to the object. An example of tag information is an employee number of a company to which a person as an object belongs when the object is a person (employee) belonging to the company. Another example of tag information is a serial number of an industrial product as an object when the object is an industrial product that moves on a production line of a factory. Still another example of the tag information is an approximate age (age) and gender, which is information indicating the characteristics of a person as an object when the object is a person walking in a public area (unspecified person). Although FIG. 3 shows a tag information table in which one tag information is associated with the object ID, a tag information table in which two or more tag information is associated with the object ID may be used.

FIG. 4 is a diagram illustrating an example of object detection by the object detection unit 12 according to the first embodiment. The object detection unit 12 detects the

objects

41a, 41b, and 41c from video data (for example, RAW image data) D2 as a camera video received from the video reception unit 11. The detection of the

objects

41a, 41b, and 41c is to acquire the positions of the

objects

41a, 41b, and 41c in the RAW image data D2 received from the video receiving unit 11.

The positions of the

objects

41a, 41b, 41c are, for example, representative point coordinates representing the

objects

41a, 41b, 41c (for example, coordinates of the center positions of the

objects

41a, 41b, 41c) and the sizes of the

objects

41a, 41b, 41c ( For example, it is specified by the

rectangular areas

51a, 51b, and 51c surrounding the

objects

41a, 41b, and 41c.

When the area surrounding the

objects

41a, 41b, and 41c is a rectangular area inside the frame line that touches the outer periphery of the

objects

41a, 41b, and 41c, the size of the

objects

41a, 41b, and 41c is the

rectangular area

51a, 51b. , 51c can be expressed using the coordinates (corner coordinates) of the four vertices. However, the index indicating the size of the

objects

41a, 41b, and 41c is not limited to the four corner coordinates of the

rectangular regions

51a, 51b, and 51c. The sizes of the

objects

41a, 41b, and 41c are the corner coordinates of the two diagonal points of the

rectangular regions

51a, 51b, and 51c (for example, the upper left corner coordinates and the lower right corner coordinates). It may be represented by other indicators. Further, the area surrounding the

objects

41a, 41b, 41c is a shape other than a rectangle (for example, a triangle, a polygon having five or more corners, a circle, an ellipse, etc.) according to the shape of the

object

41a, 41b, 41c as a target of interest. It is also possible to adopt a shape of

For the detection of the

objects

41a, 41b, 41c, for example, a method of extracting a moving object from the RAW image data D2 can be used. As a method of extracting a moving object, there is a method of extracting a foreground image area using a background difference.

The background difference refers to a process of extracting an object that does not exist in the previously acquired image by comparing the observed image with the previously acquired image. In the observation image, an area occupied by an object that does not exist in the image acquired in advance is an area where the object exists, and is included in the foreground image area. However, the method for detecting the object is not limited to the background difference, and other methods may be adopted.

In the example shown in FIG. 4, three

objects

41a, 41b, and 41c are detected. The upper left corner coordinates of the rectangular area 51a of the right object 41a in FIG. 4 are (1520, 350), and the lower right corner coordinates are (1570, 450). The upper left corner coordinates of the rectangular area 51b of the center object 41b in FIG. 4 are (900, 300), and the lower right corner coordinates are (950, 400). The corner coordinates of the upper left rectangular area 51c of the left object 41c in FIG. 4 are (400, 800), and the lower right corner coordinates are (450, 900).

Further, in FIG. 4, the frame lines of the

rectangular areas

51a, 51b, 51c which are tags related to the detected three

objects

41a, 41b, 41c are displayed. The tags in FIG. 4 are frame lines that surround the

objects

41a, 41b, and 41c.

FIGS. 5A to 5C are diagrams showing another example of the tag in the first embodiment. In FIG. 5A, the number assigned to the leader line extending from the object 41d is a tag. In FIG. 5B, the number displayed in the vicinity of the object 41e is a tag. In FIG. 5C, an arrow pointing to the object displayed directly above the object 41f is a tag. The tag may be in another display format as long as the object can be highlighted on the display screen.

The object detection unit 12 acquires the positions of the

objects

41a, 41b, and 41c, recognizes the objects for each of the

objects

41a, 41b, and 41c, and positions and sizes of the

objects

41a, 41b, and 41c (for example, center coordinates); The object IDs assigned to the

objects

41a, 41b, and 41c are linked. The object detection unit 12 creates an object information list based on the position (for example, center coordinates) of the object, the size of the object, and the object ID.

FIG. 6 is a diagram illustrating an example of the object information list created by the object detection unit 12 according to the first embodiment in a table format. As illustrated in FIG. 6, the object information list includes, for example, three items of an object position (for example, center coordinates), an object size, and an object ID. The object detection unit 12 transmits the created object information list to the tag arrangement determination unit 14.

The tag arrangement determination unit 14 receives the object information list from the object detection unit 12, and acquires tag information related to the object ID from the tag information management unit 13 using the object ID described in the object information list.

The tag arrangement determination unit 14 sets an arbitrary size as a search area from the center coordinates of the object of interest detected by the object detection unit 12. The size of the search area here is not necessarily fixed. For example, the size of the

object

41a, 41b, 41c (for example, vertical width and horizontal width) is twice as large (for example, twice as large). (Vertical width and double horizontal width).

FIG. 7 is a diagram for explaining the

search areas

61a, 61b, 61c in the first embodiment. In FIG. 7, three

objects

41a, 41b, and 41c are detected, and rectangular search areas (rectangular broken lines) 61a, 61b, and 61c including the three

objects

41a, 41b, and 41c are set. In the example shown in FIG. 7, the horizontal width of the

search areas

61a, 61b, 61c is about three times the horizontal width of the

objects

41a, 41b, 41c, and the vertical width of the

search areas

61a, 61b, 61c is the object 41a, It is about three times the vertical width of 41b and 41c.

The tag arrangement determining unit 14 distinguishes the foreground image area and the background image area from the set

search areas

61a, 61b, and 61c, and searches for an area where the tag can be arranged from the area recognized as the background image area. The size of the tag associated with the object is determined in advance by the tag placement determination unit 14. The size of the tag may be fixed as specified by the user, or may be variable according to the information amount of the tag. Here, a case will be described in which the size of the tag is a predetermined fixed size.

When the tag placement determining unit 14 finds an area in which the tag can be placed in the

search areas

61a, 61b, and 61c, the tag placement position is within the tag placeable area. Set. The area occupied by the tag set by the tag arrangement determination unit 14 is recognized as a foreground image area. If no area for tag placement is found in the

search areas

61a, 61b, 61c, the tag placement determination unit 14 expands the search area.

FIG. 8 is a diagram for explaining the enlarged search area 71b in the first embodiment. For example, in the search area 61b of the center object 41b in FIG. 8, when the area where the tag can be arranged is not found, the tag arrangement determining unit 14 enlarges the search area 61b in the object 41b in FIG. 71b (dashed rectangular frame) is set. As shown in FIG. 8, the enlarged search area 71b is set in a wider range than the search area 61b so as to surround the search area 61b before the enlargement.

The tag placement determination unit 14 sends a tag placement list having an object ID, corresponding tag information, and coordinates for placing tags (upper left corner coordinates and lower right corner coordinates) to the tag superposition unit 15.

The tag superimposing unit 15 superimposes tag information on the camera video passed from the video receiving unit 11 using the tag arrangement list received from the tag arrangement determining unit 14. The tag superimposing unit 15 superimposes tag information in the tag arrangement list, and transmits a video in which the tag information is superimposed on the camera video to the video output unit 16.

The video output unit 16 receives the video data D6 from the tag superimposing unit 15 and outputs the video data D7 based on the video data D6 to the display device 30.

<< 1-2 >> Operation FIG. 9 is a flowchart showing the operation of the video receiver 11 in the first embodiment. As shown in FIG. 9, the video receiver 11 receives video data D1 that is a camera video transmitted from the camera 20 (step S10). Next, when the video data D1 is compression-encoded data, the video reception unit 11 decodes the video data D1 and outputs RAW image data D2 that is decoded video data (step S11). .

For example, from camera 20 to H.264. When the video data D1 compression-encoded in the H.264 format is streamed, the video receiving unit 11 By performing decoding corresponding to video data in the H.264 format, for example, RGBA (Red, Green, Blue, Alpha) 32-bit RAW image data D2 is supplied to the object detection unit 12 and the tag superposition unit 15. The decoded RAW image data D2 is not limited to RGBA 32-bit RAW image data, and may be image data of other formats as long as it is uncompressed image data.

FIG. 10 is a flowchart showing the operation of the tag information management unit 13 in the first embodiment. As shown in FIG. 10, the tag information management unit 13 acquires tag information associated with the object ID designated by the tag placement determination unit 14 from the tag information table DB unit 17 (step S20), and the object information Information D4 including the ID and the acquired tag information is provided to the tag arrangement determining unit 14 (step S21).

FIG. 11 is a flowchart showing the operation of the object detection unit 12 in the first embodiment. As illustrated in FIG. 11, the object detection unit 12 performs an object detection process from the RAW image data D2 output from the video reception unit 11 (step S30). In the object detection processing, for example, the upper left corner coordinates and the upper right corner coordinates are acquired as the object center coordinates and the object size.

When one or more objects are detected in the object detection process (YES in step S31), the object detection unit 12 recognizes the detected object (step S32). If no object is detected (NO in step S31), the object detection unit 12 ends the process and notifies the tag placement determination unit 14 of the end of the process.

Object recognition (step S32) is generally performed using a machine learning technique for image analysis of an object (for example, a person, an animal, an object, etc.) shown in an image. The object recognition method is not particularly limited. Subsequently, the object detection unit 12 associates an object ID with the recognized object (step S33). The object detection unit 12 provides the tag placement determination unit 14 with an object information list (FIG. 6) having the object center coordinates, the object size, and the object ID as items (step S34).

FIG. 12 is a flowchart showing the operation of the tag placement determination unit 14 in the first embodiment. As shown in FIG. 12, the tag arrangement determining unit 14 repeats the processes of steps S40 to S47 by the number of objects using the object information list (FIG. 6) output from the object detecting unit 12.

First, the tag arrangement determination unit 14 acquires the tag information of the corresponding object ID from the tag information management unit 13 (step S40).

Next, the tag placement determination unit 14 receives the tag information of the corresponding object ID from the tag information management unit 13, and searches the region for placing a tag for one object ID (

regions

61a, 61b, 61 in FIG. 7). 61c etc.) are set (step S41). The size of the search area at this time is a size designated in advance by the user, but an area having a length twice the size of the object may be used as the search area. As described above, the size of the search area is not necessarily fixed, and may be changed according to the size of the object.

Next, the tag arrangement determination unit 14 distinguishes the background image region and the foreground image region from the set search region (step S42). Here, the background image area means an area other than the foreground image area. The foreground area includes an area occupied by the object, an area occupied by another object, and a tag area of another object. A background model is constructed by acquiring multiple images in advance, and the changed area is defined as the foreground area by comparing the background model with the newly observed image, and the other areas are defined as the background area. It is also possible to do.

Referring to FIG. 4, the tag area of the corresponding object and the other object is the foreground image area, and the other part is the background image area. Here, the foreground picture area is not limited to the corresponding object, other objects, and the tag area of the object. For example, an image to be superimposed on a camera image can be set as a foreground image area.

Next, the tag arrangement determination unit 14 searches the extracted background image area for an area in which tags (for example, 51d, 51e, 51f in FIGS. 5A to 5C) can be arranged (step S43). The size of the tag placed by the tag placement determination unit 14 can be made variable depending on the number of characters in the tag information, and can also be a fixed size. As a search method for a region in which a tag can be placed in the background image region, the tag placement region may be sequentially searched using the upper left of the search region as the origin. The search for the area where the tag can be arranged may be started from a predetermined position.

For example, the search area is divided into a plurality of grid areas arranged in a grid of an arbitrary size, and it is determined whether each of the plurality of grid areas is a background image area or a foreground image area. Based on this, an area where the tag can be arranged may be found. At that time, a search procedure for searching from which lattice region of the plurality of lattice regions is possible is not particularly limited. When a tag placement area is found in the set search area (YES in step S44), the tag placement determination unit 14 outputs the tag placement position and tag information to the tag placement list (step S45).

If no tag placement area is found in the search area (NO in step S44), the tag placement determination unit 14 enlarges the search area and resets the search area for the object (setting of the enlarged search area) (Step S46). If the search area can be reset (YES in step S47), the process returns to step S42 to distinguish the foreground image area and the background image area in the reset search area. The enlargement size of the search area may be fixed (a predetermined size), or may be dynamically changed according to the size of the object or the size of the search area before enlargement. If the search area cannot be reset (NO in step S47), the tag placement determining unit 14 determines whether there is another object for which tag information has not been acquired, and if there is (YES in step S48). ), The process returns to step S40, and if not (NO in step S48), tag information is output to the tag superimposing unit 15, and the tag arrangement determining unit 14 ends the operation.

FIG. 13 is a flowchart showing the operation of the tag superimposing unit 15 in the first embodiment. As shown in FIG. 13, the tag superimposing unit 15 superimposes tag information on the RAW image data D2 output from the video receiving unit 11 using the tag arrangement list output from the tag arrangement determining unit 14 (step S50).

When the positions of the object and the tag information are separated from each other, as shown in FIG. 5A, the object and the tag may be connected and displayed in order to show the mutual relationship. The tag superimposing unit 15 outputs the image data D6 on which the tag information is superimposed to the video output unit 16. The video output unit 16 outputs the image data D7 obtained from the tag superimposing unit 15 to the display device 30.

<< 1-3 >> Effect As described above, according to the video processing device 100, the video processing method, the video processing program, and the video monitoring system according to the first embodiment, in the camera video displayed on the display device 30 Since the tag 51 as additional information related to the object 41 can be arranged in the background image area, the tag is not set so as to overlap the other object and the tag of the other object. For this reason, there is an effect that monitoring of the object becomes easy.

<< 2 >> Embodiment 2
The video processing apparatus 10 according to the first embodiment detects the position of the object 41 as a subject from the RAW image data D2 based on the video data D1 output from the camera 20, and uses the center coordinates of the object 41 as a reference position as a search area. And the arrangement position of the tag 51 is determined in the background image area in the search area.

However, when an object moves and a tag related to the object moves following the object, a plurality of objects overlap each other, or a plurality of tags approach each other or overlap on the screen 31 of the display device 30. Sometimes. As described above, when a plurality of objects are close to each other or overlap each other, a monitor watching the screen 31 has difficulty in understanding a set of related objects and tags.

Therefore, the video processing apparatus according to Embodiment 2 has a function of displaying the colors of a plurality of tags in different colors. Furthermore, the video processing apparatus according to the second embodiment may include a function capable of attaching a mark having the same color as the associated tag color on each of the plurality of objects. Note that FIG. 1 is also referred to in the description of the second embodiment.

FIG. 14 is a diagram illustrating an example of the tag information table 17b stored in the tag information table DB unit 17 of the video processing apparatus according to the second embodiment. As shown in FIG. 14, the video processing apparatus according to the second embodiment having the tag information table 17b has display color information regarding the display color for each object ID, and the tag placement determination unit 14 is based on the display color information. This is different from the video processing apparatus 10 according to the first embodiment in that the display color of the tag is designated.

15 (a) to 15 (c) are diagrams for explaining the effect of the video processing apparatus according to the second embodiment. As shown in FIG. 15A, when the object (person) 41g and the other object (other person) 41h are sufficiently separated from each other, the tag 51g related to the object 41g and the other object 41h are related. Since the tags 51h are sufficiently separated from each other, the supervisor can easily grasp the object 41g with reference to the tag 51g and can easily grasp the object 41h with reference to the tag 51h. However, when the other object 41h moves in the direction of the white arrow (leftward in FIG. 15A) and approaches the object 41g, the tag 51g and the tag 51h are related to either the

object

41g or 41h. It becomes difficult to understand on the screen.

At this time, as shown in FIG. 15B as a comparative example, if it is assumed that the tag 51g and the tag 51h have the same display color, the monitor can grasp either of the two tags when grasping the

objects

41g and 41h. It is not easy to determine whether to refer to.

Therefore, in the video processing device according to the second embodiment, as shown in FIG. 14, the tag information table 17b has display color information related to the display color for each object ID, and the tag arrangement determining unit 14 uses the tag 51g. And the display color of 51h are designated. For this reason, in the second embodiment, even when the

objects

51g and 51h approach each other, the

Claims

A video processing device that causes additional information related to an object as a target of interest present in a camera video acquired by camera shooting to be superimposed and displayed as a tag on the camera video on a display device,
The area occupied by the object and the area occupied by the tag in the camera video is a foreground image area, and an area other than the foreground image area in the camera video is a background image area,
An object detection unit for detecting the object;
A tag information management unit for managing a database for storing the additional information;
A first process for acquiring the additional information related to the object detected by the object detection unit from the tag information management unit; and a second process for setting a search area including the object detected by the object detection unit. And when there is a displayable area in which the tag as the additional information acquired in the first process can be displayed in the background image area in the search area, the display position of the tag is A tag placement determination unit that performs a third process of determining within the displayable area;
And a tag superimposing unit that generates composite video data for superimposing and displaying the tag on the camera video.
The tag arrangement determination unit performs a fourth process of setting an enlarged search area larger than the search area when the displayable area does not exist in the background image area in the search area, and The video processing apparatus according to claim 1, wherein the third process is executed again after the process.
The video processing apparatus according to claim 1, wherein the object detection unit detects the object by image analysis of the camera video.
A sensor that detects the object and outputs a detection signal;
The video processing apparatus according to claim 1, wherein the object detection unit detects the object based on the detection signal.
The tag information management unit provides the tag arrangement determination unit with the additional information acquired from a database storing identification information and additional information of the objects related to each other. The video processing device according to any one of claims.
The tag information management unit provides the tag arrangement determination unit with the additional information and the display color information acquired from the database that stores identification information, additional information, and display color information of the objects related to each other.
The video processing apparatus according to claim 1, wherein the tag arrangement determination unit displays the tag with a color based on the display color information.
The tag information management unit provides the tag arrangement determination unit with the additional information and the priority information acquired from the database storing the identification information, additional information, and priority information of the objects related to each other,
The tag placement determination unit, when a plurality of tags overlap, displays a tag related to a higher priority among the plurality of tags in the foreground. The video processing apparatus according to the item.
A video processing method for superimposing additional information related to an object as a target of interest present in a camera video acquired by camera shooting as a tag on the camera video on a display device,
The area occupied by the object and the area occupied by the tag in the camera video is a foreground image area, and an area other than the foreground image area in the camera video is a background image area,
A detecting step of detecting the object;
A first process for acquiring the additional information related to the object detected by the detection step from a database managing the additional information; and a first region for setting a search area including the object detected by the detection step. 2 and when there is a displayable area in which the tag as the additional information acquired in the first process can be displayed in the background image area in the search area, the display position of the tag An arrangement determining step for performing a third process for determining within the displayable area;
And a superimposing step of generating composite video data for superimposing and displaying the tag on the camera video.
A video processing program for causing a computer to superimpose and display additional information related to an object as a target of interest present in a camera video acquired by camera shooting as a tag on the camera video on a display device,
The area occupied by the object and the area occupied by the tag in the camera video is a foreground image area, and an area other than the foreground image area in the camera video is a background image area,
In the computer,
A detecting step of detecting the object;
A first process for acquiring the additional information related to the object detected by the detection step from a database managing the additional information; and a first region for setting a search area including the object detected by the detection step. 2 and when there is a displayable area in which the tag as the additional information acquired in the first process can be displayed in the background image area in the search area, the display position of the tag An arrangement determining step for performing a third process for determining within the displayable area;
And a superimposing step for generating composite video data for superimposing and displaying the tag on the camera video.
A camera that generates a camera image by shooting a subject;
A display device for displaying video based on video data;
A video processing device that superimposes and displays additional information related to an object as a target of interest existing in the camera video as a tag on the camera video in the display device;
The area occupied by the object and the area occupied by the tag in the camera video is a foreground image area, and an area other than the foreground image area in the camera video is a background image area,
The video processing device includes:
An object detection unit for detecting the object;
A tag information management unit for managing a database for storing the additional information;
A first process for acquiring the additional information related to the object detected by the object detection unit from the tag information management unit; and a second process for setting a search area including the object detected by the object detection unit. And when there is a displayable area in which the tag as the additional information acquired in the first process can be displayed in the background image area in the search area, the display position of the tag is A tag placement determination unit that performs a third process of determining within the displayable area;
And a tag superimposing unit that generates composite video data for superimposing and displaying the tag on the camera video.