JP7218786B2 - Image processing device, image processing method and program - Google Patents

Description

The present technology relates to an image processing device, an image processing method, and a program, and more particularly to an image processing device and the like that processes moving image data obtained by imaging a state described in a description section by a writer.

2. Description of the Related Art Conventionally, several proposals have been made for improving the visibility of characters and the like in image display methods in image distribution. For example, Patent Literature 1 discloses a method of extracting a character portion from an image and superimposing only that portion with a high-resolution or character-recognized text result to improve visibility. This is based on the assumption of image distribution at a low bit rate, and it is stated that it is necessary to improve the visibility of the information in the character portion.

Further, Patent Document 2 discloses a method of recognizing characters in an image, specifying the character color and the background color, and then adjusting the character color and the background color for display. This is also one of the techniques for improving the visibility of characters.

The methods disclosed in Patent Documents 1 and 2 are methods for areas that are always visible in the image but have low visibility, such as low bit rates and colors similar to the background and character colors. It is not effective when the visibility is lowered because the characters are hidden.

JP 2003-179888 A JP 2015-069234 A

An object of the present technology is to ensure the visibility of the description on the description section that is hidden by the writer while maintaining the positional relationship between the writer and the description on the description section.

The concept of this technology is
a description determination processing unit that processes moving image data obtained by capturing a state in which a description is written in a description section and determines a description part;
The image processing apparatus includes a description superimposition processing unit that generates display data for displaying each portion determined as the description portion as a description and superimposes the display data on the moving image data.

In the present technology, the description determination processing unit processes the moving image data obtained by capturing the state in which the description is written in the description section, and determines the description part. For example, the description determination processing unit extracts the difference value for each pixel between the current frame image and the reference frame image, grasps groups in which pixels having the difference value greater than or equal to a threshold are continuous, and for each group, It may be determined whether or not it is a description part.

In this case, for example, the description determination processing unit uses the shape evaluation value, the extraction time evaluation value, and the still time evaluation value for each group to determine whether or not it is a description portion. A value is determined based on the aspect ratio of the smallest rectangle that contains the chunk and the largest rectangle that is contained inside the chunk, and the extraction time estimate is calculated from the current frame and the previous multiple for the pixels contained in the chunk. Based on the difference value between the reference frame image in the frame and the still time evaluation value for the pixels included in the group, the current frame and the previous frame image in a plurality of frames before the current frame It may be determined based on the difference value between.

The description superimposition processing unit generates display data for displaying each portion determined as a description portion as a description and superimposes it on the moving image data. For example, the description superimposition processing unit may perform processing for enhancing the visibility of the description when superimposing the display data on the moving image data.

In this case, for example, the description superimposition processing unit may superimpose the display data on the moving image data after performing processing to reduce the resolution of the moving image data. Also, in this case, for example, the description superimposition processing unit processes the display data so that each portion determined to be the description portion is displayed with borders, and then superimposes this display data on the moving image data. may be Further, in this case, for example, the description superimposition processing unit processes the display data based on the color information of the moving image data so that the color of each portion determined to be the description portion is different from the color of the background image, and then performs this display. The data may be superimposed on the moving image data.

As described above, in the present technology, display data for displaying each portion determined as a description portion as a description is generated and superimposed on the moving image data. Therefore, it is possible to ensure the visibility of the description on the description section that is hidden by the writer while maintaining the positional relationship between the writer and the description on the description section.

Note that, in the present technology, description superimposition control for controlling the description superimposition processing unit so as not to superimpose the display data on the video data when it is detected that the face of the writer is facing forward is based on the video data. A part may be further provided. When the narrator stops and begins to give an oral explanation or commentary, it is thought that the facial expressions and gestures of the narrator are more important as information than the description. It is possible to display without being disturbed.

Further, in the present technology, for example, an imaging unit that captures an image of a state written by a writer in a description part, and an imaging signal processing unit that processes an imaging signal obtained by the imaging unit to obtain moving image data are further added. may be provided. Further, in the present technology, for example, a transmission unit that transmits the moving image data obtained by the description superimposition processing unit to an external device may be further provided.

According to the present technology, it is possible to ensure the visibility of the description on the description section that is hidden by the writer while maintaining the positional relationship between the writer and the description. Note that the effects described in this specification are merely examples and are not limited, and additional effects may be provided.

1 is a block diagram showing a configuration example of an image processing system as an embodiment; FIG. It is a figure which shows an example of the captured image of a certain frame with a camera. 1 is a block diagram showing a configuration example of an image processing apparatus; FIG. 4 is a flowchart for explaining processing in a description determination processing unit; It is a figure for demonstrating how to obtain|require the value of shape evaluation. FIG. 4 is a diagram showing an example of a display image displayed on the display of the display device; FIG. 12 is a block diagram showing another configuration example of the image processing apparatus; FIG. 10 is a diagram showing an example of a display image displayed on the display of the display device when the writer is facing forward; It is a block diagram which shows the structural example of a camera.

DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, modes for carrying out the invention (hereinafter referred to as "embodiments") will be described. The description will be made in the following order.
1. Embodiment 2. Modification

<1. Embodiment>
[Configuration example of image processing system]
FIG. 1 shows a configuration example of an image processing system 10 as an embodiment. This image processing system 10 has a camera 101, an image processing device 102, and a predetermined number of display devices 103-1 to 103-N.

The camera 101 obtains moving image data by capturing a state in which a description 201 is added to a description part 202 while adding a description 203 . FIG. 2 shows an example of an image of a certain frame captured by the camera 101 . In this captured image, the description 203 exists above the description part 202, and the description 201 exists in front of it. As the writer 201 moves, gestures, and gestures, the part of the description 203 that is hidden by the writer 201 in this frame can also be imaged by the camera 101 in the subsequent frame.

As the writer 201, for example, a lecturer such as a professor who gives a lecture to students in a lecture room of a university, an employee who gives a presentation to other employees in a conference room of a company, and the like are assumed. The description part 202 is a part such as a whiteboard or a blackboard that can be described, or it may be a wall part of a lecture room or a conference room that can be described.

The image processing device 102 processes the moving image data obtained by the camera 101 to determine the description portion, generates display data for displaying each portion determined as the description portion as a description, and converts the display data into the moving image. The moving image data superimposed on the data and superimposed with the display data is distributed to the display devices 103-1 to 103-N. Distribution is performed wirelessly or by wire.

The display devices 103-1 to 103-N receive moving image data superimposed with the display data from the image processing device 102, and display images based on the moving image data. In this case, the description is superimposed on the image displayed on the display. Therefore, the visibility of the description 203 on the description section 202 hidden by the writer 201 is ensured while maintaining the positional relationship between the writer 201 and the description 203 .

The display devices 103-1 to 103-N may be arranged in a room such as a lecture room or conference room where the writer 201 is currently giving an explanation, another lecture room or conference room, or even a remote location. may If the display devices 103-1 to 103-N are arranged relatively close to the image processing device 102, a wireless LAN such as Wi-Fi can be used. On the other hand, when the display devices 103-1 to 103-N are located at remote locations, distribution via the Internet can be considered.

Delivery of moving image data from the image processing device 102 to the display devices 103-1 to 103-N does not necessarily have to be performed live. For example, moving image data obtained by the image processing device 102 may be recorded and stored in a recording unit, reproduced at appropriate timing, and distributed to the display devices 103-1 to 103-N.

Details of the image processing apparatus 102 will be described. The image processing apparatus 102 extracts the difference value for each pixel between the current frame image and the reference frame image, grasps groups in which pixels whose difference values are equal to or greater than a threshold are continuous, and writes the description part for each group. Determine whether or not Specifically, the image processing apparatus 102 uses the shape evaluation value, the extraction time evaluation value, and the stationary time evaluation value for each group to determine whether or not it is a description part.

The image processing apparatus 102 obtains the shape evaluation value from the relationship between the aspect ratios of the minimum quadrilateral that includes the group and the maximum quadrilateral that is included inside the group. The image processing apparatus 102 also obtains the extraction time evaluation value based on the difference value between the reference frame images in the current frame and a plurality of previous frames for the pixels included in the group. Further, the image processing device 102 obtains the static time evaluation value based on the difference value between the current frame and the previous frame image in the plurality of frames before the current frame for the pixels included in the group.

Further, the image processing device 102 performs processing for enhancing the visibility of the description when superimposing the display data on the moving image data. For example, the image processing apparatus 102 superimposes the display data on the moving image data after performing processing to reduce the resolution of the moving image data. Also, for example, the image processing apparatus 102 processes the display data so that each portion determined to be the descriptive portion is displayed with borders, and then superimposes this display data on the moving image data. Further, for example, the image processing apparatus 102 processes the display data based on the color information of the moving image data so that the color of each portion determined to be the description portion is different from the color of the background image, and then converts the display data into a moving image. superimposed on the image data.

FIG. 3 shows a configuration example of the image processing apparatus 102. As shown in FIG. The image processing apparatus 102 includes a control unit 121, an image data input unit 122, a description determination processing unit 123, a display data generation unit 124, a superimposition unit 125, an image compression processing unit 126, a network interface 127, It has a recording unit 128 .

A control unit 121 controls the operation of each unit of the image processing apparatus 102 . The image data input unit 122 is a part for inputting moving image data obtained by the camera 101 . For example, the image data input unit 122 may be a cable connect unit that connects with the camera 101 . Further, for example, when the image data input unit 122 is connected to the camera 101 through communication, a wireless or wired communication interface may be used.

The description determination processing unit 123 processes the moving image data input to the image data input unit 122 to determine the description part. The description determination processing unit 123 executes the processing from the start to the end of the flowchart of FIG. 4 for each frame by software based on a program, for example.

First, in step ST1, the description determination processing unit 123 performs pixel-by-pixel comparison between the image of a certain frame and the reference frame image (first, the frame image in the state before the description is performed in the description unit 202). A difference is extracted, and from the difference extraction result, pixels with large pixel value differences, that is, pixels that have changed from the state of the reference frame image are extracted, and groups (predetermined number) of such pixels are grasped.

In this way, the group grasped in step ST1 includes the description 203 portion, but also includes the description person 201 and the like. In the following steps, it is determined whether or not it is the part of the description 203 for each group. The values of the shape evaluation, the extraction time evaluation, and the stationary time evaluation are used for this determination.

Next, the description determination processing unit 123 obtains a shape evaluation value α in step ST2. The description determination processing unit 123 obtains this value α based on the aspect ratio relationship between the minimum quadrangle (external quadrangle) that includes the group and the maximum quadrangle (internal quadrangle) that is included inside the group. FIG. 5(a) shows an example of a summary corresponding to the description 203. The quadrangle indicated by arrow Po is the outer quadrangle, and the quadrangle indicated by arrow Pi is the inner quadrangle. Also, FIG. 5(b) shows an example of a grouping corresponding to the part of the writer 201, the quadrangle indicated by the arrow Qo is the outer quadrangle, and the quadrangle indicated by the arrow Qi is the inner quadrangle.

In the case of a grouping corresponding to the part of the description 203, the inner quadrangle (Pi) is smaller than the outer quadrangle (Po), so the shape evaluation value α is high. On the other hand, in the case of the grouping corresponding to the part of the description 201, the grouping corresponding to the part of the description 203 is larger than that of the grouping corresponding to the part of the description 203. Therefore, the size of the inner quadrangle (Qi) is the same as that of the outer quadrangle (Qo). have. Therefore, the shape evaluation value α becomes low.

Next, the description determination processing unit 123 obtains the extraction time evaluation value β in step ST3. The description determination processing unit 123 obtains the value β based on the difference values between the current frame and the reference frame images in a plurality of previous frames for the pixels included in the group. The higher the number of frames with large , the higher the value.

For example, the description determination processing unit 123 accumulates the results of whether or not the pixels included in the group (all pixels or some pixels) are frames (difference extraction frames) having a large difference from the reference frame image. Let the number of consecutive difference extraction frames (for example, the average value when this evaluation is performed for a plurality of pixels) be the evaluation value β.

In the case of the description 203, since a large difference continues to be extracted after the description, the evaluation value β becomes high. On the other hand, in the case of the part of the moving writer 201, extraction of a large difference is interrupted when the writer 201 has completely passed, so the evaluation value β becomes low. However, when the writer 201 does not move and stops, the evaluation value β may become high even though it is the part of the writer 201 .

Next, the description determination processing unit 123 obtains the stationary state evaluation value γ in step ST4. The description determination processing unit 123 obtains this value γ based on the difference value between the current frame and the previous frame image in a plurality of frames before the current frame for the pixels included in the group. , the greater the number of frames with large difference values, the lower the value.

For example, the description determination processing unit 123 accumulates pixel differences from one frame before for pixels included in the group (all pixels or some pixels), and the reciprocal of the magnitude of the difference value (this evaluation is performed for a plurality of pixels, the average value) is used as the evaluation value γ.

In the case of description 203, the pixel difference value with respect to the image one frame before is zero or small, so the evaluation value γ is high. On the other hand, in the case of the part of the writer 201 moving or gesturing, the pixel difference value from the image of one frame before is large, so the evaluation value γ is low.

Next, in step ST5, the description determination processing unit 123 substitutes the evaluation values α, β, and γ obtained in steps ST2 to ST4 into the evaluation function E (α, β, γ), and determines whether the description is correct or not. Obtain an evaluation value for determining whether Further, in step ST5, the description determination processing unit 123 determines that the grouping is the description part (the part of the description 203) when the evaluation value is greater than the threshold value THr.

Next, at step ST6, the description determination processing unit 123 basically additionally holds each pixel included in the group determined to be the description portion at step ST5 as a pixel of the description portion. However, if each pixel contained in a group determined to be a description part is a pixel that was previously held as a description part, it corresponds to a part that has been erased after that, so it cannot be used as a pixel of the description part. Remove from hold.

Here, holding means, for example, writing and holding the value of each pixel included in the group determined to be the description portion in the corresponding pixel position of the frame memory. Instead of writing the value of each pixel included in the group determined to be the description part into the corresponding pixel position of the frame memory, only the pixel position information of each pixel included in the group determined to be the description part is stored. It may be configured to be held.

Next, in step ST7, the description judgment processing unit 123 sets the value of each pixel currently held as the description part to the value corresponding to the description 203 based on the result of holding or deletion in step ST6. An image is generated, and the frame image is used as a new reference frame image. The updated reference frame image is used in the description determination process for the next frame.

Returning to FIG. 3, the display data generation unit 124 displays each pixel as a description part based on the pixel information of the description part updated by holding or deleting processing in the description determination processing unit 123 in each frame. Generates display data to be displayed.

The superimposing unit 125 superimposes the display data generated by the display data generating unit 124 on the moving image data input to the image data input unit 122 . As a result, the description 203 described above the description section 202 can be superimposed and displayed on the image based on the moving image data.

In this case, the superimposing unit 125 performs, for example, part or all of the following processing in order to enhance the visibility of the description 203 superimposed on the image.

For example, the superimposing unit 125 superimposes the display data on the moving image data after performing processing for lowering the resolution of the moving image data, for example, processing for removing high frequency components using a spatial low-pass filter. As a result, the image based on the moving image data becomes blurred, and the visibility of the description 203 superimposed on the image can be improved.

Also, for example, the superimposing unit 125 processes the display data so that each portion determined to be the descriptive portion is displayed with borders, and then superimposes this display data on the moving image data. Note that the bordering process may be performed by the display data generator 124 .

For example, if the original color of the description part is black, the bordering process is performed with white, and conversely, if the original color of the description part is white, the bordering process is performed with black. The border color is not limited to this. By edging in this way, for example, when the original color of the description part is similar to the color of the clothes of the writer 201, the visibility of the description 203 superimposed on the image based on the moving image data is enhanced. be able to.

Further, for example, the superimposing unit 125 performs color change processing of the display data based on the color information of the moving image data so that the color of each portion determined as the description portion is different from the color of the background image (image based on the moving image data). After that, this display data is superimposed on the moving image data. Note that this color change processing may be performed by the display data generator 124 . By changing the color of the display data in this way, for example, if the original color of the description part resembles the color of the clothes of the writer 201, the description 203 is displayed superimposed on the image of the moving image data. visibility can be improved.

The image compression processing unit 126 performs data compression encoding processing using, for example, the MPEG method on the moving image data on which the display data has been superimposed by the superimposition unit 125 to generate encoded image data. The network interface 127 transmits the encoded image data generated by the image compression processing unit 126 to the display devices 103-1 to 103-N via a wireless or wired network. Note that the encoded image data generated by the image compression processing unit 126 is temporarily recorded in the recording unit 128, and the network interface 127 reads out the encoded image data from the recording unit 128 at appropriate timing, and displays the data on the display device. 103-1 through 103-N.

The operation of the image processing system 10 shown in FIG. 1 will be briefly described. The camera 101 captures the state in which the writer 201 is adding the description 203 to the description part 202 and provides moving image data. This moving image data is supplied to the image processing device 102 .

In the image processing device 102, the moving image data from the camera 101 is processed to determine the description portion, and display data for displaying each portion determined as the description portion as a description is generated. superimposed. Moving image data (encoded image data) superimposed with display data is distributed from the image processing device 102 to the display devices 103-1 to 103-N through a wireless or wired network.

The display devices 103-1 to 103-N perform image display based on the moving image data received from the image processing device 102. FIG. In this case, the image displayed on the display is the one in which the description is superimposed on the image based on the moving image data.

Next, the operation of the image processing apparatus 102 shown in FIG. 3 will be briefly described. Moving image data captured by the camera 101 is input to the image data input unit 122 . This moving image data is supplied to the superimposing unit 125 and the description determination processing unit 123 .

The description judgment unit 123 processes the moving image data, judges the description part for each frame, and stores it as the pixels of the description part. Determination of whether or not it is a descriptive portion is performed by extracting a difference value for each pixel between the current frame image and the reference frame image, and performing shape evaluation and extraction for each group in which pixels whose difference value is equal to or greater than a threshold are continuous. It is done using the values of the Time Estimate and the Quiet Time Estimate.

In this case, the portion of the description 203 that is hidden by the writer 201 during a certain period becomes unhidden by the writer 201 during another period as the writer 201 moves, makes gestures, and so on. After a certain amount of time has passed, most of the portion of the description 203 written in the description portion 202 is retained as pixels of the description portion.

The pixel information of the description part updated for each frame by the description determination processing unit 123 is sent to the display data generation unit 124 . The display data generation unit 124 generates display data for displaying each pixel as a description part based on the pixel information of the description part in each frame. This display data is supplied to the superimposing unit 125 .

The superimposing unit 125 superimposes the display data generated by the display data generating unit 124 on the moving image data input to the image data input unit 122 . In this case, the superimposing unit 125 performs processing for enhancing the visibility of the description 203 superimposed and displayed on the image, for example, processing for lowering the resolution of the moving image data, and bordering each portion determined to be the description portion. A process of processing the display data so that it is displayed, a process of changing the color of the display data so that the color of each part judged to be a description part is different from the color of the background image (image based on moving image data), etc. .

The moving image data superimposed with the display data is supplied from the superimposing unit 125 to the image compression processing unit 126 . In the image compression processing unit 126, the moving image data on which the display data is superimposed is subjected to data compression encoding processing using, for example, the MPEG method, to generate encoded image data.

This encoded image data is directly supplied to the network interface 127, or is temporarily recorded in the recording unit 128, reproduced at appropriate timing, and supplied to the network interface 127. FIG. The network interface 127 transmits encoded image data to the display devices 103-1 to 103-N through a wireless or wired network.

FIG. 6 shows an example of display images displayed on the displays of the display devices 103-1 to 103-N. In this display image, a description 203 based on display data is displayed superimposed on an image based on moving image data in which a writer 201 exists in front of a description portion 202 . Also, in this example, a portion of the description 203 is shown outlined. Therefore, the visibility of the description 203 on the description section 202 hidden by the writer 201 is ensured while maintaining the positional relationship between the description 201 and the description 203 on the description section 202 .

As described above, in the image processing system 10 shown in FIG. 1, the image processing device 102 generates display data for displaying each portion determined as a description portion as a description and superimposes the display data on the moving image data. Therefore, the visibility of the description 203 on the description section 202 that is hidden by the writer 201 can be ensured while maintaining the positional relationship between the description 201 and the description 203 on the description section 202 .

In the image processing system 10 shown in FIG. 1, the image processing apparatus 102 performs processing for enhancing the visibility of the description 203 superimposed and displayed on the image when the display data is superimposed on the moving image data. Processing to reduce the resolution of moving image data, processing to process display data so that each portion determined to be a descriptive portion is bordered and displayed, and processing to change the color of each portion determined to be a descriptive portion to a background image (moving image data). It performs processing such as changing the color of the display data so that it differs from the color of the image (image based on data). Therefore, the visibility of the description 203 superimposed on the image can be improved.

<2. Variation>
In the above-described embodiment, the description 203 is always displayed superimposed on the moving image data regardless of the state of the writer 201 . However, when the writer 201 begins to speak, for example, while facing the front, the lecture attendees and the chairman of the conference may be expected to pay attention to every move of the writer 201. The description 203 may not be superimposed on the data image.

FIG. 7 shows a configuration example of the image processing apparatus 102 in that case. 7, parts corresponding to those in FIG. 3 are denoted by the same reference numerals, and detailed description thereof will be omitted. The face recognition processing unit 129 processes the moving image data input to the image data input unit 122, recognizes the face of the writer 201, and detects whether the writer 201 is facing forward. , the detection result is sent to the control unit 121 .

When the writer 201 is not facing the front, the control section 121 operates the description judgment processing section 123, the display data generating section 124 and the superimposing section 125 in the same manner as in the above embodiments. On the other hand, when the writer 201 is facing the front, the control section 121 temporarily suspends the operations of the description judgment processing section 123 and the display data generation section 124, and the superimposition section 125 inputs image data. The moving image data input to the unit 122 is output as it is. Note that the operation of the description judgment processing unit 123 may be continued even when the writer 201 faces the front.

Other configurations and operations of the image processing apparatus 102 shown in FIG. 7 are the same as those of the image processing apparatus 102 shown in FIG. 3 described above.

FIG. 8 shows an example of a display image displayed on the displays of the display devices 103-1 to 103-N when the writer 201 faces the front. In this display image, the description 203 based on the display data is not superimposed on the image based on the moving image data in which the writer 201 exists on the front side of the description part 202 . As a result, lecture attendees and conference chairpersons can pay attention to every move of the writer 201 through the display screen.

It should be noted that preventing the description 203 from being superimposed on the moving image data is not limited to when the description 201 is facing the front, and may be performed in other cases. . In the image processing apparatus 102 of FIG. 7, the face recognition processing unit 129 detects whether or not the writer 201 is facing the front, and based on the detection result, the description 203 is displayed on the image based on the image data. Avoid overlapping display. However, it is conceivable to artificially determine whether or not to superimpose the description 203 on an image based on moving image data.

Also, in the above embodiment, the image processing device 102 is provided separately from the camera 101 . However, a configuration in which the function of the image processing device 102 is provided inside the camera 101 is also conceivable.

FIG. 9 shows a configuration example of the camera 101 in that case. The camera 101 has a control section 111 , an imaging section 112 , an imaging signal processing section 113 and an image data output section 114 . A control unit 111 controls the operation of each unit of the camera 101 . The imaging unit 112 has an imaging lens and an imaging device (not shown), captures an image, and outputs an imaging signal. The imaging element is an imaging element such as a CCD (Charged Coupled Device), a CMOS (Complementary Metal-Oxide Semiconductor), or the like.

The image pickup signal processing unit 113 performs sample hold and gain control, conversion from analog signal to digital signal, white balance adjustment, gamma correction, etc. on the image pickup signal (analog signal) output from the image pickup unit 112 . , to generate moving image data (captured image data). The image data output unit 114 outputs the moving image data generated by the imaging signal processing unit 113 to an external device. be. Also, for example, the image data output unit 114 is a network interface that transmits moving image data to an external device via a wired or wireless network.

The camera 101 further has a description determination processing section 123 , a display data generation section 124 , a superimposition section 125 , an image compression processing section 126 , a network interface 127 and a recording section 128 . These are the same as described in the image processing apparatus 102 (see FIG. 3) described above, so description thereof will be omitted here.

Further, in the above-described embodiment, moving image data obtained by imaging the state in which the writer 201 describes while adding the description 203 to the description part 202, such as a whiteboard or a blackboard, is processed. However, other similar moving image data can also be processed. For example, it is possible to improve the visibility of the description by similarly processing moving image data obtained by imaging the state in which the writer is adding descriptions on paper while explaining on the desk.

The description on the description section 202 is not limited to characters, and may be graphics or marks. good too.

Moreover, this technique can also take the following structures.
(1) A description determination processing unit that processes moving image data obtained by imaging a state in which a description is written by a description part and determines a description part;
An image processing apparatus comprising a description superimposition processing unit that generates display data for displaying each portion determined as the description portion as a description and superimposes the display data on the moving image data.
(2) The description determination processing unit
Extracting the difference value for each pixel between the current frame image and the reference frame image, grasping groups in which pixels having the difference value greater than or equal to a threshold value are continuous, and determining whether or not each group is the above described part. The image processing apparatus according to (1) above.
(3) The description determination processing unit
determining whether or not it is the description portion using the shape evaluation value, the extraction time evaluation value, and the stationary time evaluation value for each group;
The description judgment processing unit
obtaining the shape evaluation value from the relationship between the aspect ratios of the minimum quadrilateral that includes the group and the maximum quadrilateral that is included inside the group;
Obtaining the extraction time evaluation value based on the difference value between the current frame and the reference frame image in a plurality of frames before the current frame for the pixels included in the group,
(2) above, wherein the still time evaluation value is obtained based on a difference value between a current frame and a frame image one frame before in a plurality of frames before the current frame for pixels included in the group Image processing device.
(4) The image according to any one of (1) to (3), wherein the description superimposition processing unit performs processing for enhancing the visibility of the description when superimposing the display data on the moving image data. processing equipment.
(5) The description superimposition processing unit
The image processing device according to (4), wherein the display data is superimposed on the moving image data after the moving image data is subjected to a process of reducing resolution.
(6) The description superimposition processing unit
The image processing according to (4) or (5) above, wherein after the display data is processed so that each portion determined to be the description portion is displayed with a border, the display data is superimposed on the moving image data. Device.
(7) The description superimposition processing unit
After processing the display data based on the color information of the moving image data so that the color of each portion determined to be the description portion is different from the color of the background image, the display data is superimposed on the moving image data. The image processing device according to any one of (4) to (6).
(8) Description for controlling the description superimposition processing unit so as not to superimpose the display data on the moving image data when it is detected that the face of the writer is facing forward based on the moving image data. The image processing apparatus according to any one of (1) to (7), further comprising a superimposition control section.
(9) an imaging unit that captures the state described by the writer in the description unit;
The image processing apparatus according to any one of (1) to (8), further comprising an imaging signal processing unit that processes the imaging signal obtained by the imaging unit to obtain the moving image data.
(10) The image processing apparatus according to any one of (1) to (9), further comprising a transmission section that transmits the moving image data obtained by the description superimposition processing section to an external device.
(11) A description determination processing step in which the description determination processing unit processes moving image data obtained by imaging the state of the description written by the writer in the description unit to determine the description part;
An image processing method, comprising: a description superimposition processing step in which a description superimposition processing unit generates display data for displaying each portion determined as the description portion as a description, and superimposes the display data on the moving image data.

DESCRIPTION OF SYMBOLS 10... Image processing system 101... Camera 102... Image processing apparatus 103-1-103-N... Display apparatus 111... Control part 112... Imaging part 113... Imaging signal processing Section 114... Image data output section 121... Control section 122... Image data input section 123... Description determination processing section 124... Display data generation section 125... Superimposition section 126... Image Compression processing unit 127 Network interface 128 Recording unit 129 Face recognition processing unit 201 Descriptor 202 Description unit 203 Description

Claims (18)

a description determination unit that determines the description part described by the description;
a state detection unit that detects the state of the writer;
a superimposing unit that generates display data corresponding to the description part based on the state of the writer detected by the state detection unit, and superimposes the display data on the image data obtained by imaging the writer. Device. The image processing apparatus according to claim 1, wherein the description determination unit determines the description part from image data obtained by imaging the writer. 3. The image processing apparatus according to claim 1, wherein the state detection unit detects the state of the writer from image data obtained by imaging the writer. 4. The image processing according to any one of claims 1 to 3, wherein the superimposing unit determines whether or not to superimpose the display data on the image data based on whether the writer is in a predetermined state. Device. The image processing apparatus according to claim 4, wherein the superimposing unit does not superimpose the display data on the image data when the writer is in the predetermined state. The image processing apparatus according to Claim 5, wherein the predetermined state is a state in which the writer is talking with a specific person. The image processing apparatus according to Claim 5, wherein the predetermined state is a state in which the writer faces a specific direction. The image processing apparatus according to any one of claims 1 to 3, wherein the superimposing unit does not superimpose the display data on the image data when the writer faces the front. 9. The image processing apparatus according to any one of claims 1 to 8, wherein the superimposing unit superimposes, on the image data, the display data for displaying each part determined as the description part as a description. 10. The image processing apparatus according to any one of claims 1 to 9, wherein the superimposing unit performs processing for enhancing visibility of description when superimposing the display data on the image data. The image processing apparatus according to any one of claims 1 to 9, wherein the superimposing unit superimposes the display data on the image data after performing a process of reducing the resolution of the image data. 10. Any one of claims 1 to 9, wherein the superimposing unit processes the display data so that each part determined to be the description part is displayed with a border, and then superimposes the display data on the image data. The described image processing device. The superimposing unit processes the display data based on the color information of the image data so that the color of each portion determined to be the description portion is different from the color of the background image, and then converts the display data into the image data. The image processing apparatus according to any one of claims 1 to 9, wherein the images are superimposed. further comprising a face recognition unit that recognizes the face of the writer,
The image processing apparatus according to any one of claims 1 to 13, wherein the state detection section detects the state of the writer based on a result of face recognition by the face recognition section. The image processing apparatus according to any one of claims 1 to 14, wherein the image data includes at least part of an area in which the description part is described. The image processing apparatus according to any one of claims 1 to 15, wherein the description portion includes at least one of characters, figures, and marks. a description determination step in which the description determination unit determines the description part described by the description;
A state detection step in which the state detection unit detects the state of the writer;
A superimposition step in which the superimposition unit generates display data corresponding to the description part based on the state of the writer detected in the state detection step, and superimposes the display data on the image data obtained by imaging the writer. An image processing method. a description determination step in which the description determination unit determines the description part described by the description;
A state detection step in which the state detection unit detects the state of the writer;
A superimposition step in which the superimposition unit generates display data corresponding to the description part based on the state of the writer detected in the state detection step, and superimposes the display data on the image data obtained by imaging the writer. A program for causing a computer to execute an image processing method having

Images