JP6091552B2 - Movie processing apparatus and movie processing system - Google Patents

Description

  The present invention relates to a moving image processing apparatus that processes information such as characters displayed on video of moving image data, and a moving image processing system including the moving image processing apparatus.

  2. Description of the Related Art Conventionally, in a network such as the Internet, a predetermined computer such as a moving image server or a moving image database stores moving image data and distributes moving image data to the viewer terminal. In order to promote such moving image distribution, various moving image processing apparatuses and moving image processing systems have been proposed. For example, in a video processing device or a video processing system, in addition to video data, video information related to program information, performer information, caption information, CM information, etc. of video data is distributed so that video information can be used for the viewer terminal. There is what I did.

  For example, the metadata distribution apparatus described in Patent Document 1 extracts and converts the metadata of the content of a net program broadcast on the local station from the metadata of the content of the key station using the extraction conversion table and the station-specific data. Since the extracted and converted metadata is distributed, the metadata of the content of the key station is distributed to the receiver as the metadata of the content of the local station. In the program, server type broadcasting is performed by using metadata of contents of the key station.

JP 2006-325134 A

  However, in a moving image processing device or a moving image processing system such as the above-described metadata distribution device, moving image information cannot be provided unless the broadcast station prepares metadata of moving image information in advance. Therefore, moving image information cannot be provided for moving image data for which such metadata is not prepared.

  In addition, video data includes video data that displays postings with texts, such as seminars and commentary, and video data with captions. The text displayed in these video data is video information. May not be provided. Note that it is conceivable to extract character information by extracting a still image frame in which characters are displayed from the moving image data and performing character recognition on the still image frame. Since the image quality is rough, it is difficult to extract character information by conventional character recognition processing.

  Furthermore, the viewer performs keyword search using an Internet search engine in order to search for video data of interest. However, since characters that are not provided as video information as described above cannot be obtained as search results, The video data may not be searched. Even when the viewer can search for moving image data by keyword search, it is often difficult to search for a scene related to the keyword in the moving image.

  Therefore, in consideration of the above circumstances, the present invention more reliably detects character information displayed in moving image data and enhances the convenience of the detected character information to improve the use and spread of the moving image distribution service. With the goal.

  In order to solve the above problems, a first moving image processing apparatus of the present invention includes a frame cutout unit that cuts out a plurality of still image frames at predetermined frame intervals from moving image data, and the plurality of still image frames. Approximate determination of the still image frames consecutive in the front and rear is sequentially performed, and when it is determined as approximate in the approximation determination, the preceding still image frame is set as a processing target frame and the subsequent still image frame is processed. An approximation determination unit that is excluded from the target frame, a sharpening unit that generates an edge-enhanced frame in which an edge is emphasized by performing a sharpening process on the processing target frame, and a binary by applying a binarization process to the edge-enhanced frame A binarization unit that generates an image frame; a character recognition unit that performs character recognition processing on the binary image frame to acquire character information; Along with the character information, at least metadata for recording the moving image information on the moving image data from which the character information is acquired and the still image information of the still image frame from which the character information is acquired is generated for each character information. And a metadata generation unit.

  According to the first moving image processing apparatus of the present invention, it is possible to provide text information metadata related to the content of moving image data even when moving image information metadata is not prepared in advance accompanying the moving image data. it can. In addition, since metadata of various character information displayed in the moving image data is created, the viewer can quickly determine which scene (still image data) in which moving image data the keyword of interest is displayed. It becomes possible to search. Furthermore, when the still image frame approximates the previous still image frame, the processing burden can be greatly reduced by excluding the character recognition processing target. As described above, according to the present invention, it is possible to more reliably detect the character information displayed in the moving image data, improve the convenience of the detected character information, and improve the use and spread of the moving image distribution service. It becomes possible.

  In order to solve the above problem, the second moving image processing apparatus of the present invention is the above-described first moving image processing apparatus of the present invention, wherein the binarization unit calculates a color temperature range of the edge enhancement frame. A binarization process using the plurality of threshold values for the edge enhancement frame is obtained based on each of a plurality of color temperatures for each predetermined threshold interval in the color temperature range. To generate a plurality of binary image frames, and the character recognition unit performs a character recognition process on each of the plurality of binary image frames to generate the character information for each of the plurality of binary image frames. A character recognition result including the character recognition result, and comparing each character recognition result to obtain the character information only from the binary image frame from which the optimum character recognition result was obtained. Of the binary image frame, and generates the meta data based on the character information optimal character recognition result has been obtained only from the binary image frame obtained.

  According to the second moving image processing apparatus of the present invention, character information can be extracted from the result of binarization processing with an optimum threshold. For example, even in the case of still image frames with the same color temperature range, the thresholds to be set for the binarization process may differ depending on the lighting conditions at the time of shooting. Also, it is possible to extract optimum character information.

  In order to solve the above-described problem, the third moving image processing apparatus of the present invention is the above-described first or second moving image processing apparatus of the present invention, wherein the approximate determination unit is configured to perform the still image frames consecutive in the front and rear. The change values of the RGB values and the luminance histogram are compared, and if the comparison value is equal to or greater than a predetermined approximation threshold, still and subsequent still image frames are determined to be approximate.

  According to the third moving image processing apparatus of the present invention, the approximation determination unit can maintain the accuracy of the approximation process and can greatly reduce the burden on the approximation process.

  In order to solve the above-described problem, a fourth moving image processing device according to the present invention is the above-described moving image processing device according to any one of the first to third aspects of the present invention, wherein the approximate determination unit is the stationary stationary image. In a picture frame, quadtree space division is performed, acceleration is converted using a second-order differential value of the amount of displacement of a singular point in each region, and comparison is made to determine whether or not a continuous still picture frame is approximated. It is characterized by that.

  According to the fourth moving image processing apparatus of the present invention, the approximation determination unit can determine the approximation more accurately for the still image frames continuous in the front and rear.

  In order to solve the above-described problem, the fifth moving image processing apparatus of the present invention is the above-described fourth moving image processing apparatus of the present invention, wherein the approximate determination unit performs quadtree space division of the still image frame. And calculating the distribution and direction vector of each singular point, and the character recognizing unit calculates the distribution and direction vector of each singular point calculated by the approximation determining unit to a predetermined value composed of the distribution and direction vector of each singular point. The character recognition processing is performed by comparing with training data.

  According to the fifth moving image processing apparatus of the present invention, it is possible to maintain the accuracy of the approximation determination in the approximation determination unit and reduce the processing burden in the character recognition unit.

  In order to solve the above problems, a sixth moving image processing apparatus of the present invention is the above-described moving image processing apparatus according to any one of the first to third aspects of the present invention, wherein the character recognition unit The character recognition processing is performed by calculating the distribution and direction vector of each singular point of the subject image and comparing it with predetermined training data composed of the distribution and direction vector of the singular point. The predetermined training data includes a singular point and a direction vector in a deteriorated state of the font, in addition to a singular point and a direction vector for each font.

  According to the sixth moving image processing apparatus of the present invention, since the character recognition unit calculates a singular point composed of a distribution and a direction vector, it is possible to reduce the processing load by reducing the amount of data used in the character recognition process. It is also possible to reduce the data amount of training data that needs to be prepared in large numbers, and to reduce the server cost for storing the training data.

  In order to solve the above problem, according to a seventh moving image processing device of the present invention, in the sixth moving image processing device of the present invention described above, the character recognizing unit has a quadtree space for the binary image frame. By performing the division, each singular point of the subject image is calculated, and the distribution and direction vector of the singular point are calculated.

  According to the seventh moving image processing apparatus of the present invention, the character recognition unit calculates a singular point composed of a distribution and a direction vector by quadtree space division, so that the processing burden in the character recognition process can be further reduced. it can.

  In order to solve the above-described problem, a moving image processing system according to the present invention includes the moving image processing device according to any one of the first to seventh aspects of the present invention described above and the metadata generation unit of the moving image processing device. And a character information database for storing the metadata for each character information.

  According to the moving image processing system of the present invention, the viewer obtains the metadata of the character information corresponding to the keyword from the character information database by searching using the keyword of interest, and records it in this metadata. It is possible to read the moving image information and reproduce the moving image data.

  According to the present invention, it is possible to more reliably detect character information displayed in moving image data and improve the convenience of the detected character information to improve the use and spread of the moving image distribution service.

It is a block diagram which shows the outline of the moving image processing system which concerns on one Embodiment of this invention. It is a flowchart which shows the character information detection operation | movement based on the moving image data in the moving image processing apparatus which concerns on one Embodiment of this invention.

  First, an overall configuration of a moving image processing system 1 according to an embodiment of the present invention will be described with reference to FIG. As shown in FIG. 1, in the moving image processing system 1, a moving image processing apparatus 2 that detects character information based on moving image data stores a moving image database that stores moving image data via a predetermined network 3 such as the Internet or a LAN. (DB) 4 and a character information database (DB) 5 for storing character information metadata (tag) are communicably connected. The moving image DB 4 and the character information DB 5 are communicably connected to the search engine 6 via the network 3, and the search engine 6 can search for the stored moving image data and character information metadata, respectively. In the moving image processing system 1, a viewer terminal 7 that can reproduce moving image data is also connected to the moving image DB 4, the character information DB 5, and the search engine 6 via the network 3.

  First, the moving image processing apparatus 2 will be described. For example, the moving image processing device 2 may be provided independently of the moving image DB 4 and the character information DB 5 on the network 3 or may be provided as a computer that manages the moving image DB 4 and the character information DB 5. In the present embodiment, an example in which one moving image processing device 2 is provided on the network 3 will be described. However, a plurality of moving image processing devices 2 may be provided on the network 3. The moving image processing apparatus 2 may be provided for each moving image category (business type).

  The moving image processing apparatus 2 performs character recognition processing on moving image data. The moving image data that can be subjected to character recognition processing is a moving image in which postings with characters are frequently displayed, such as seminars and explanations. It is not limited to videos such as movies with subtitles, and may be any video that displays characters. For example, videos related to finance with frequent stock price data and company names, product names, and company names are displayed. A wide range of videos related to advertisements.

  The moving image processing apparatus 2 includes, for example, a control unit 10, a storage unit 11, a communication unit 12, a frame cutout unit 13, an approximation determination unit 14, a sharpening unit 15, a binarization unit 16, and character recognition. Unit 17 and metadata generation unit 18. The frame cutout unit 13, the approximation determination unit 14, the sharpening unit 15, the binarization unit 16, the character recognition unit 17, and the metadata generation unit 18 are stored in the storage unit 11 and controlled by the control unit 10. It may be configured by a program that operates in.

  The control unit 10 includes a CPU (Central Processing Unit) and the like, and is configured to control the overall operation of the moving image processing apparatus 2. The storage unit 11 has a memory such as a ROM (Read Only Memory) and a RAM (Random Access Memory), and a recording medium such as a hard disk, and stores information, data, programs, and the like controlled by the control unit 10. Configured.

  The communication unit 12 is an interface for connecting the moving image processing apparatus 2 to the network 3 such as the Internet or a LAN, that is, connecting the moving image processing apparatus 2 to the moving image DB 4 or the character information DB 5 via the network 3.

  For example, the communication unit 12 communicates with the moving image DB 4 via the network 3 to receive moving image data for character information detection by the moving image processing apparatus 2 from the moving image DB 4. For example, when the moving image acquisition operation and the character information detection start operation are performed by the operator of the moving image processing device 2 and the moving image data and the moving image DB 4 from which the moving image data is acquired are selected, The moving image data is acquired from the moving image DB 4. Further, the communication unit 12 may sequentially acquire the moving image data recorded in the moving image DB 4 in accordance with the operation of a moving image acquisition crawler (not shown) included in the moving image processing device 2. A video acquisition crawler (not shown) may acquire all the video data in the network 3, but searches and acquires video data based on the category (industry) and keywords selected by the operator. May be.

  Note that the communication unit 12 is not limited to the moving image DB 4 as a moving image data acquisition destination, and acquires moving image data by receiving broadcast waves from a broadcasting station via the network 3 or by a broadcast receiver (not shown). Alternatively, the moving image data may be acquired from an external terminal such as a smartphone or a personal computer directly connected to the moving image processing device 2.

  Moreover, the communication part 12 transmits the metadata of the character information produced | generated with the moving image processing apparatus 2 to character information DB5 by communicating with character information DB5 via the network 3, for example. Note that the moving image processing apparatus 2 may be configured to be able to select the text information DB 5 that is the transmission destination of metadata by the communication unit 12.

  The frame cutout unit 13 acquires a plurality of still image frames from the moving image data received from the moving image DB 4 by the communication unit 12. Particularly in the present embodiment, the frame cutout unit 13 performs frame cutout for cutting out still image frames at predetermined frame intervals, for example, at time intervals of 1 second, with respect to the moving image data, and performs a plurality of still images at each frame interval. Get a picture frame. Note that the frame cutout unit 13 may acquire a still image frame in which the number of image pixels is increased by increasing the image quality in order to increase the amount of singular points to be described later.

  The approximate determination unit 14 performs approximate determination on the plurality of still image frames obtained by the frame cutout unit 13 and excludes the approximate still image frames from the processing target frames of the character recognition process. In the present embodiment, in particular, the approximate determination unit 14 sequentially performs an approximate determination of two still image frames that are consecutive in the front and the back of a plurality of still image frames. This still image frame is set as a processing target frame, and the subsequent still image frame is excluded from the processing target frame. When the subsequent still image frame is excluded from the processing target frame in the previous approximation determination, the frame to be compared with the subsequent still image frame in the current approximate determination is set as the processing target frame in the previous approximation determination. It becomes a still image frame.

  For example, the approximation determination unit 14 compares the RGB value and the change value of the histogram of the luminance for the still image frames continuous before and after as an approximation determination, and if the comparison value is equal to or greater than a predetermined approximation threshold, The image frame is determined to be approximate. Further, the approximate determination unit 14 performs quadtree space division on successive still image frames, converts the acceleration into a second-order differential value of the displacement amount of the singular point in each region, and compares the accelerations with each other to obtain a stationary still image. It is determined whether or not the image frame is approximate. Or the approximation determination part 14 may determine an approximation by either of the comparison using said histogram, and the comparison using said quadtree space division | segmentation.

  It should be noted that if the position where characters are to be displayed is determined in advance, such as video data that displays postings with characters, such as seminars or commentary, or video data with subtitles, approximate determination is made. For example, the unit 14 may acquire data of only the character display portion using a template prepared in advance, and may perform the approximation process by focusing on the character display portion. In this case, the moving image processing apparatus 2 manages a template for each character display portion and has functions such as new creation, change, and deletion of a template.

  The sharpening unit 15 generates an edge-enhanced frame in which the edge of the subject image is enhanced by performing a sharpening process on the processing target frame obtained by the approximation determining unit 14.

  In the sharpening process by the sharpening unit 15, for example, an unsharp mask is generated by calculating a Gaussian distribution (standard deviation) of the processing target frame, and a sharp frame is generated by adding the unsharp mask to the processing target frame. . Further, in this sharp frame, a section where the luminance is displaced between pixels is detected, the amount of luminance displacement in that section is calculated, the amount of displacement is converted into acceleration, and the edge increases as the acceleration increases in the sharp frame. Edge enhancement processing is performed to further emphasize the edge to generate an edge enhancement frame.

  The binarization unit 16 performs binarization processing on the edge enhancement frame generated by the sharpening unit 15 to generate a binary image frame. For example, the binarization unit 16 performs edge detection on the edge-enhanced frame using a zero-crossing method or the like before binarization processing, and binarizes the frame in which a singular point is detected by the edge detection. The treatment may be performed.

  In particular, in the present embodiment, the binarization unit 16 calculates the color temperature range (frequency band in which each dot of the image exists) of the edge enhancement frame (or the original still image frame), and performs predetermined processing in the color temperature range. A plurality of threshold values are acquired based on each of a plurality of color temperatures for each threshold interval. Then, the binarization unit 16 generates a plurality of binary image frames by performing binarization processing using a plurality of thresholds on the edge enhancement frame. Note that the binarization unit 16 generates a binary image frame having too many black portions or white portions even if binarization processing is performed with threshold values based on the maximum value and the minimum value of the color temperature range of the edge enhancement frame. Therefore, binarization processing with these threshold values is not performed. Further, the binarization processing unit 16 refers to the density distribution (black portion and white portion) of the generated plurality of binary image frames, and excludes the binary image frame having too many black portions or white portions. Also good.

  The character recognition unit 17 performs character recognition processing on the binary image frame generated by the binarization unit 16 and acquires character information. In this character recognition process, the character recognition unit 17 recognizes characters one by one from the binary image frame, extracts, for example, singular points indicating the subject image of the binary image frame, and collects the extracted singular points. Is a character candidate of one character. For example, the character recognition unit 17 calculates each singular point of the subject image by performing quadtree space division on the binary image frame. When the approximation determination unit 14 performs quadtree space division to calculate singular points, the character recognition unit 17 does not perform quadtree space division and the singular points calculated by the approximation determination unit 14. May be used.

  Even when characters are displayed obliquely in a binary image frame (original still image frame), the character recognition unit 17 introduces a reference line and sets a vector space to thereby determine a singular point of a character candidate. The vector direction can be accurately corrected. For example, the character recognition unit 17 acquires a reference line from a vector in the arrangement direction of adjacent character candidates in the same manner as the mechanism used in 3D space recognition, and the character candidates so that the reference line is horizontal or vertical. Correct the vector direction of the singular point.

  Also, the character recognition unit 17 registers in advance singularity distributions and direction vectors for various characters in various predetermined fonts and their deterioration states as training data for character recognition. Then, the character recognition unit 17 determines whether the character candidate corresponds to any character of the training data by comparing the singular point of the character candidate with the training data, and recognizes each character on the binary image frame. .

  Furthermore, the character recognition unit 17 performs dictionary data processing on the characters recognized as described above, and in this dictionary data processing, two or more adjacent character strings are stored in a pre-registered word dictionary. Compare and collate. And the character recognition part 17 determines whether a character string corresponds to any word of a word dictionary, and acquires character information based on the determination result. Even if the character string does not correspond to any word in the word dictionary, for example, if it corresponds to a misread pattern, the character recognition unit 17 automatically corrects the character string to the correct character string for the misread pattern. It may be information. As described above, the character information recognized by the character recognition unit 17 may be stored as additional training data together with data including a collection of singular points in the binary image frame.

  The character recognition unit 17 adds priorities to the training data and words in the word dictionary, and in the character recognition process, the character recognition unit 17 compares the candidates with the character candidates and character strings in order from the training data and words having the highest priority. Use it. For example, the character recognition unit 17 sets a high priority for training data and words that are frequently recognized.

  In addition, the character recognition unit 17 may include a character recognition engine that performs character recognition processing as described above for each category (business type) of a moving image, and further, training data and a word dictionary may be registered in advance for each category of a moving image. The character recognition unit 17 determines a category included in the moving image information recorded in the moving image data, or determines a category input by the operator. The character recognition unit 17 may use a character recognition engine corresponding to the determined category and perform character recognition processing preferentially using training data or a word dictionary corresponding to the category.

  Particularly in the present embodiment, the character recognition unit 17 performs character recognition processing on each of a plurality of binary image frames generated by the binarization unit 16. Then, the character recognition unit 17 obtains a character recognition result including character information for each of a plurality of binary image frames, and compares the character recognition results. At this time, the character recognition unit 17 determines, for example, the number of recognized characters and the number of characters derived from the dictionary as meaningful characters among the recognized characters as the character recognition result, and the number of these characters is large. The object is determined as the optimum character recognition result. Note that the priority of characters that have meaning may be set higher than characters that are simply recognized. And the character recognition part 17 acquires character information only from the binary image frame from which the optimal character recognition result was obtained.

  The metadata generation unit 18 generates metadata for each character information obtained by the character recognition unit 17. The metadata generation unit 18 generates, for example, metadata in which the moving image information regarding the moving image data from which the character information is acquired and the still image information of the still image frame from which the character information is acquired together with the character information. .

  As the video information of the metadata, the video ID of the video data, the number of frames, the frame size, the format format, etc. may be recorded. In addition, the title of the video data, author information, creation date, video category, thumbnail (URL ) Etc. may be recorded. As the still image information of the metadata, for example, the relation ID and time information (time stamp) in the moving image data of the still image frame from which the character information is acquired, the frame number (unique ID) of this still image frame, and the like are recorded. It's okay. Further, the metadata generation unit 18 may record the processing date / time and processing status data of the character recognition processing in the metadata. In the present embodiment, in particular, the metadata generation unit 18 is a character acquired from only a binary image frame from which an optimum character recognition result is obtained among a plurality of binary image frames generated by the binarization unit 16. Generate metadata based on the information.

  The moving image DB 4 is a database that stores moving image data and distributes moving images according to an access from the viewer terminal 7 by a download method or a streaming method. In addition, the moving image DB 4 can provide the moving image data itself to the moving image processing device 2 in accordance with an acquisition operation from the moving image processing device 2. The moving image data stored in the moving image DB 4 may store moving image information such as preset moving image titles and contents in addition to video data and audio data. Can be searched. Further, the moving image DB 4 may be configured to grant access authority to a predetermined viewer and distribute the moving image data in response to a request from the viewer terminal 7 of the viewer.

  In the present embodiment, an example in which one moving image DB 4 is provided on the network 3 will be described. However, a plurality of moving image DBs 4 may be provided on the network 3. The moving image DB 4 may be provided for each moving image category (business type). The moving image DB 4 is not limited to the one used by one moving image processing device 2, and may be provided so as to be usable by a plurality of moving image processing devices 2.

  The character information DB 5 is a database that stores character information metadata generated by the moving image processing apparatus 2 and provides character information metadata in response to access from the viewer terminal 7. The metadata stored in the character information DB 5 can be searched by the search engine 6 by using the character information as a search keyword. Further, the character information DB 5 may be configured to give access authority to a predetermined viewer and to provide metadata of the character information in response to a request from the viewer terminal 7 of the viewer.

  Further, the character information DB 5 may store metadata so that metadata of character information having a high storage frequency or high search frequency is preferentially searched. Further, when the viewer terminal 7 searches for predetermined character information via the search engine 6, the character information DB 5 may provide metadata for the character information one by one to the search engine 6. A list including a plurality of metadata may be provided to the search engine 6.

  In the present embodiment, an example in which one character information DB 5 is provided on the network 3 will be described. However, a plurality of character information DBs 5 may be provided on the network 3. The character information DB 5 may be provided for each category (business type) of the moving image. The character information DB 5 is not limited to that used by one moving image processing apparatus 2, and may be provided so as to be usable by a plurality of moving image processing apparatuses 2.

  The viewer terminal 7 may be any terminal that can be connected to the network 3 and can reproduce moving image data. For example, the viewer terminal 7 may be a mobile terminal such as a smartphone, a mobile phone, and a tablet, or a stationary terminal such as a personal computer and a television. Good.

  Next, the character information detection operation based on the moving image data by the moving image processing device 2 in the moving image processing system 1 having such a configuration will be described with reference to FIG.

  First, in the moving image processing system 1, a moving image provider of moving image data uploads moving image data to the moving image DB 4.

  On the other hand, in the video processing device 2, for example, when the operator starts the character information detection start operation and selects the video data and the video DB 4 from which the video data is acquired, the communication unit 12 starts from the video DB 4. Movie data is acquired (step S1).

  Subsequently, the frame cutout unit 13 acquires a plurality of still image frames at predetermined frame intervals from the moving image data acquired by the communication unit 12 (step S2).

  Then, the moving image processing apparatus 2 sequentially performs character information detection for a plurality of still image frames (step S3).

  In the character information detection of each still image frame, first, it is determined whether or not there is a processing target frame for which the previous character information was detected (step S4). If there is a previous processing target frame (step S4: Yes), the process proceeds to an approximation process (step S5). On the other hand, when there is no previous processing target frame (step S4: No), the process proceeds to the sharpening process (step S6) with the current still image frame as the processing target frame. In this case, the current still image frame becomes the previous processing target frame when the character information of the next still image frame is detected (step S3).

  In the approximation process (step S5), the approximation determination unit 14 determines whether or not the current still image frame approximates the previous process target frame. If the current still image frame approximates the previous processing target frame (step S5: Yes), the current still image frame is excluded from the processing target frame and the process proceeds to sharpening processing (step S6). Without proceeding, the process proceeds to the character information detection (step S3) of the next still image frame.

  On the other hand, if the current still image frame does not approximate the previous processing target frame (step S5: No), the process proceeds to the sharpening process (step S6) using the current still image frame as the processing target frame. In this case, the current still image frame becomes the previous processing target frame when the character information of the next still image frame is detected (step S3).

  In the sharpening process (step S6), the sharpening unit 15 generates an edge-enhanced frame by performing the sharpening process on the processing target frame.

  Further, the process proceeds to binarization processing (step S7), and the binarization unit 16 acquires a plurality of threshold values based on the color temperature range of the edge enhancement frame, and uses the plurality of threshold values to generate an edge enhancement frame. Binarization processing is performed to generate a plurality of binary image frames.

  Further, the process proceeds to a character recognition process (step S8), and the character recognition unit 17 performs the character recognition process on each of the plurality of binary image frames. Then, the character recognition unit 17 compares the character recognition results of the plurality of binary image frames, and acquires character information from the binary image frame from which the optimum character recognition result is obtained (step S9).

  Subsequently, the process proceeds to metadata creation (step S10), and the metadata generation unit 18 creates text information metadata.

  The metadata created by the moving image processing apparatus 2 in this way is uploaded to the character information DB 5 by the communication unit 12 via the network 3 (step S11). The character information DB 5 sorts the uploaded metadata so that the user can easily use it.

  In the present embodiment, as described above, the moving image processing apparatus 2 continues to the front and rear with respect to the plurality of still image frames and the frame cutout unit 13 that extracts a plurality of still image frames at predetermined frame intervals from the moving image data. Approximate determination of the still image frame to be performed, and if it is determined to be approximate in the approximate determination, the preceding still image frame is set as the processing target frame and the subsequent still image frame is excluded from the processing target frame. Unit 14, a sharpening unit 15 that performs sharpening processing on the processing target frame to generate an edge enhancement frame that emphasizes edges, and a binary that performs binarization processing on the edge enhancement frame to generate a binary image frame The characterizing unit 16, a character recognizing unit 17 that performs character recognition processing on the binary image frame to acquire character information, In addition, a metadata generation unit 18 that generates, for each character information, metadata that records moving image information related to the moving image data from which the character information has been acquired and still image information of the still image frame from which the character information has been acquired. It is prepared for.

  With such a configuration, according to the present embodiment, even when video information metadata is not prepared in advance accompanying the video data, it is possible to provide text information metadata related to the content of the video data. it can. In addition, since metadata of various character information displayed in the moving image data is created, the viewer can quickly determine which scene (still image data) in which moving image data the keyword of interest is displayed. It becomes possible to search. Furthermore, when the still image frame approximates the previous still image frame, the processing burden can be greatly reduced by excluding the character recognition processing target. As described above, according to the present invention, it is possible to more reliably detect the character information displayed in the moving image data, improve the convenience of the detected character information, and improve the use and spread of the moving image distribution service. It becomes possible.

  Further, according to the present embodiment, in the moving image processing apparatus 2, the binarization unit 16 calculates the color temperature range of the edge enhancement frame, and each of a plurality of color temperatures for each predetermined threshold interval in the color temperature range. A plurality of threshold values are acquired based on the image, and binarization processing using the plurality of threshold values is performed on the edge enhancement frame to generate a plurality of binary image frames. Character recognition processing is performed on each of the value image frames to obtain character recognition results including character information for each of the plurality of binary image frames, and each character recognition result is compared to obtain an optimum character recognition result. The metadata generation unit 18 obtains character information from only the binary image frame from which the optimum character recognition result is obtained from among the plurality of binary image frames. Configured to generate the metadata based on.

  With such a configuration, the moving image processing apparatus 2 can extract character information from the result of binarization processing with an optimum threshold. For example, even in the case of still image frames with the same color temperature range, the thresholds to be set for the binarization process may differ depending on the lighting conditions at the time of shooting. Also, it is possible to extract optimum character information.

  Further, according to the present embodiment, in the moving image processing apparatus 2, the approximation determination unit 14 compares the change values of the RGB value and the luminance histogram for the consecutive still image frames, and the comparison value is equal to or greater than a predetermined approximation threshold. If so, it is configured to determine that still image frames consecutive in the front and rear are approximate.

  With such a configuration, the approximation determination unit 14 can maintain the accuracy of the approximation process and can significantly reduce the burden on the approximation process.

  Further, according to the present embodiment, in the moving image processing apparatus 2, the approximate determination unit 14 performs quadtree space division on the continuous still image frames to obtain a second-order differential value of the displacement amount of the singular point in each region. Then, it is configured to determine whether or not the still and successive frames are approximated by converting to acceleration and comparing.

  With such a configuration, the approximation determination unit 14 can determine the approximation more accurately for the still image frames consecutive in the front and rear.

  Further, according to the present embodiment, in the moving image processing apparatus 2, the approximate determination unit 14 calculates the distribution and direction vector of each singular point when performing the quadtree space division of the still image frame, and the character recognition unit 17. Is configured to perform character recognition processing by comparing the distribution and direction vector of each singular point calculated by the approximation determination unit 14 with predetermined training data composed of the distribution and direction vector of each singular point.

  With such a configuration, it is possible to maintain the accuracy of the approximation determination in the approximation determination unit 14 and reduce the processing burden on the character recognition unit 17.

  Further, according to the present embodiment, in the moving image processing apparatus 2, the character recognition unit 17 calculates the distribution and direction vector of singular points of the subject image of the binary image frame, and includes the distribution and direction vectors of singular points. Character recognition processing is performed by comparing with predetermined training data.

  With such a configuration, the character recognition unit 17 calculates a singular point composed of a distribution and a direction vector, so that the amount of data used in the character recognition process can be reduced and the processing load can be reduced. It is also possible to reduce the data amount of the training data that needs to be performed and reduce the server cost for storing the training data.

  Further, according to the present embodiment, in the moving image processing device 2, the character recognition unit 17 calculates each singular point of the subject image by performing quadtree space division on the binary image frame, and the singular point. The point distribution and the direction vector are calculated.

  With such a configuration, the character recognizing unit 17 calculates a singular point composed of a distribution and a direction vector by quadtree space division, so that the processing burden in the character recognition processing can be further reduced.

  Further, according to the present embodiment, the moving image processing system 1 includes character information that stores metadata for each character information generated by the moving image processing device 2 and the metadata generation unit 18 of the moving image processing device 2 as described above. DB (database) 5.

  With such a configuration, in the moving image processing system 1, the viewer searches using the keyword of interest to acquire the text information metadata corresponding to the keyword from the text information DB 5, and this metadata. It is possible to read the moving image information recorded in the video and reproduce the moving image data.

  In the present embodiment, the configuration has been described in which the character recognition unit 17 pre-registers the distribution of singular points and the direction vector for each character of each font as training data for character recognition. However, the present invention is not limited to this configuration. . For example, in another embodiment, a training data database for registering training data is separately provided, and the moving image processing apparatus 2 is configured to access the training data database and acquire training data during character recognition processing. Also good.

  In addition to the training data for each character of each font, the character recognition unit 17 and the training data database may be configured to register training data for the decorated character. Decorated characters are, for example, characters used in presentations, etc. that are displayed in a different color than the default font, characters that are italicized or bold, Characters, shadowed characters, etc.

  As another embodiment, the moving image processing apparatus 2 may be configured to recognize a face image of a specific person (particularly a celebrity) in a still image frame and generate the person information as metadata. it can. In this case, the moving image processing device 2 registers the distribution of the feature points of the face image of a specific person and the direction vector (in this case, the 3D direction vector) as training data in advance, and adds the specific data along with the training data. The person information of the person is also registered. Then, the moving image processing device 2 performs face recognition processing from the distribution of the feature points in the still image frame (binary image frame) and the vector direction, and determines whether or not the extracted face image corresponds to the training data. . If the extracted face image corresponds to training data, person information associated with the training data is generated as metadata associated with moving image data and still image frames, and registered in a person information database (not shown). To do.

  Similarly, the moving image processing apparatus 2 can be configured to recognize an image of a landmark (particularly a famous building) in a still image frame and generate the landmark information as metadata.

  Furthermore, as another embodiment, the moving image processing apparatus 2 recognizes an image of a specific landscape (especially, a landscape such as a sea that can be assumed from colors) in a still image frame, and generates the landscape information as metadata. It can also be configured. In this case, the moving image processing apparatus 2 registers a histogram indicating the color distribution of a specific landscape as training data in advance, and also registers the landscape information of the specific landscape along with the training data. Then, the moving image processing device 2 calculates a histogram of the color distribution in the still image frame (processing target frame), and determines whether or not the histogram corresponds to the training data. When the calculated histogram corresponds to the training data, landscape information associated with the training data is generated as metadata associated with the moving image data and the still image frame, and is registered in the landscape information database (not shown). .

  In the present embodiment, the configuration in which the moving image processing apparatus 2 has the function of character recognition processing has been described, but the present invention is not limited to this configuration. For example, in another embodiment, in the moving image processing system 1, the moving image processing device 2 or another server may be configured to provide a program or application having the same character recognition processing function as the moving image processing device 2. Good.

1 video processing system 2 video processing device 3 network 4 video database (DB)
5 Character information database (DB)
6 Search Engine 7 Viewer Terminal 10 Control Unit 11 Storage Unit 12 Communication Unit 13 Frame Cutout Unit 14 Approximate Determination Unit 15 Sharpening Unit 16 Binarization Unit 17 Character Recognition Unit 18 Metadata Generation Unit

Claims (7)

A frame cutout unit that cuts out a plurality of still image frames at predetermined frame intervals from video data;
For each of the plurality of still image frames, approximation determination of the still image frames consecutive in the front and rear is sequentially performed, and when it is determined to be approximation in the approximation determination, the preceding still image frame is set as a processing target frame. And an approximate determination unit that excludes the subsequent still image frame from the processing target frame;
A sharpening unit that generates an edge-enhanced frame that enhances an edge by applying a sharpening process to the processing target frame;
A binarization unit that binarizes the edge enhancement frame to generate a binary image frame;
A character recognition unit that obtains character information by performing character recognition processing on the binary image frame;
Along with the character information, at least metadata for recording moving image information related to the moving image data from which the character information has been acquired and still image information of the still image frame from which the character information has been acquired is generated for each character information. A metadata generation unit;
Equipped with a,
The binarization unit calculates a color temperature range of the edge enhancement frame, acquires a plurality of threshold values based on each of a plurality of color temperatures for each predetermined threshold interval in the color temperature range, and converts the edge A binarization process using each of the plurality of thresholds is performed on the emphasized frame to generate a plurality of the binary image frames,
The character recognition unit performs character recognition processing on each of the plurality of binary image frames to obtain a character recognition result including the character information for each of the plurality of binary image frames. In comparison, the character information is obtained only from the binary image frame where the optimum character recognition result was obtained,
The metadata generation unit generates the metadata based on the character information acquired from only the binary image frame from which an optimum character recognition result is obtained among the plurality of binary image frames. A moving image processing apparatus. The approximate determination unit compares the change value of the histogram of RGB value and luminance for the still image frames that are continuous before and after, and if the comparison value is equal to or greater than a predetermined approximate threshold, The moving image processing apparatus according to claim 1 , wherein the moving image processing apparatus is determined to be approximate. The approximate determination unit performs quadtree space division on the still image frames that are continuous before and after, converts the acceleration into a second-order differential value of the amount of displacement of the singular point in each region, and compares the accelerations with each other to compare them continuously. 3. The moving image processing apparatus according to claim 1, wherein it is determined whether or not the still image frame is approximate. The approximate determination unit calculates a distribution and direction vector of each singular point when performing quadtree space division of the still image frame,
The character recognition unit performs the character recognition process by comparing the distribution and direction vector of each singular point calculated by the approximation determination unit with predetermined training data including the distribution and direction vector of each singular point. The moving image processing apparatus according to claim 3 . The character recognition unit calculates the distribution and direction vector of each singular point of the subject image of the binary image frame, and compares the character recognition process with predetermined training data including the distribution and direction vector of the singular point. moving image processing apparatus according to any one of claims 1 or claim 2, wherein the performing. The character recognition unit calculates each singular point of the subject image by performing quadtree space division on the binary image frame, and calculates a distribution and a direction vector of the singular point. The moving image processing apparatus according to claim 5 . The moving image processing apparatus according to any one of claims 1 to 6 ,
A character information database for storing the metadata for each character information generated by the metadata generation unit of the video processing device;
A video processing system comprising:

Images