JP2022148719A - Image processing device, image processing method, and program - Google Patents

Abstract

To reduce overlap between a recognized object in an image and information about the object.SOLUTION: An image processing device 3 includes an object recognition unit 302 that recognizes an object to be recognized in an image, and an identification information synthesizing unit 303 that synthesizes the identification information about the object to be recognized recognized by the object recognition unit 302 with the image, and the identification information synthesizing unit 303 derives an area that does not overlap with the object to be recognized on the basis of the position and size of the object to be recognized in the image, and determines the position and dimensions of a display area that displays the identification information so as to fit within the derived area.SELECTED DRAWING: Figure 1

Description

The disclosure of the present specification relates to an image processing device, an image processing method, and a program.

2. Description of the Related Art Techniques for recognizing an object in an image captured by an imaging device are used, for example, in monitoring systems for the purpose of crime prevention, systems for investigating customer purchasing trends in stores, and the like. In the above-described system, for example, only objects that satisfy predetermined conditions are recognized among the objects in the image, and the frame indicating the recognized object and the identification information of the object are combined with the image, and the display device synthesizes the image. to display.

As a related technique, Japanese Patent Application Laid-Open No. 2002-200001 describes displaying a main subject with the highest priority among a plurality of displayed subjects and items related to the main subject, and displaying items related to subjects other than the main subject. describes a display control method for adjusting the display position of an item related to another subject so that the display of the subject does not overlap with the display of the subject.

JP 2013-141092 A

However, with the display control method described in Patent Literature 1, the display of an item related to another subject may overlap the display of the subject associated with that item or the display of another subject. In addition, when recognizing objects in an image, a specific priority order may not be set for the recognized objects. Therefore, when a plurality of objects are recognized from an image, the identification information of those objects overlaps with the recognized objects, making it difficult to accurately grasp the state of the recognized objects.

In view of the circumstances as described above, it is an object of one aspect of the present invention to provide a technique capable of reducing overlap between an object recognized in an image and information about the object.

An image processing apparatus according to an aspect of the present invention combines an object recognition unit that recognizes a recognition target object appearing in an image, and identification information related to the recognition target object recognized by the object recognition unit, with the image. an identification information synthesizing unit, wherein the identification information synthesizing unit derives a region that does not overlap with the recognition target object based on the position and size of the recognition target object in the image, and The position and dimensions of the display area for displaying the identification information are determined so as to fit within the space.

An image processing method according to an aspect of the present invention comprises: a computer recognizing a recognition target object appearing in an image; and synthesizing identification information related to the recognized recognition target object with the image. and synthesizing the identification information with the image includes: deriving a region that does not overlap with the object to be recognized based on the position and size of the object to be recognized in the image; determining a position and size of a display area for displaying the identification information so as to fit therein.

A program according to an aspect of the present invention causes a computer to recognize a recognition target object appearing in an image and synthesize identification information about the recognized recognition target object with the image. , synthesizing the identification information with the image includes: deriving a region that does not overlap with the recognition target object based on the position and size of the recognition target object in the image; determining the position and size of the display area for displaying the identification information.

According to the above aspect, it is possible to reduce overlap between the object recognized in the image and the information about the object.

1 is a diagram illustrating a system configuration example of an image processing system according to an embodiment; FIG. 4 is a flowchart illustrating an example of processing performed by an image processing apparatus according to an embodiment; 9 is a flowchart illustrating an example of identification information display area determination processing according to an embodiment; FIG. 4 is a flowchart illustrating an example of display area selection processing in the flowchart of FIG. 3; FIG. It is a figure which shows an example of the synthetic image displayed on a display apparatus. FIG. 10 is a diagram illustrating an example of a problem that may occur when objects to be recognized increase. FIG. 10 is a diagram showing an example of a composite image after the date and time and identification information display areas have been updated; FIG. 10 is a diagram illustrating an example of a method for extracting candidate areas when updating a display area for date and time and identification information; FIG. 10 is a diagram showing an example of a composite image displaying date and time and identification information in different display areas; It is a figure explaining the hardware configuration example of a computer.

Embodiments of the present invention will be described below with reference to the drawings. It should be noted that, in the following description, a detailed description of known processing among the processing for recognizing an object appearing in an image will be omitted.

FIG. 1 is a diagram illustrating a system configuration example of an image processing system according to an embodiment. An image processing system 1 illustrated in FIG. 1 includes a camera 2 , an image processing device 3 and a display device 4 .

The camera 2 takes an image of an object within an imaging range θ from a viewpoint P and transfers it to the image processing device 3 as image data (hereinafter simply referred to as “image”). An image herein includes a frame of video data. That is, the camera 2 may be an imaging device that outputs a still image, or may be an imaging device that outputs a moving image. The camera 2 and the image processing device 3 are connected by a transmission cable such as a USB (Universal Serial Bus) cable, for example. The camera 2 and the image processing device 3 may be connected by wireless communication conforming to a known short-range wireless communication standard or the like, for example. Also, the camera 2 and the image processing device 3 may be connected via a communication network (not shown) such as the Internet, for example.

The image processing device 3 recognizes an object that satisfies a predetermined condition from the image transferred from the camera 2, and performs a process of synthesizing identification information related to the recognized object with the image. Further, the image processing device 3 causes the display device 4 to display an image synthesized with the identification information. The image processing device 3 includes an image acquisition unit 301 , an object recognition unit 302 , an identification information synthesis unit 303 , an image output unit 304 and a storage unit 310 .

An image acquisition unit 301 acquires an image from the camera 2 . The image acquisition unit 301 may acquire images actively or passively. When actively acquiring an image, the image acquiring unit 301 acquires an image captured by the camera 2 by transmitting an imaging instruction to the camera 2 at predetermined time intervals, for example.

The object recognition unit 302 recognizes an object that satisfies a predetermined condition (hereinafter referred to as “recognition target object”) among the objects appearing in the acquired image. The object to be recognized is not limited to one type, and may be two or more types. The object recognition unit 302 recognizes an object to be recognized using, for example, a trained model 311 stored in the storage unit 310 . The trained model 311 is a model generated (constructed) by learning the features of the object to be recognized by machine learning using a plurality of learning images related to the object to be recognized. When input, it outputs information indicating an object having the learned features. The learned model 311 can be generated using, for example, YOLO (You Only Look Once). Note that the learning method of machine learning is not limited to a specific method. Further, the object recognition method by the object recognition unit 302 is not limited to the method using the trained model 311 .

The identification information synthesizing unit 303 synthesizes (adds) the identification information regarding the recognition target object recognized by the object recognizing unit 302 and the information regarding date and time into the image. The date and time information is, for example, information indicating the image capturing date and time. Further, the identification information synthesizing unit 303 of the image processing apparatus 3 of the present embodiment determines the position and size of the display area for displaying the information on date and time and the identification information. More specifically, the identification information synthesizing unit 303 determines the position and size of the display area for displaying the date and time information and the identification information so that the date and time information and the identification information do not overlap with the object to be recognized. In the present embodiment, it is assumed that the identification information synthesizing unit 303 determines the position and size of the display area, and synthesizes the information about the date and time and the identification information in the image in the display area. , the image processing device 3 is provided with a determining unit for determining the position and size of the display area, and a synthesizing unit for synthesizing the information on the date and time and the identification information in the determined display area within the image. The determining unit and synthesizing unit may be configured to function as the identification information synthesizing unit 303 .

The image output unit 304 causes the display device 4 to display an image obtained by synthesizing the identification information and the information about date and time (hereinafter referred to as a "composite image"). When the above processing in the object recognition unit 302 and the identification information synthesis unit 303 is not performed, the image output unit 304 causes the display device 4 to display the image acquired by the image acquisition unit 301 . Also, the composite image is stored in the storage unit 310 as composite image data 312 .

The storage unit 310 stores various data, programs, etc. including a trained model 311 and synthesized image data 312 . For example, the image acquired by the image acquiring unit 301 may be stored in the storage unit 310 separately from the combined image data 312 . Also, instead of the composite image data 312, the storage unit 310 may store an image acquired by the image acquisition unit 301 and object identification information associated with the image.

An image processing system 1 illustrated in FIG. 1 recognizes an object to be recognized in an image captured by a camera 2, and displays an image synthesized (added) with identification information of the recognized object on a display device 4. do. This type of image processing system 1 is used, for example, as a monitoring system for the purpose of crime prevention or as a system for investigating the purchasing trends of customers in a store.

FIG. 2 is a flowchart illustrating an example of processing performed by the image processing apparatus according to one embodiment. For example, the image processing device 3 performs processing according to the flowchart illustrated in FIG. 2 in parallel with processing for acquiring an image from the camera 2 . "Identification information" in the following description includes identification information about a recognized object and information about date and time, unless otherwise specified.

The image processing apparatus 3 of the present embodiment inputs the acquired image to the trained model 311 and outputs information indicating an object to be recognized in the image (step S1). After that, the image processing device 3 determines the position and size of the frame surrounding the object to be recognized based on the output information (step S2). The processing of steps S1 and S2 is performed by the object recognition unit 302 . The object recognition unit 302 performs the processes of steps S1 and S2 using known object recognition methods and object detection methods in the field of image processing. The information output by the object recognition unit 302 in step S1 includes information indicating that the object to be recognized does not exist in the image.

Next, the image processing device 3 determines whether or not the conditions for updating the display area of the identification information are satisfied (step S3). The determination in step S3 is performed by the identification information synthesis unit 303, for example. The conditions for updating are, for example, an image that is currently being processed and an image that was captured earlier in time than the image and has already undergone processing for synthesizing identification information. It includes the condition that at least one of the number and positions of the objects is different. The update condition may include, for example, a condition that the number of images synthesized with object identification information is a predetermined value or a positive integer (natural number) multiple of that value. For example, if it is set to update the display area each time object identification information is synthesized for N images (N is an integer of 1 or more), the update condition is set to be the target of processing. The cumulative number X of images satisfies X/N=a (where a is a positive integer), and the number Y of images to be processed since the last time the display area was determined satisfies Y=N. , or

If the update condition is satisfied (step S3; YES), the identification information synthesizing unit 303 performs the identification information display area determination process (step S4), and then inserts the frame surrounding the object to be recognized and the identification information into the image. (step S5). If the update condition is not satisfied (step S3; NO), the identification information synthesis unit 303 omits the identification information display area determination process (step S4), and inserts the frame surrounding the object to be recognized and the identification information into the image. Synthesize (step S5). After step S5, the image processing device 3 outputs the synthesized image (step S6). The processing in step S6 includes processing for storing the image synthesized by the identification information synthesizing unit 303 in the storage unit 310 as synthesized image data 312, and displaying the image synthesized by the identification information synthesizing unit 303 on the display device 3 by the image output unit 304. and processing to cause

The image processing device 3 of the present embodiment performs the processing of steps S1 to S6 on each of a plurality of images continuously captured by the camera 2, thereby obtaining a plurality of images obtained by synthesizing information about an object to be recognized. The images are displayed on the display device 4 in chronological order. The processing of steps S1 to S6 for each of a plurality of images may be performed in a pipeline or by a sequential execution method (that is, when the processing of steps S1 to S6 for one image is completed, the next one method of performing the processing of steps S1 to S6 on an image). Further, the image processing device 3 may perform processing according to the flowchart illustrated in FIG. 2 using a set of images captured by the camera 2 in the past.

As described above, the image processing apparatus 3 according to the present embodiment determines the identification information display area when a predetermined condition is satisfied while repeating the process of synthesizing the identification information related to the recognition target object recognized from the acquired images. Processing (step S4) is performed to update the display area of the identification information. As the identification information display area determination process, the image processing apparatus 3 of the present embodiment performs, for example, the process according to the flowcharts illustrated in FIGS. 3 and 4 .

FIG. 3 is a flowchart illustrating an example of identification information display area determination processing according to an embodiment. FIG. 4 is a flowchart illustrating an example of display area selection processing in the flowchart of FIG. The processing according to the flowcharts of FIGS. 3 and 4 is performed by, for example, the identification information synthesizing unit 303 of the image processing device 3 .

When starting the identification information display area determination process, the identification information synthesizing unit 303 derives an out-of-frame area in the image based on the position and dimensions of the frame surrounding the object to be recognized (step S41), and extracts an out-of-frame area from within the out-of-frame area. A candidate area is extracted as a candidate for the display area of (step S42). The out-of-frame area is an area excluding the area within the frame surrounding the object to be recognized, out of the entire area in the image that can be set as the identification information display area. In step S42, the identification information synthesizing unit 303 extracts, as a candidate area, a rectangular area having a predetermined dimension or more from the rectangular areas that can be set in the out-of-frame area.

After the process of step S42, the identification information synthesizing unit 303 determines whether or not there is a candidate area (step S43). If there is no candidate area (step S43; NO), the identification information synthesizing unit 303 determines a predetermined area as the display area (step S44), and ends the identification information display area determination process. For the predetermined area in step S44, the display area is determined based on, for example, the initial setting information (for example, the area of predetermined size at the bottom right of the screen) regarding the display area of the identification information.

If there is a candidate area (step S43; YES), the identification information synthesizing unit 303 next determines whether or not there are a plurality of candidate areas (step S45). If there is only one candidate area (step S45; NO), the identification information synthesizing unit 303 determines the extracted candidate area as the display area (step S46), and terminates the identification information display area determination process. If there are a plurality of candidate areas (step S45; YES), the identification information synthesizing unit 303 performs a display area selection process (step S47) for selecting a display area from among the plurality of candidate areas, and then selects the identification information display area. End decision processing.

As the display area selection process, the identification information synthesizing unit 303 performs, for example, a process according to the flowchart of FIG. When starting the display area selection process, the identification information synthesizing unit 303 determines whether or not there is a candidate area in which the date and time and all of the identification information can be displayed in the maximum character size among the plurality of candidate areas (step S4701). . The maximum character size is the maximum character size among multiple character sizes that can be used to display identification information. is. If there is a candidate area that can be displayed (step S4701; YES), the identification information synthesizing unit 303 selects one of the candidate areas that can display all of the date and time and the identification information with the maximum character size as the display area (step S4702). ) to end the display area selection process. If there is one candidate area that can be displayed, the identification information synthesizing unit 303 selects that one candidate area as the display area in step S4702. If there are a plurality of candidate areas that can be displayed, in step S4702, the identification information synthesizing unit 303 selects, among the plurality of candidate areas, a candidate area that is the same as or closest to the position of the display area at the current point in time after the update. Elect as display area. In this case, the identification information synthesizing unit 303 that has finished the display area selection process finishes the identification information display area determination process illustrated in FIG. 3, and performs the process of step S5 illustrated in FIG. In step S5, the identification information synthesizing unit 303 synthesizes a label in which date and time and identification information are described in the maximum character size in the display area of the image to be processed selected in step S4702.

If there is no candidate area that can display all the date and time and the identification information with the maximum character size (step S4701; NO), the identification information synthesizing unit 303 next displays all the date and time and the identification information with a character size equal to or larger than the minimum character size. (step S4703). The minimum character size is the smallest size among a plurality of character sizes that can be used for displaying identification information, and is set, for example, in consideration of the readability of identification information combined with an image. If there are candidate areas that can be displayed (step S4703; YES), the identification information synthesizing unit 303 selects one of the corresponding candidate areas that can maximize the character size as a display area (step S4704). , to end the display area selection process. If there is one corresponding candidate area, the identification information synthesis unit 303 selects the one candidate area as the display area in step S4704. If there are a plurality of applicable candidate areas, as alternative processing in step S4704, the identification information synthesizing unit 303 selects, for example, a candidate area that is the same as or closest to the position of the current display area as the post-update display area. may

If there is no candidate area that can display all of the date and time and the identification information with a character size equal to or larger than the minimum character size (step S4703; NO), the identification information synthesizing unit 303, for example, selects a candidate area as an identification information display area, A candidate area for the date and time display area is selected (step S4705), and the display area selection process ends. For example, if there is no candidate area that can display all of the date and time and the identification information together in one candidate area due to factors such as the number of characters of the identification information to be displayed is large, the date and time and The identification information is individually displayed in different candidate areas (display areas) within the image. By separately displaying the date and time and the identification information, it is possible to prevent difficulty in reading the identification information due to the character size becoming too small.

Note that the identification information display area determination processing performed by the image processing device 3 (more specifically, the identification information synthesizing unit 303) is not limited to the processing described above with reference to FIGS. 3 and 4, and can be changed as appropriate.

As described above, in the image processing apparatus 3 of the present embodiment, when the date and time and the identification information are combined with the image and displayed, the date and time and the identification information surround the object to be recognized (more specifically, the object to be recognized). The position and dimensions of the display area for the date and time and identification information are determined so that they do not overlap with the area inside the frame. Therefore, even when there are a plurality of recognition target objects recognized from an image and there is no priority order for the plurality of objects, it is possible to reduce overlap of the date and time and the identification information with the recognition target objects. In addition, by performing the process of determining (updating) the position and size of the display area only when the predetermined conditions as described above are satisfied, the processing can be compared with the process of determining the position and size of the display area for each image. Therefore, an increase in the processing load of the image processing device 3 can be suppressed.

A specific example of the image processing system 1 according to the present embodiment will be described below with reference to FIGS. 5 to 9. FIG.

FIG. 5 is a diagram showing an example of a composite image displayed on the display device. FIG. 6 is a diagram illustrating an example of a problem that may occur when objects to be recognized increase. FIG. 7 is a diagram showing an example of a composite image after the date and time and identification information display areas have been updated. FIG. 8 is a diagram illustrating an example of a method for extracting candidate areas when updating the display area of the date and time and identification information. FIG. 9 is a diagram showing an example of a composite image in which date and time and identification information are displayed in different display areas.

A synthesized image 1000 illustrated in FIG. 5 is an image obtained by synthesizing an image (original image) of a desk to be monitored by the camera 2 with a frame indicating a recognized object to be recognized, identification information, and the like. When the objects to be recognized include a keyboard, a pencil, and an eraser, the object recognition unit 302 of the image processing device 3 recognizes the keyboard 11, the pencil 12, and the eraser 13 appearing in the original image as objects to be recognized. . At this time, the identification information synthesizing unit 303 creates a first frame 1011 surrounding the keyboard 11, a second frame 1012 surrounding the pencil 12, a third frame 1013 surrounding the eraser 13, date and time, and identification of the object to be recognized. A synthesized image 1000 is generated by synthesizing a label 1050 in which information is described with the original image. The label 1050 is displayed so as not to overlap any of the first frame 1011 , the second frame 1012 and the third frame 1013 .

After some time has passed after the synthetic image 1000 is displayed, the state of the desk changes, and for example, the cup 14 is recognized as a new object to be recognized, as shown in the synthetic image 1001 shown in FIG. There is something. In this case, in the conventional image processing apparatus, as shown in FIG. 6, the position and dimensions of the label 1050 with the date and identification information in the image do not change, and the label 1050 overlaps part of the cup 14 . There is When such a composite image 1001 is displayed on the display device 4, the state of the portion of the cup 14 overlapping the label 1050 cannot be visually recognized. In order to prevent such overlap between the recognition target object (cup 14) and the date and identification information (label 1050), the image processing apparatus 3 of the present embodiment, for example, recognizes the recognition target from the image to be processed. When the number of objects is greater than the number of objects to be recognized from the image processed immediately before, identification information display area determination processing (step S4) is performed, and the date and identification information are written. The display position of the label 1050 is updated. In the synthesized image 1001 illustrated in FIG. 6, an area (candidate area) 1042 in which the label 1050 can be displayed so as not to overlap the area within the first frame 1011 and the area within the fourth frame 1014 in the upper right portion of the image. There is For this reason, in the image processing system 1 of the present embodiment, when the cup 14 is newly recognized as an object to be recognized, a label 1050 with date and identification information is displayed as in the composite image 1002 illustrated in FIG. is changed from lower right to upper right. Therefore, it is possible to generate a composite image 1002 in which the date and identification information (label 1050) do not overlap with any of the keyboard 11, pencil 12, eraser 13, and cup 14, which are objects to be recognized, and display it on the display device 4. .

In the identification information display area determination process (step S4) for updating the display area of the date and identification information (label 1050), for example, as shown in FIG. The area 1020 that can be set as the display area (label 1050) is divided into a plurality of blocks. It is derived as a region (step S41). In the example shown in FIG. 8, a set 1031 of blocks including the portion to be the area within the first frame 1011, a set 1032 of blocks including the portion to be the area within the second frame 1012, and a set 1032 of blocks including the portion within the third frame 1013 A set 1033 of blocks including a portion to be the area of , and a set 1034 of blocks including a portion to be the area within the fourth frame 1014 are shaded. That is, in the example shown in FIG. 8, the white portion within the area 1020 becomes the out-of-frame area. Note that the size (the number of pixels) of one block in the region 1020 can be changed as appropriate, and one pixel may be one block.

After deriving the out-of-frame area, the image processing device 3 extracts a candidate area as a candidate for the date and identification information (label 1050) display area from the out-of-frame area (step S42). If the condition for extracting the candidate area is, for example, a rectangular area of 3 or more horizontal blocks by 3 or more vertical blocks, which is in contact with the outer circumference of the area 1020 that can be set as the display area for the date and identification information (label 1050). Four candidate regions 1041, 1042, 1043 and 1044 shown in 8 are extracted. The image processing device 3 (more specifically, the identification information synthesizing unit 303) selects the optimum candidate area for displaying the date and identification information from the extracted four candidate areas 1041, 1042, 1043, and 1044. to select the display area. At this time, the image processing device 3 may determine the display area by performing the display area selection process as described above with reference to FIG. 4, or may determine the display area by another process. For example, the image processing device 3 may simply use the candidate area with the largest size (the number of blocks) as the date and identification information display area. Such processing can suppress an increase in the processing load of the image processing device 3 by setting the size of one block to 2 pixels or more in the horizontal direction and 2 pixels or more in the vertical direction. Pixels may be performed as one block.

After a certain amount of time has passed after the synthetic image 1002 illustrated in FIG. 7 is displayed, the state of the desk changes. may change. When objects to be recognized (keyboard 11, pencil 12, eraser 13, and cup 14) are positioned as illustrated in FIG. 9, one label 1050 illustrated in FIG. 7, for example, is displayed from within the frame-outside area. possible candidate regions cannot be extracted. For this reason, the image processing apparatus 3, for example, as shown in FIG. and By dividing the display area for displaying the date and the display area for displaying the identification information in this way, the information (the number of characters) to be displayed in one display area is reduced. becomes possible.

Note that the image processing apparatus 3 of the present embodiment is not limited to a dedicated apparatus that performs the functions and processes described above. It may be implemented by a program to be executed.

FIG. 10 is a diagram illustrating a hardware configuration example of a computer. The computer 6 illustrated in FIG. 10 includes a processor 601 , memory (main storage device) 602 , auxiliary storage device 603 , input device 604 , display device 605 , input/output interface 606 , media drive device 607 , and communication interface 608 . These elements of computer 6 can send data to and receive data from each other, such as via bus 610 .

The processor 601 functions (operates) as the image processing apparatus 3 described above by executing an OS (Operating System) program and a program including processes illustrated in FIGS. controls the behavior of The processor 601 includes, for example, one or more CPUs (Central Processing Units). The processor 601 may include, for example, a DSP (Digital Signal Processor). A processor 601 that executes a program including instructions corresponding to the processes described above with reference to FIGS. It is an example of hardware that implements the functions of the image output unit 304 . The program to be executed by the processor 601 may include a program including processing for learning the features of the object to be recognized using the learning image and generating the trained model 311 .

The memory 602 is a main storage device that stores a part of the program executed by the processor 601, data created by the processor 601 during execution of the program, data to be referred to, and the like. The memory 602 includes RAM (Random Access Memory) 602a and ROM (Read Only Memory) 602b. The auxiliary storage device 603 is a storage device having a larger storage capacity than the memory 602, such as a HDD (Hard Disk Drive) or an SSD (Solid State Drive). The memory 602 and the auxiliary storage device 603 provide storage areas for storing programs to be executed by the processor 601 and various data used when the processor 601 executes the programs. For example, the auxiliary storage device 603 is an example of hardware that implements the functions of the storage unit 310 of the image processing device 3, and provides a storage area for storing the learned model 311 and the synthesized image data 312 described above. The memory 602 and auxiliary storage device 603 are examples of non-temporary recording media that store programs that cause a computer to execute the processes described above with reference to FIGS.

The input device 604 is an operation input device, such as a keyboard device and a mouse device, that receives an operation input to the computer 6 . The display device 605 is a display device such as a liquid crystal display (for example, the display device 4 in FIG. 1) capable of presenting a synthesized image to the user. The input/output interface 606 is a hardware interface that receives input signals from an external device such as the camera 2 and outputs output signals to an output device (for example, a printer) (not shown). The input/output interface 606 is an example of hardware that implements the functions of the image acquisition unit 301 of the image processing apparatus 3 illustrated in FIG.

The medium drive device 607 reads information stored in the portable recording medium 650 and writes information to the portable recording medium 650 . The portable recording medium 650 includes, for example, optical discs, magnetic discs, magneto-optical discs, and memory cards. The portable recording medium 850 may be read-only, or may be writable or rewritable. A portable recording medium 650 can be used to provide a storage area for storing the trained model 311 . A combination of the media drive device 607 and the portable recording medium 650 can implement the function of the storage unit 310 of the image processing device 3 . Portable recording medium 650 is an example of a non-temporary recording medium that stores a program that causes a computer to execute the processes described above with reference to FIGS. Media drive 607 may be connected to bus 610 or the like via input/output interface 606 .

The communication interface 608 connects the computer 6 to a communication network such as the Internet and controls communication with external devices via the communication network. Communication interface 608 is connected to a communication network through wired or wireless communication. The communication interface 608 can be used, for example, to acquire learning images and trained models 311 stored in a server device connected to a communication network, and to transfer synthesized image data 312 to the server device. . Also, the computer 6 and the camera 2 may be connected using the communication interface 608 instead of the input/output interface 606 .

The hardware configuration of the computer 6 illustrated in FIG. 10 is merely an example of the hardware configuration of a computer that can be used as the image processing device 3. FIG. A computer that can be used as the image processing device 3 may not include some of the components illustrated in FIG. 10 (for example, the media drive device 607 and the communication interface 608). The computer may also include, for example, an FPGA (Field Programmable Gate Array) or an ASIC (Application Specific Integrated Circuit) that performs part of the processing described above with reference to FIGS.

The above-described embodiments are specific examples for easy understanding of the invention, and the present invention is not limited to the above-described embodiments. The image processing apparatus, image processing method, and program according to the present invention can be modified and modified in various ways without departing from the scope of the claims.

For example, if there is an area in the image to be processed where the object to be recognized is not recognized for a long period of time (area that is not captured), the identification information of the recognized object and information on the date and time in the area where the object is not recognized. may determine the position and size of the display area for displaying the The region where the recognition target object is not recognized for a long period of time may be, for example, a fixed region specified based on the imaging range θ of the camera 2, or may be a past processing (for example, step S41 illustrated in FIG. 3). and the process of S42)) may vary based on the history of the areas extracted as candidate areas.

The invention described in the scope of claims at the time of filing of the present application will be additionally described below.
[Appendix 1]
an object recognition unit that recognizes an object to be recognized in an image;
an identification information synthesizing unit that synthesizes identification information about the object to be recognized recognized by the object recognition unit with the image;
including
The identification information synthesizing unit derives a region that does not overlap with the recognition target object based on the position and size of the recognition target object in the image, and displays the identification information so as to fit within the derived region. An image processing device that determines the position and size of a display area to be displayed.
[Appendix 2]
The identification information synthesizing unit derives the area that does not overlap with the object to be recognized when a predetermined condition associated with updating the position and size of the display area is satisfied, and places the area within the derived area. The image processing apparatus according to appendix 1, wherein the position and size of the display area in which the identification information is displayed are determined so as to fit.
[Appendix 3]
the identification information includes character information indicating the type of the object to be recognized;
3. The image processing apparatus according to appendix 1 or 2, wherein the identification information synthesizing unit adjusts a character size of the identification information to be displayed based on the determined position and size of the display area.
[Appendix 4]
The identification information synthesizing unit adjusts the character size of the identification information within a range from a predetermined maximum character size to a minimum character size set as the character size of the identification information to be displayed in the display area. The image processing apparatus according to appendix 3, characterized by:
[Appendix 5]
The identification information synthesizing unit extracts a candidate for a rectangular display area having a predetermined size or more in the area that does not overlap with the derived object to be recognized, and selects the display area from among the extracted candidates for the display area. 5. The image processing apparatus according to any one of appendices 1 to 4, characterized in that an area is selected.
[Appendix 6]
The identification information synthesizing unit synthesizes information indicating the date and time when the image was captured together with the identification information in the display area of the image,
When the identification information synthesizing unit cannot determine a single display area capable of displaying all of the information indicating the date and time and the identification information, the identification information synthesizing unit selects a first display area for displaying the information indicating the date and time. 6. The image processing apparatus according to appendix 5, wherein the position and dimensions of the second display area for displaying the identification information are determined.
[Appendix 7]
The predetermined condition is recognized by a first image currently being processed and a second image captured temporally before the first image and to which the identification information has already been synthesized. 7. The image processing apparatus according to any one of appendices 2 to 6, wherein at least one of the number and positions of the objects to be recognized are different.
[Appendix 8]
8. A statement according to any one of appendices 2 to 7, wherein the predetermined condition includes that the number of images synthesized with the identification information is a predetermined value or a positive integral multiple of that value. image processing device.
[Appendix 9]
the computer
recognizing an object to be recognized in an image;
synthesizing identification information about the recognized object to be recognized with the image;
and run
Synthesizing the identification information with the image includes deriving a region that does not overlap with the recognition target object based on the position and size of the recognition target object in the image, determining the position and dimensions of a display area for displaying the identification information;
An image processing method characterized by:
[Appendix 10]
to the computer,
recognizing an object to be recognized in an image;
synthesizing identification information about the recognized object to be recognized with the image;
and
Synthesizing the identification information with the image includes deriving a region that does not overlap with the recognition target object based on the position and size of the recognition target object in the image, determining the position and dimensions of a display area for displaying the identification information;
A program characterized by

1 image processing system 2 camera 3 image processing device 301 image acquisition unit 302 object recognition unit 303 identification information synthesis unit 304 image output unit 310 storage unit 311 trained model 312 synthesized image data 4 display device 6 computer 601 processor 602 memory 602a RAM
602b ROM
603 Auxiliary storage device 604 Input device 605 Display device 606 Input/output interface 607 Media drive device 608 Communication interface 610 Bus 650 Portable recording medium 1000, 1001, 1002, 1003, 1004 Synthetic image 11 Keyboard 12 Pencil 13 Eraser 14 Cup 1011, 1012 , 1013, 1014 Frames 1041, 1042, 1043, 1044 Candidate areas 1050, 1051, 1052 Labels

Claims (10)

an object recognition unit that recognizes an object to be recognized in an image;
an identification information synthesizing unit that synthesizes identification information about the object to be recognized recognized by the object recognition unit with the image;
including
The identification information synthesizing unit derives a region that does not overlap with the recognition target object based on the position and size of the recognition target object in the image, and displays the identification information so as to fit within the derived region. An image processing device that determines the position and size of a display area to be displayed. The identification information synthesizing unit derives the area that does not overlap with the object to be recognized when a predetermined condition associated with updating the position and size of the display area is satisfied, and places the area within the derived area. 2. The image processing apparatus according to claim 1, wherein the position and size of said display area for displaying said identification information are determined so as to fit therein. the identification information includes character information indicating the type of the object to be recognized;
3. The image processing apparatus according to claim 1, wherein the identification information synthesizing unit adjusts the character size of the identification information to be displayed based on the determined position and size of the display area. The identification information synthesizing unit adjusts the character size of the identification information within a range from a predetermined maximum character size to a minimum character size set as the character size of the identification information to be displayed in the display area. 4. The image processing apparatus according to claim 3. The identification information synthesizing unit extracts a candidate for a rectangular display area having a predetermined size or more in the area that does not overlap with the derived object to be recognized, and selects the display area from among the extracted candidates for the display area. 5. The image processing apparatus according to any one of claims 1 to 4, wherein an area is selected. The identification information synthesizing unit synthesizes information indicating the date and time when the image was captured together with the identification information in the display area of the image,
When the identification information synthesizing unit cannot determine a single display area capable of displaying all of the information indicating the date and time and the identification information, the identification information synthesizing unit selects a first display area for displaying the information indicating the date and time. 6. The image processing apparatus according to claim 5, wherein the position and dimensions of the second display area for displaying the identification information are determined. The predetermined condition is recognized by a first image currently being processed and a second image captured temporally before the first image and to which the identification information has already been synthesized. 7. The image processing apparatus according to any one of claims 2 to 6, wherein at least one of the number and positions of objects to be recognized are different. 8. The method according to any one of claims 2 to 7, wherein the predetermined condition includes that the number of images synthesized with the identification information is a predetermined value or a positive integral multiple of that value. The described image processing device. the computer
recognizing an object to be recognized in an image;
synthesizing identification information about the recognized object to be recognized with the image;
and run
Synthesizing the identification information with the image includes deriving a region that does not overlap with the recognition target object based on the position and size of the recognition target object in the image, determining the position and dimensions of a display area for displaying the identification information;
An image processing method characterized by: to the computer,
recognizing an object to be recognized in an image;
synthesizing identification information about the recognized object to be recognized with the image;
and
Synthesizing the identification information with the image includes deriving a region that does not overlap with the recognition target object based on the position and size of the recognition target object in the image, determining the position and dimensions of a display area for displaying the identification information;
A program characterized by

Images