JP7282519B2 - Image processing device or image processing server - Google Patents

Description

The present invention relates to an image processing apparatus and the like for photographing and video monitoring.

In recent years, as internationalization advances, many tourists are visiting Japan. Also, in competitive sports, opportunities for photographing athletes from various countries have increased significantly.

However, it is relatively difficult for both professional photographers and ordinary photographers to find a particular athlete from among a large number of athletes in, for example, a competitive sports scene. In addition, it is often the case that a plurality of athletes move quickly during a competition in a competitive sport, and the location of the athlete is often lost. The same applies not only to competitive sports, but also to the case of photographing or monitoring a specific person in a crowd.

Patent Literature 1 describes a plurality of cameras for photographing a subject from a plurality of directions, and a plurality of image processing devices for extracting predetermined regions from images captured by corresponding cameras among the plurality of cameras. Also, an image generation device is described that generates a virtual viewpoint image based on image data of a predetermined region extracted from images captured by the plurality of cameras by the plurality of image processing devices.
Patent Document 2 describes an automatic focus detection device that drives a focus lens based on an AF evaluation value obtained from a captured image and performs automatic focus detection control.

JP 2017-211828 A Patent No. 5322629

However, in a sports scene where many players are gathered, the players may be overlapped and lost. It also made it more difficult to capture the players at the right time, as they were often out of view.
Furthermore, recently, when a photographer takes a picture, he/she often leaves a memo associated with the picture, for example, by voice. However, there is a drawback that when the intention of shooting is left as metadata information, it cannot be left in cooperation with other metadata. In addition, depending on how the game progresses, the result may differ from the contents of the memos taken by the cameraman. In this case, there is also a problem that the memo of the metadata that is different from the intention of the cameraman's shooting is left as it is.

Moreover, in particular, professional photographers need to immediately send the photographs they have taken to news bureaus or the like. Furthermore, even if I found a player I wanted to shoot, I had to focus on that player and then chase after the player I wanted to shoot. This is very difficult in fast-moving sports, and has the drawback of not being able to take good pictures if you are concentrating on this chasing.

In addition, on the server side, it is possible to grasp omnidirectional video and various information about the game and the field of competition, and it is possible to obtain various valuable information inside and outside the ground, but the conventional system does not make full use of the server. There was a problem that it was not
Similarly, there were many cases in which general users who were monitoring the game on stadiums or home terminals lost track of specific athletes or lost track of the game. Similarly, in car races, airplane races, horse races, and the like, there are times when a specific vehicle, airplane, horse, or the like is lost. Furthermore, when tracking a specific person on a street corner, the specific person may get lost in the crowd and be lost. In addition, there is a problem that if the user is concentrating on visually following such a specific object of interest, shooting, focusing, exposure adjustment, etc. for the object cannot be performed smoothly.

Moreover, professional photographers often add metadata to images, but the task is complicated and time-consuming, and the accuracy of the contents of the memos may be poor. Furthermore, there are cases where metadata at a certain timing should be corrected or changed over time, but there is a problem that the metadata cannot be changed easily.
SUMMARY OF THE INVENTION It is an object of the present invention to solve the above problems and to provide an image processing apparatus capable of adding useful information in a timely manner and correcting it as appropriate.

In the image processing device,
display means for displaying an image;
selection means for selecting a specific target from among the images displayed on the display means;
Designation information generation means for generating designation information regarding the specific target selected by the selection means;
transmitting means for transmitting the specified information generated by the specified information generating means to a server;
Acquisition means for acquiring location information of the specific target based on the specified information from the server and predetermined metadata created in the server;
Control means for displaying additional information based on the position information of the specific target acquired by the acquisition means or information related to the metadata on the display means;
characterized by having

According to the present invention, it is possible to provide an image processing apparatus capable of adding useful information in a timely manner and correcting it appropriately.

1 is a block diagram of an entire system including an image processing apparatus of an embodiment; FIG. It is a block diagram on the server side. 3 is a detailed block diagram of the terminal side; FIG. 3 is a detailed block diagram of the terminal side; FIG. 3 is a block diagram showing a functional configuration example of a tracking unit 371 on the terminal side; FIG. FIG. 10 is a diagram showing an example of a player-of-interest display start sequence; FIG. 13 is a diagram showing another example of a player-of-interest display start sequence; FIG. 4 is a diagram showing an example of a file structure; 4 is a flow chart showing an example of control on the camera side; 10 is a flowchart showing an example of player-of-interest display tracking control on the camera side; 10 is a flowchart showing an example of player-of-interest display tracking control on the camera side; FIG. 11 is a flowchart showing an example of a player-of-interest detection control on the server side; FIG. FIG. 11 is a flowchart showing an example of a player-of-interest detection control on the server side; FIG. FIG. 11 is a flowchart showing an example of a player-of-interest detection control on the server side; FIG. FIG. 11 is a flowchart showing an example of a player-of-interest detection control on the server side; FIG. FIG. 11 is a flowchart showing an example of a player-of-interest detection control on the server side; FIG. FIG. 11 is a flowchart showing an example of a player-of-interest detection control on the server side; FIG. FIG. 10 is a flowchart showing an example of try determination control on the server side; FIG. FIG. 10 is a flowchart showing an example of try determination control on the server side; FIG. FIG. 10 is a diagram showing a refereeing operation for judging whether or not there is a try; FIG. 10 is a flowchart showing an example of try determination control on the server side; FIG. FIG. 10 is a flowchart showing an example of try determination control on the server side; FIG. 10 is a flowchart showing an example of try determination control on a terminal side such as a camera; 4 is a flowchart showing an example of foul determination control on a terminal side such as a camera;

Modes for carrying out the present invention will be described below using examples.

First, an overview of a system using an image processing device for supporting shooting and video monitoring in Embodiment 1 will be described with reference to FIG. FIG. 1 is a block diagram of the entire system including the image processing apparatus of the embodiment.

In Figure 1, the server (image processing server) side, which has multiple cameras for the server (fixed cameras and mobile cameras using drones, etc.), can determine the position of the player of interest (specific target) in the entire field of the stadium and the game. to grasp the latest situation in real time. Then, an example is shown in which the server timely provides necessary information, for example, for camera shooting or image monitoring, to the terminal held by each spectator.

In general, there are many places that cannot be seen or tracked by the angles and fields of view taken by professional photographers and general photographers. The same is true for spectators outside the stadium who are not filming. On the other hand, in the server-side system, it is possible to grasp and map in advance omnidirectional images and information on the entire game field (field coordinate information, etc.) based on images from multiple server cameras.
Therefore, the server grasps and distributes information that is invisible and difficult for each user to understand, so that the service to the audience can be greatly improved.

That is, multiple cameras (fixed cameras and moving cameras) for the server can track the position of each player, scores, fouls, judgment results of referees, and other up-to-date situations. The information displayed on a large screen can also be analyzed by a server. As a result, the overall situation can be accurately recognized and timely transmitted to camera terminals, etc. owned by professional photographers and spectators, as well as terminals such as smartphones and tablets. As a result, the spectators can grasp the latest situation of the competition in a timely manner. In particular, professional photographers need to send the photos they have taken immediately to the news bureau, etc., but it is difficult to accurately grasp the overall situation of the competition just by looking at the camera screen because the field of view is narrow. However, by using the configuration of the present embodiment, the conditions of the game can be known quickly, so that it is possible to quickly select photographs to be sent to the news bureau or the like.

Terminals (image processing devices) used by professional photographers and spectators include a digital camera, a smart phone, a configuration in which a camera and a smart phone are connected, a tablet PC, a TV, and the like. A similar service can be provided to spectators who are watching the competition at home through a terminal (image processing device) such as a PC or TV through the Internet or television broadcasting, so that the situation of the competition can be grasped more accurately. You can enjoy the competition more.

In FIG. 1, 101 to 103 are cameras for servers, 101 (fixedly installed camera 1), 102 (fixedly installed camera 2), 103 (fixedly installed camera 3), 104 (large screen). , 110 (server), 111 (input means), and 112 (base station) acquire video and audio for providing information to general professional photographers and spectators. In this embodiment, there are three server cameras 101 to 103, but one or more cameras may be used. Also, these server cameras may be cameras mounted on, for example, drones instead of fixedly installed cameras. In addition to acquiring video and audio, input information other than video (for example, audio) can be taken in from the input means, and services for general professional photographers and spectators can be expanded.

Reference numeral 105 denotes a wired/wireless LAN or Internet, and 106 denotes a connection line for inputting information output from the input means 111 to the server 110 . Reference numeral 107 denotes a connection line for transmitting/receiving a signal to/from the base station 112, and 108 denotes an antenna section for performing wireless communication with the base station.
That is, the blocks in the 100s are for supporting video shooting by professional photographers and general spectators.

On the other hand, in FIG. 1, 401 (terminal 1), 402 (terminal 2), and 403 (terminal 3) are terminals. It is a video display terminal device such as Here, 404 (antenna), 405 (antenna), and 406 (antenna) are antennas for wireless communication by 401 (terminal 1), 202 (terminal 2), and 203 (terminal 3), respectively.

When the server detects the position of the player of interest, for example, the terminal sends the ID information of the player of interest to the server side, and the server side sends various information such as the position information of the player to the terminal. Since the players are moving and the game situation is changing, it is necessary to detect the player of interest in a short period of time. Therefore, for wireless communication here, 5G or the like is used, for example.

Note that 401 (terminal 1), 402 (terminal 2), and 403 (terminal 3) may have a configuration in which a camera and a smart phone are connected and combined. In the lower right of FIG. 1, 301 is a smart phone, which mainly controls communication with the server. By installing application software on this smartphone, various video acquisition services can be realized. Reference numeral 300 denotes a (digital) camera, which is an image processing device mainly used by professional photographers and spectators to take pictures and monitor images. Here, the camera 300 is connected to the smartphone 301 via USB, Bluetooth (registered trademark), or the like. An antenna 320 is used by the smartphone 301 to wirelessly communicate with the base station 112 .

If the terminal is a smartphone, etc., the exchange of video and control signals with the server is performed wirelessly. good. For example, if the wireless communication environment is 5G, communication is performed wirelessly, and if the wireless communication environment is LTE, information with a large amount of data is sent by wire, and control signals with a small amount of data are sent wirelessly. Furthermore, it is also possible to switch to wired communication depending on how busy the wireless communication line is.

Next, FIG. 2 is a detailed block diagram of the server side, and FIG. 2 will be described. In FIG. 2, the same reference numerals as in FIG. 1 indicate the same configuration, and the description thereof is omitted.
201 is an Ethernet (registered trademark) controller, and 204 is a detection unit that detects a play position according to a player's role (so-called position). Here, the roles (positions) of players are set in advance by registration or the like. For example, as a role (position) in the case of rugby, 1 and 3 are props, 2 are hookers, 4 and 5 are locks, 6 and 7 are flanker, 8 is called number 8, 9 is scrum half, 10 is called called a standoff. Also, 11 and 14 are called wings, 12 and 13 are called centers, and 15 is called a fullback.
As for where these players are, during set-pieces, the forwards are often in the attacking front and the backs are in the attacking back.

In other words, the player's position is generally determined according to the player's role (position), so it is better to follow the player after grasping the above-mentioned role (position) of the player of interest, and to chase the player more effectively and accurately. can be done.
Usually, the player's role can often be recognized from the uniform number. However, in some cases, the No. 10 player gets injured, the No. 15 player stands off (enters the No. 10 position), the reserve player enters No. 15, and so on. Here, reserve players have uniform numbers from 16 to 23. However, the position is not decided only by the jersey number. Accordingly, the detection unit 204 detects the play position corresponding to the player's preset role, and information on the detected play position is taken into the CPU 211 in the server 110. There may be some changes due to player substitutions.

Reference numeral 205 denotes a contour information detection unit. For example, when a professional cameraman or spectator monitors an image on a terminal and takes a picture from that position and angle at the magnification of the camera, the server 110 detects the position of the player of interest. Notify the terminals 401 to 403 and the like. In addition, by notifying the terminals 401 to 403 of the profile information of the captured player of interest from the server 110, the terminals 401 to 403 can recognize the player of interest more reliably. The contour information detected by the block 205 is taken into the CPU 211 .

Reference numeral 206 denotes a player's face recognition unit, which finds the player from the video using AI, particularly image recognition technology such as Deep Learning, based on pre-registered facial photograph information of the player of interest. Information on the result of face recognition detected by the face recognition unit 206 is also taken into the CPU 211 .
A player physique recognition unit 207 finds a player based on pre-registered physique photo information of a player of interest using the above-described image recognition technology.

Reference numeral 208 denotes a player's uniform number detection unit, which uses the image recognition technology described above to find a player of interest from a number (such as uniform number) registered in advance. Needless to say, when detecting the player's number, not only the number on the back side of the bibs but also the number written on the front side may be detected. A position information generating unit 209 recognizes the position, direction, and angle of view of each camera from the position information of the cameras 101, 102, 103, etc. using GPS, etc., and the information on the direction and angle of view of the camera. Then, based on the images from each camera, the absolute positional information of the player on the ground is obtained by triangulation. The position information generating unit 209 also acquires positions in the screen such as poles and lines (for example, sidelines and endlines) of the competition field, which are set in advance in the stadium as reference indicators for detecting reference positions, from the video in advance. You can leave it.

Then, using those coordinates as reference coordinates, the absolute position of the player of interest in the stadium with respect to the field may be acquired.
210 is camera position information for detecting the position of each terminal and the direction and angle of view of the camera of each terminal from the position information, direction information and angle of view information of each terminal sent from each terminal 401 to 403; This is the direction detector.
Reference numeral 211 denotes a CPU (Central Processing Unit) as a computer, which is a central processing unit that executes control shown in the following embodiments based on a control computer program stored in a program memory 712 as a storage medium. .

It also serves as display control means, and controls information to be displayed on a display unit 214, which will be described later. A data memory 213 stores various data referred to by the CPU 211 . The data memory 213 stores information on past games, information on past players, information on today's game (competition), information on the number of spectators, information on the weather, information on featured players, current situations of players, and the like. . The information on the featured player includes information such as the face, jersey number, and physique.
1101 is a data bus line within the server 110 .

Next, details of the terminal side (camera) as an image processing device will be described with reference to FIGS. 3 and 4. FIG. FIG. 3 and FIG. 4 are block diagrams showing configuration examples of a terminal, and show the overall configuration of a (digital) camera 500 as an example of a terminal using two drawings.
The terminal side (camera) as the image processing device shown in FIGS. 3 and 4 is capable of shooting moving images and still images, and recording shooting information. 3 and 4, a CPU (Central Processing Unit) 318, a program memory 319, and a data memory 320 are shown redundantly, but they are the same block and only one is built in each.
In FIG. 3, 301 is an Ethernet (registered trademark) controller. A storage medium 302 stores moving images and still images (photographs) captured by a digital camera in a predetermined format.

Reference numeral 303 denotes an image sensor such as a CCD or CMOS, which converts an optical image from a light signal to an electric signal, further converts this information from analog information to digital data, and outputs the data. A signal processing unit 304 performs various corrections such as white balance correction and gamma correction on the digital data output from the image sensor 303 and outputs the data. A sensor driving unit 305 controls horizontal and vertical line driving for reading out information from the image sensor 303 and the timing at which the image sensor 303 outputs digital data.

Reference numeral 306 denotes operation unit input means. Inputs are made in response to selection and setting of various conditions for photographing with a digital camera, trigger operation for photographing, selection operation for using the flash, battery replacement operation, and the like. Also, in the operation unit input means 306, it is possible to select/set whether or not to perform AF (autofocus) on the player of interest based on the position information from the server. Selection/setting information as to whether or not to AF (autofocus) this player of interest is output from the operation unit input means 306 to the bus line 370 .
Furthermore, in the operation unit input means 306, it is possible to select/set whether or not to automatically track the player of interest based on the position information from the server. Information such as which player is to be designated as a player of interest (specific target) and whether or not automatic tracking of the player of interest is to be executed based on position information from the server is generated by the operation unit input means 306 as selection means. be. That is, the operation unit input means 306 functions as a designation information generation means for generating designation information regarding a specific object.

Reference numeral 307 denotes a wireless communication unit that functions as a transmission/reception unit, and is used for wireless communication between camera terminals possessed by professional photographers and general spectators with the server side. A magnification detection unit 308 detects the imaging magnification of the digital camera. Reference numeral 309 denotes an operation unit output unit for displaying UI information such as menus and setting information on an image display unit 380 that displays image information captured by a digital camera or the like. 310 is a compression/decompression circuit. After the digital data (RAW data) from the image sensor 303 is developed by the signal processing unit 304, it is compressed by the compression/decompression circuit 310 into a JPEG image file or HEIF image file, or RAW data is compressed as it is to form a RAW image file. On the other hand, when a RAW image file is developed within the camera to generate a JPEG image file or HEIF image file, processing is performed to decompress the compressed information and return it to RAW data.

Reference numeral 311 denotes a face recognition unit, which refers to face photograph information registered in advance in the server for a player of interest, and finds the player in the video by image recognition using AI, particularly, technology such as Deep Learning. is. Information on the result of face recognition detected by face recognition section 311 is taken into CPU 318 via bus line 370 .

A physique recognition unit 312 refers to the physique photo information registered in advance in the server regarding the player of interest, and finds the player of interest from the video using the above-described image recognition technology.
Reference numeral 313 denotes a player's uniform number detection unit, which finds a player from the uniform number of the player of interest (of course, the number on the front side may also be used) by the above-described image recognition technology. A direction detection unit 314 detects the direction in which the lens of the terminal faces. A position detection unit 315 detects the position information of the terminal using, for example, GPS.

A power management unit 316 detects the state of the power supply of the terminal, and supplies power to the entire terminal when it detects that the power button is pressed while the power switch is off. Reference numeral 318 denotes a CPU as a computer, which executes control shown in the following embodiments based on a computer program for control stored in a program memory 319 as a storage medium. It also serves as display control means and controls image information displayed on the image display section 380 . Note that the image display unit 380 is a display unit using liquid crystal, organic EL, or the like.
The data memory 320 is for storing the setting conditions of the digital camera, photographed still images, moving images, and attribute information of the still images and moving images.

In FIG. 4, 350 is a photographing lens unit which has a fixed 1st group lens 351 , a zoom lens 352 , an aperture 355 , a fixed 3rd group lens 358 , a focus lens 359 , a zoom motor 353 , an aperture motor 356 and a focus motor 360 . A fixed 1st group lens 351, a zoom lens 352, an aperture 355, a fixed 3rd group lens 358, and a focus lens 359 constitute a photographing optical system. For convenience, each of 351, 352, 358, and 359 is illustrated as one lens, but each may be composed of a plurality of lenses. Also, the photographing lens unit 350 may be configured as an interchangeable lens unit that can be attached to and detached from the digital camera.

A zoom control unit 354 controls the operation of the zoom motor 353 to change the focal length (angle of view) of the photographing lens unit 350 . A diaphragm control unit 357 controls the operation of a diaphragm motor 356 to change the aperture diameter of the diaphragm 355 .

The focus control unit 361 calculates the defocus amount and defocus direction of the photographing lens unit 350 based on the phase difference between the pair of focus detection signals (A image and B image) obtained from the image sensor 303 . The focus control unit 361 then converts the defocus amount and defocus direction into the drive amount and drive direction of the focus motor 360 . The focus control unit 361 controls the operation of the focus motor 360 based on this drive amount and drive direction, and drives the focus lens 359 to control the focus of the photographing lens unit 350 (focus adjustment).

In this manner, the focus control unit 361 performs phase difference detection autofocus (AF). Note that the focus control unit 361 may perform contrast detection AF that searches for a contrast peak of the image signal obtained from the image sensor 303 .
371 is a tracking unit for tracking the player of interest with the digital camera itself. Tracking here means, for example, moving a frame display surrounding the player of interest within the screen, and focusing or adjusting the exposure on the player of interest tracked by the frame.

Here, the tracking unit 371 will be explained using FIG.
FIG. 5 is a block diagram showing a functional configuration example of the tracking unit 371 of the digital camera. The tracking unit 371 is composed of a matching unit 3710 , a feature extraction unit 3711 and a distance map generation unit 3712 . The feature extraction unit 3711 identifies the image area (subject area) to be tracked based on the position information sent from the server. Then, a feature amount is extracted from the image of this subject area. On the other hand, the collating unit 3710 refers to the extracted feature amount in the captured image of each frame continuously supplied, and searches for an area having a high degree of similarity with the object area of the previous frame as the object area.

Note that the distance map generation unit 3712 can acquire distance information to the subject from a pair of parallax images (images A and B) from the imaging device, and the accuracy of specifying the subject area in the matching unit 3710 can be increased. However, the distance map generator 3712 may not necessarily be provided.
When the matching unit 3710 searches for an area highly similar to the object area as the object area based on the feature amount of the object area in the image supplied from the feature extraction unit 3711, for example, template matching or histogram matching is used. .

Next, an example of a player-of-interest display start sequence will be described with reference to FIG. This is the sequence performed by server 110 and camera 500 . FIG. 6A is a diagram showing an example of a sequence in which the server 110 side responds to a question (request) from the camera 500 side. The server 110 side then provides the camera 500 side with information on the absolute position of the player of interest.
The camera 500 notifies the server 110 of attention player designation information (ID information such as uniform number and player name). At that time, the user may touch the position of the player of interest on the screen of the terminal, or may surround the player of interest with the finger while touching the screen with the finger.

Alternatively, a list of multiple players may be displayed on the screen using the menu, and the name of the player of interest may be touched. Alternatively, a character input screen may be displayed on the screen and the player's name and player's jersey number may be entered. You may At this time, by touching the face position of the player of interest on the screen, the face may be image-recognized, or the player's uniform number may be image-recognized, and the player's name and uniform number may be sent. Alternatively, the face image itself may be sent to the server without image recognition, and image recognition may be performed on the server side. At this time, if there is a predetermined password, it is also sent to the server. On the server side, information regarding the absolute position of the player is sent to the camera based on the information specifying the player of interest (ID information such as uniform number and player name) by a block that supports image shooting. If a password is also sent from the camera, the contents of the information to be sent to the camera are changed accordingly.

Further, in FIG. 6A, the camera also sends to the server information such as the position information of the camera used by the professional photographer and the audience, the direction of the camera, the magnification of the camera, and the like. On the server side, a free-viewpoint image is created at the position and direction the camera is looking at, and the image seen by the actual camera is recognized from the magnification of the camera. The positional information indicating where the player is in the image actually seen by the camera, the player's contour information, etc. are sent to the camera. Based on the positional information, contour information, etc. sent from the server, the camera displays the player of interest more accurately and conspicuously on the screen of the display unit of the camera, and performs AF and AE on the player of interest.

An example of the sequence for starting the display of the player of interest for finding the player of interest has been briefly described, but there are many cases where it is desired to continue chasing the player. Therefore, next, a sequence for continuing tracking of the player of interest will be described with reference to FIG. 6(B). In FIG. 6B, the camera 500 serving as a terminal makes inquiries (requests) to the server 110 many times, for example periodically, to continuously recognize the player's position. In FIG. 6B, the camera periodically sends a player of interest display start sequence (A1, B1, . . . ) to the server, and the server periodically sends a player of interest display start sequence (A2, B2, ..). The operation of receiving and recognizing the position of the player of interest is repeated many times.

Further, FIG. 7 is a diagram showing another example of the player-of-interest display start sequence on the camera side, showing a method of automatically tracking the player-of-interest even by the camera itself. The ID information of the player of interest is sent from the camera 500 to the server 110, and the position information of the player of interest is temporarily obtained from the server. After the position information is obtained and the position of the player of interest is narrowed down by referring to the position information, the camera 500 itself continues to track the player of interest by image recognition. At that time, tracking is performed by the tracking unit 371 shown in FIGS. In the sequence of tracking the display of the player of interest in FIG. 7, the camera 500 itself tracks the player of interest using image recognition technology. , request the location information of the featured player again. Specifically, when the camera loses sight of the player of interest, the camera sends the player of interest display start sequence (A1) to the server again, receives the player of interest display start sequence (B2) from the server, and displays the player of interest. Display the position on the screen. After that, the camera itself tracks the player of interest by image recognition again.

In this embodiment, an example of a service for assisting professional photographers and spectators in photographing is described, but the service may also be used for remote camera control. By sending this information from the server, it is possible to track the player and capture the decisive moment with the remote camera mounted on the automatic pan-tilt head.
Also, although this embodiment has been described using an example of shooting assist, the terminal may be a home TV. That is, when a spectator watching TV specifies a player of interest, the server may send the positional information of the player of interest to the TV and display the player of interest conspicuously by displaying a frame or the like. Note that the player of interest may be indicated not only by the frame but also by a cursor (for example, an arrow), and the color and brightness of the region of the position of the player of interest may be made different from other portions. When the player of interest is outside the screen of the terminal, an arrow or characters may be used to indicate in which direction the player is out of the screen of the terminal. Also, when you are outside the display screen, how far away from the current viewing angle the terminal is (is it deviated), how much the terminal must be rotated before the player of interest can enter the display screen, etc. , numbers or scales may be used.

Furthermore, when the player of interest is on the screen, the additional information is displayed on the screen, and when the player of interest moves out of the screen, the user is instructed not to indicate that the player is out of the screen with an arrow or the like. can be selected.
Alternatively, if the situation of the game is automatically determined and, for example, when the player of interest has left the bench, even if the player of interest moves off the screen, the fact that the player is off the screen may not be displayed with an arrow or the like. good. If the user can select a mode in which display of additional information is automatically erased or a mode in which display of additional information is not automatically erased, usability is further improved.
In this embodiment, the cameraman's shooting assistance function is the main focus, but in the future, this part may be a robot camera. If you can get this much information from the server, instead of the cameraman, it is possible for the robot camera mounted on the automatic platform to track the athlete and capture the decisive moment.

Note that the captured still image (photograph) is converted into an image file compressed by JPEG, for example, in the compression/decompression circuit 310 after image processing.
In this embodiment, a still image (photograph) is recorded with various kinds of metadata added thereto. The metadata includes, for example, shooting date and time, shooting location information (latitude, longitude, altitude, direction), shooting information (aperture value, exposure time, ISO sensitivity, shutter speed, photometry method, etc.). , lens focal length, presence/absence of flash, exposure correction value, camera shake correction, color space, etc.), camera information (camera model name, lens name), etc. are also included. These metadata are recorded in the image file in, for example, Exif (Exchangeable Image File Format) format.

FIG. 8 is a diagram showing an example of a file structure. FIG. 8A shows the structure of a DCF (Design rule for Camera File system) thumbnail file in this embodiment, and FIG. 8B shows the structure of a DCF basic file. FIG. 4 is a diagram showing;
As shown in FIG. 8A, the DCF thumbnail file is composed of SOI (Start Of Image), Exif information, thumbnail file, EOI (End Of Image), and the like. On the other hand, as shown in FIG. 8B, the DCF basic file is composed of SOI, Exif information, thumbnails, DCF basic main images, EOI, and the like.
Here, SOI (Start Of Image) represents the beginning of JPEG data, and EOI (End Of Image) represents the end of JPEG data. Also, App1 (Application marker segment 1) stores Exif information.

In this embodiment, when a player is photographed, the Exif information including the player name is stored in the image file as Exif information. Furthermore, the match card, match venue, match time, time elapsed in the match, etc. may be stored.
It is also possible to determine who the photographed player is and store detailed information specific to the player from the database as metadata. When a player scores a try, for example, the player's entry timing, the time in the current game, and the player's behavior record during the match .”) may be stored as metadata.

In addition, the server side uses the metadata creation means to create general information (date, match card, match venue, match time, shooting time, elapsed time in the match, and if a player is photographed, the name of the player, etc.), judgment Create metadata for information, post-match information.
In this embodiment, it is assumed that the camera stores the above-described general information and metadata about cameraman's voice information as metadata corresponding to still images (photographs). The server side also has a function of storing all information (general information, judgment information, game end information, professional cameraman voice information, etc.). As metadata corresponding to still images (photographs), both the camera side and the server side may store all information (general information, judgment information, game end information, professional cameraman's voice information).

In this embodiment, as metadata corresponding to photos, the camera stores only the latest metadata for each photo, and the server stores not only the latest metadata for each photo, but also the past metadata. may be stored including the change history of .
Further, instead of adding the above-mentioned metadata to and storing only the photograph of the player of interest, the metadata may be added to and stored with respect to all the photographs of the athletes.
The server can add judgment information by the referee or information after the match to metadata such as general information and cameraman's voice information added to the photograph. When metadata is added on the server side in this way, the camera may be notified of the fact one by one.

Also, when the server receives a photo (image) from a camera (image processing device), the server may transfer the photo and metadata to a predetermined photo agency (other than the image processing device) as a destination. In that case, when the metadata of the photo is changed, the photo and the metadata may be resent to the photo agency together.
In this embodiment, when a photograph of a specific player of interest is taken, game information and information about the specific player are sent as metadata from the server to the device terminal (camera) as one of the destinations. Further, it is sent to a photo agency or the like registered in advance as a destination other than the camera (image processing device). The cameraman additionally inputs various kinds of metadata such as game information before the start of the game, and also adds the cameraman's voice information as metadata to the captured image (picture) at the time of photographing. Furthermore, when the photo with the metadata is sent to the server, the server inputs the result of judgment by a referee or the like as metadata.

Such metadata increases the added value of the photograph. In addition, metadata can be added in a short time, and high-value-added photos can be quickly sent to news bureaus.
Furthermore, information such as comments by the voice of the cameraman can be modified or combined with information recognized by the camera and information held by the server.

Next, an example of the control flow on the camera side in FIG. 7 will be described using the flow charts of FIGS. 9A, 9B and 10. FIG. 9A, 9B, and 10 show a player-of-interest display tracking control flow on the camera side.
In FIG. 9A, S101 represents initialization. In S102, it is determined whether or not photography (still image photography) has been selected. If photography is selected, the process proceeds to S103, and if photography is not selected, the process proceeds to S101. In S103, camera setting information is obtained. In S104, it is determined whether or not shooting (designation) of the player of interest has been selected. If shooting of the player of interest has been selected, the process proceeds to S105, and if shooting of the player of interest has not been selected, the process proceeds to S110 and performs other processing. do. In step S105, information on the player of interest (ID information of the player of interest, etc.) and a password, if any, are sent from the camera to the server. As a result, the server side detects the position information of the player of interest and transmits it to the camera. In S106, the positional information of the player of interest is received from the server.

In S107, the camera itself tracks the player of interest while referring to the position information sent from the server. Here, for example, the camera itself performs image recognition and tracks the player of interest. In that case, the player is tracked based on the player's jersey number, the player's facial information, the player's physique, or any combination thereof. That is, the shape of a part or the whole of the player of interest is image-recognized and tracked. However, if the user's shooting position is bad, the camera's field of view is narrow, or depending on the shooting angle, it may be hidden behind other subjects, so it may be lost. Send a location request.

S107-2 shows an example of mark display as additional information for the player of interest. That is, as the additional information, a cursor indicating the player of interest is displayed, a frame is displayed at the position of the player of interest, the color or brightness of the position of the player of interest is changed so as to stand out, or a combination thereof is displayed. Characters other than the mark may be displayed. Then, in a state in which a live view image from the image sensor is displayed on the image display unit, additional information indicating the position of the player of interest is superimposed.

FIG. 9B shows a specific example of the flow of mark display in S107-2. In S120, the relative position of the player of interest on the display unit is calculated based on the position information received from the server. In S121, while the live view image from the image sensor is being displayed on the image display unit, a mark indicating the position of the player of interest is superimposed on the image.
It should be noted that the tracking operation in S107 as described above may be skipped and may be selected by the user with a selection switch so as not to be executed. Alternatively, a mode may be provided in which the tracking action is performed when the player of interest is on the screen, but not when the player is out of the screen, and the mode can be selected. Furthermore, it automatically judges the situation of the competition, for example, when the player of interest enters the bench, automatically stops the tracking operation of the player of interest outside the screen (displaying additional information such as arrows). may be controlled. Alternatively, if the server recognizes that the player of interest has entered the bench regardless of whether it is on the screen or off the screen, the display of the position of the player of interest on the screen, the autofocus on the player of interest, and the automatic exposure adjustment for the player of interest are stopped. may be controlled to

In S108, it is determined whether or not it is OK (successful) to continue tracking the player of interest. Continue photographing players. On the other hand, if the tracking fails, the result in S108 is No, and the process proceeds to S109.
In S109, it is determined whether or not the shooting of the player of interest is finished. When the shooting of the player of interest is finished, the process returns to S101.

In S111 of FIG. 10, it is determined whether or not the pre-selected player of interest has actually been photographed. If the player of interest has actually been photographed, the process advances to S112, where the voice information of the cameraman at the time of photographing is transmitted to the camera operator (not shown). input from the microphone. If the player of interest has not actually been photographed, the process proceeds to S107. At S113, the voice information input at S112 is converted into character information by voice recognition and stored. At S114, metadata (general information) of the player of interest is input from the server. It should be noted that information (for example, contents of general information, etc.) related to metadata acquired from the server can be displayed on the display device on the camera side when the user selects it through a menu operation. Alternatively, if the user has set in advance by operating the menu, it may be automatically displayed on the screen of the display device. In this case, other information such as the position information of the player of interest may be displayed simultaneously, or only information related to metadata acquired from the server may be selectively displayed. Alternatively, in S115, audio information converted into character information for the photographed photograph is used as metadata. In S116, the metadata obtained in S114, S116, etc. is added as Exif information to the captured image (photograph) and recorded in the recording medium. In S117, the photograph with the metadata added is sent to the server, and then the process proceeds to S107 in FIG. 9A.

Next, Example 2, which is a modified example of FIG. 10, will be described with reference to the flowchart of FIG. 11 showing an example of the player-of-interest display tracking control flow on the camera side. In FIG. 11, the same reference numerals as those in FIG. 10 indicate the same steps, so the description is omitted.
In the second embodiment, metadata added at the time of shooting (such as voice information converted into text information) is replaced with metadata such as referee judgment information.
That is, in S118 of FIG. 11, it is determined whether or not the metadata of the photographed photograph has been updated on the server side. If the metadata of the photographed photograph is updated, the process advances to S119 to replace the metadata at the time of photographing (audio information at the time of photographing converted into character information, etc.) with judgment information of the referee. That is, update. In addition, when the user selects information on the metadata acquired from the server (for example, the contents of the updated metadata) on the display device on the camera side by menu operation, it is possible to display it. If the metadata of the photographed photograph is not updated, the process proceeds to S107 in FIG. 9A. In addition, if the player of interest has not been photographed in S111, the process proceeds to S118.

In the second embodiment, the metadata is rewritten according to the judgment information, but the judgment information may be added to the photograph as metadata and recorded. In this way, along with the cameraman's voice memo, judgment information such as referee's information can be added to the photo, so that various additional information will remain for the photo, and the photo (image) can be analyzed and sorted later. useful when doing
It should be noted that when the judgment information such as referee is stored in the photograph as metadata, the judgment information may be displayed as character information on the screen. This allows the photographer to confirm on the screen what kind of judgment has been made for the photographed photograph. Specifically, for example, it is possible to display on the screen a message such as ``~ The player entered the game in the 5th minute of the second half, and at the 35th minute of the second half, it was thought that the try was a success, but the result of the judgment was a no try''.

Similarly, the metadata of the cameraman's voice information at the time of shooting, the judgment information of the referee, and the metadata of the result information after the game may be added to the photograph as metadata, stored, and displayed on the screen.
In addition, for example, information such as the time when a player entered a game as a starting member and left the game as a substitute player, or the time when the player entered the game as a substitute player, is added to the photo and stored or displayed as metadata focusing on the player. You can do it.

As explained above, if you upload a photo with metadata from the camera to the server, and then change or add metadata (judgment information, match result information, etc.) for the same photo on the server side, the camera The side may perform only metadata changes.
It should be noted that the server may not only store general player information regarding registered players as a database, but may also store such information as a metadata database.

In addition, in the embodiment, the server 110 can read the image of the entire competition field, for example, and obtain the coordinates, so that the images taken by professional photographers and spectators can be analyzed. Thereby, it is also possible to grasp from where the shooting is being performed with respect to the competition field. In other words, the server grasps in advance an image of the entire playing field from a plurality of server cameras (fixed cameras and moving cameras). As a result, it becomes possible to map the absolute position information of the player of interest on the field and the video that professional photographers and spectators see on terminals and digital cameras.

In addition, when a terminal such as a professional cameraman or a spectator's camera receives the player's absolute position information from the server, it becomes possible to map this absolute position information and the image currently being shot or monitored. For example, let (X, Y) be the absolute position information of the player of interest within the field from the server. It is necessary to convert this absolute position information into relative position information (X', Y') when viewed from the camera according to the position information of each camera. The above conversion from absolute position information to relative position information may be performed on the camera side as in S120, or the relative position information may be sent to individual terminals (cameras, etc.) after conversion on the server side. .

When performing the above conversion on a terminal such as a camera, relative position information (X', Y '). This relative positional information is used as positional information within the display screen of the camera.
On the other hand, when the server performs the above conversion, the server converts the absolute position information (X, Y) into relative position information (X′, Y'). The server sends this relative position information to each camera, and the camera that receives this relative position information converts this relative position information into the position information within the display screen of each camera. and

As described above, it is possible to reduce the possibility of losing sight of the player of interest on the terminals of professional photographers and spectators' cameras.
With the configuration described above, cameras owned by camera operators or general spectators send information on the player of interest to the server, and track the player of interest based on the information obtained from the server. Further, along with this, the position of the player of interest can also be detected by the camera held by the cameraman or by the general spectator.

For example, in a rugby game, the metadata includes information 48 hours before the start of the match, change information several hours before the start of the match, information such as changes in players during the match, and voice comment information of the cameraman at the time of shooting, for example. etc. can be considered. Further, it is conceivable that information on referee's judgment, information on match results at the end of the match, information that the server obtains from the Rugby Union, etc., are included. In addition, there is information obtained as a result of obtaining device information of a camera or the like and analyzing the device information of the camera by a server, and information obtained from a photographer such as a photographer.

These metadata are collectively stored on the server and the necessary information is sent to the camera. The camera adds the information obtained from the server as metadata to the photo.
The server can also determine who the player has been photographed and store player-specific information as metadata. For example, the player's entry timing, the time in the current match, and the action of the player ("AAA player enters the starting line-up and scores one try in the 36th minute of the first half.") may be stored as metadata. .

Next, a control flow for detecting a player of interest on the server side corresponding to the first embodiment shown in FIG. 10 will be described with reference to FIGS. 12 and 13. FIG. Here, a case will be described in which voice information of the cameraman is included as metadata at the time of shooting.
Based on the ID information and the like of the player of interest sent from a terminal such as a camera, the server performs image recognition of the player of interest and searches for the player of interest on the screen. The server detects the player's position information based on the images from multiple cameras for the server (fixed cameras, moving cameras, etc.), and sends the player's position information to professional photographers and spectator camera terminals. . In particular, when a professional cameraman or spectator takes a picture, if there is positional information of the player of interest from the server, the player of interest can be reliably photographed without mistaking the player. Furthermore, information from the server is also important when tracking a player of interest with a camera and losing sight of the player in a blind spot. On the server side, the player's position information is continuously detected based on the images from multiple cameras for the server.

FIG. 12 shows a main flow of detection control of a player of interest on the server side, corresponding to the first embodiment.
In FIG. 12, initialization is first performed in S201. Next, in S202, it is determined whether or not photography has been selected in the camera, and if photography has been selected, the process advances to S203 to obtain camera setting information. At this time, if there is a password in the setting information of the camera, it is also obtained. If photography is not selected, the process proceeds to S201. In S204, it is determined whether or not shooting (designation) of the player of interest has been selected. If shooting of the player of interest has been selected, the process proceeds to S205. number). If shooting of the player of interest is not selected in S204, the process proceeds to S210 to perform other processing.

In S206, the server finds the player of interest in the screen by image recognition based on the images from a plurality of cameras (fixed cameras, moving cameras, etc.) based on the ID information of the player of interest. At S207, the server tracks the player of interest based on the images from the multiple cameras. In S208, it is determined whether or not a photograph (still image) of the player of interest has been taken, based on information from the camera. If there is no shooting, the process returns to S205.
In S208-2, the photograph attached with metadata at the time of photographing is received from the camera. Here, the metadata includes metadata such as cameraman's voice information when the cameraman releases the shutter and other general information.

In step S208-3, the photograph in step S208-2 is sent to the photo agency, which is a predetermined destination registered in advance.
As a result, photo agencies can receive photos with metadata at an early stage, making it possible to utilize photo information.
In S209, it is determined whether or not the shooting of the player of interest has ended. When the shooting of the player of interest ends, the process returns to S201, and when the shooting of the player of interest continues, the process returns to S205. Then, again, the server recognizes information from cameras owned by cameramen or general spectators. Then, information from a plurality of cameras (fixed cameras and moving cameras) for the server is searched to find the player of interest. Carry out tracking.

If photography is not selected in S202, it is determined in S211 whether metadata input has been selected. If data input is not selected, the process returns to S201.
Metadata input in S212 is input before the start of a rugby match. For example, there is a change of participating players (information several hours ago).

As a result, the metadata information before the start of the match is temporarily accumulated in advance on the server side. Then, when the cameraman actually takes a picture, that is, when the shutter is released, it becomes possible to add the cameraman's voice information and metadata to the picture on the server side.
As a result, it is possible to leave the cameraman's voice memo as metadata of the photo, and for example, information about the game before the start of the game will be automatically entered in the server without the cameraman's input. Become. Therefore, the photographer can concentrate on shooting and voice memo, and can send photos with audio metadata to the press immediately after shooting.

Next, an example of a method for finding a player of interest in S206 and tracking him in S207 will be described with reference to FIG.
FIG. 13 shows a target player detection control flow using uniform number information. In FIG. 13, in S401, the server obtains the uniform number from the data memory 213 based on the ID information of the player of interest, searches for this uniform number from the image information of a plurality of cameras for the server by image recognition, Get the position information of the number player. In S402, the absolute position information of the player of interest is obtained by integrating the position information obtained from the images of the plurality of cameras for the server. By synthesizing the information of a plurality of cameras for the server in this way, the accuracy of the absolute position information of a player with a certain uniform number is improved. In S403, the absolute position of the player of interest detected in S402 is transmitted to terminals such as cameras owned by professional photographers and spectators. In S404, it is determined whether or not the player of interest is being tracked. If the player of interest is being tracked, the process returns to S401, and if the player of interest is not being tracked, the flow in FIG.

Next, an example of replacing the voice information of the picture cameraman with the judgment information of the referee and using it as metadata will be described with reference to the flowchart of FIG. 14 . This FIG. 14 is a flowchart on the server side corresponding to the second embodiment on the camera side shown in FIG.
Since S201 to S207, S208-2, S208-3, and S209 to 212 are the same as in FIG. 12, their description is omitted.
In S208, it is determined based on the information from the camera whether or not the player of interest is actually photographed, that is, whether or not the shutter is released. If the player of interest is actually photographed, that is, if the shutter is released, the process proceeds to S208-2, and if the player of interest is not actually photographed, that is, the shutter is not released, the process proceeds to S213.
In S213, it is determined whether or not there is determination information in the server when the cameraman's voice information is input. If Yes, the process proceeds to S214, and if No, the process proceeds to S205.

In S214, the metadata of the cameraman's voice information is changed to the judgment information of the referee. Also, since the metadata is changed, the name of the photo is updated.
In this way, the metadata acquired before the start of the game is accumulated on the server side, and when the cameraman actually takes a picture, the cameraman's voice information is added to the picture as metadata and sent to the server. send to On the server, it will be possible to automatically rewrite the metadata when the referee's judgment differs from the cameraman's voice memo.
In S214-2, the metadata in S214 is sent to the camera. As a result, the camera side can display information (for example, contents of determination information, etc.) related to the metadata sent from the server on the camera side display device when the user selects it by menu operation.

This allows the camera to synchronize with server-side metadata. The camera does not change the photo (still image), only the metadata of the photo.
Next, in S214-3, the photo whose metadata and name have been changed in S214 is sent to the photo agency, and then the process proceeds to S205.
As a result, when the metadata changes, the photo agency will be able to receive the photo with the name and number changed, which will increase the utilization of photo information, simplify the work of photographers, and speed up the process. Photos can be sent to the press.

Next, FIG. 15 is a flowchart on the server side corresponding to Example 3 in which both cameraman's voice information and referee's judgment information are metadata.
Since S201 to S207, S208-2, S208-3, and S209 to 212 are the same as in FIG. 12, their description will be omitted.
In S208, it is determined based on the information from the camera whether or not the player of interest is actually photographed, that is, whether or not the shutter is released. If the player of interest is actually photographed, that is, if the shutter is released, the process proceeds to S208-2, and if the player of interest is not actually photographed, that is, the shutter is not released, the process proceeds to S215.
In S215, it is determined whether or not there is determination information in the server when the voice information of the cameraman was input. If Yes, the process proceeds to S216, and if No, the process proceeds to S205.
In S216, the metadata of the judgment information is stored together with the metadata of the cameraman's voice information. Well, change the name of the photo because the metadata has changed.

As a result, the metadata information before the start of the game is accumulated on the server side, and the cameraman's voice information when the cameraman actually took the picture is added to the picture as metadata. can also be stored as metadata.
In S216-2, the metadata of S216 is sent to the camera. As a result, the camera side can display information (for example, contents of determination information, etc.) related to the metadata sent from the server on the camera side display device when the user selects it by menu operation.
This enables the camera to update the metadata to the latest one. In the camera, it doesn't change the photo, only the photo's metadata.
S216-3 sends the photo with the modified metadata to the photo agency at S216.

As a result, when the metadata is changed, the photo agency can receive the photo with the changed name, and can quickly publish the photo using the photo and the metadata. It also simplifies the work of photographers and allows them to quickly send photos to the news bureau.

Next, with reference to the flow charts of FIGS. 16 and 17, an example will be described in which metadata is information in which professional cameraman's voice information, referee's decision information, and game result information are written together as information at the time of shooting.
16 are the same as those in FIG. 12 except for S209, so description thereof will be omitted.

If the shooting of the player of interest has ended in S209, the process proceeds to S219 in FIG. At S219, it is determined whether or not the match has ended.
In S220, it is determined whether or not there is post-match metadata. If there is post-match metadata, the process proceeds to S221, and if there is no post-match metadata, the process proceeds to S201.
In S221, the metadata of the audio information input by the cameraman, the metadata of the judgment information, and the metadata information after the end of the game are added. Change the name of the photo along with the metadata change.
This makes it possible to add the cameraman's voice information, the judgment result of the referee, and the information after the game as metadata to the photograph.

In S221-2, the metadata of S221 is sent to the camera. As a result, the camera side can display information (for example, judgment information, game end information, etc.) related to the metadata sent from the server when the user selects it by operating the menu on the display device of the camera side.
This enables the camera to update the metadata to the latest one. The camera does not change the photo, only the metadata.
At S221-3, the photograph with the metadata of S221 is sent to the photo agency.
In the above embodiment, when photos having metadata are uploaded from the camera to the server, the camera updates only the metadata for each photo. On the server side, on the other hand, when the metadata is updated, the photo (captured image) file is updated, so there are multiple photos, and the photo name is updated.

As a result, the cameraman's voice information, the judgment result of the referee, and the game result information after the game is over can be stored in the photograph as metadata. It is possible to increase the added value of the photograph.
For professional photographers and general spectators, the judgment in the game is something to worry about. There are three referees, one referee and two touch judges. In particular, there are cases where judgment cannot be made by human eyes alone, such as judgment in a scene where it appears that a try has been scored, or whether or not a foul has occurred in a game. For this reason, video judgment (TMO (television match official)) is used to help referees make judgments when it is difficult to make judgments with the naked eye.

Professional photographers and general spectators want to know immediately whether the image taken just now was certified as a try or not as a try when shooting a try scene with a terminal such as a camera owned by a professional cameraman or general spectator. . Therefore, the judgment is accurately recognized by tracking the subsequent judgment from the images of multiple cameras and analyzing the information displayed on the electronic bulletin board on the server. By transmitting the referee's decision results to the terminals of professional photographers and general spectators' cameras, etc., users can timely and correctly recognize the decision results.

There are multiple ways to detect whether a try has been made. FIG. 18 shows a try determination control flow (on the server side), which will be specifically described below.
In FIG. 18, S1101 represents initialization. Here, the TRY judge flag is cleared. In S1102, it is determined whether or not photography has been selected. If photography has been selected, the process advances to S1103, and if photography has not been selected, the process advances to S1101. In S1103, camera setting information is obtained. In S1104, the ball used in the game is tracked from the images of the multiple cameras.

At S1105, it is determined whether the TRY judge flag is 0 or not. In S1106, it is determined whether or not there is a try. Here, a try is a state in which the player brings the ball into the try area. For example, a player knocks the ball forward just before a try, or a defender puts his hand or body into the ball when he is about to try, preventing him from grounding. be. That is, the try is not in a definite state. S1107 sets 1 to the TRY judge flag. S1108 determines whether or not there is a try.
19, 20, 21, and 22 show specific examples of try determination, which will be described later.

In S1109, it is determined whether or not the determination result of the presence or absence of a try is obtained in the control of S1108. proceed to S1110 sets 0 to the TRY judge flag. In S1111, the server sends information as to whether or not there was a try to terminals such as cameras owned by professional photographers and general spectators. In S1112, it is determined whether or not the game has ended. If the game has ended, the process proceeds to S1101, and if the game has not ended, the process proceeds to S1104.
The above is the control for determining whether or not there is a try. However, the CPU may not only execute this control, but may execute it in parallel with, for example, the player-of-interest detection control shown in FIG. 12 and the like.

Note that the control executed concurrently by the server is not limited to this, and a plurality of other controls may be executed simultaneously. On the other hand, the same applies to terminals such as cameras held by professional photographers and general spectators, and a plurality of other controls may be performed simultaneously.
When shooting a try scene, you have to correctly recognize whether the try succeeded or failed. The server confirms the determination of whether the try was successful or the conversion was successful. Here, an example of a trie is used for explanation. However, similar control can be executed not only for tries but also for other scoring scenes.

Analyze images from multiple cameras, and when there is a play that seems to have resulted in a try, analyze the movement of the ball with multiple cameras to see if the ball was properly grounded within the designated area. recognize whether or not Based on the movement of the ball that has been recognized, the server sends information as to whether or not it was a try, together with player information, to terminals such as cameras owned by professional photographers and general spectators.
FIG. 19(A) shows a flow for judging whether or not there is a try based on the movement of the ball on the server side.
In FIG. 19A, in S1201, the server detects the location of the ball from the images of multiple cameras. In S1202, it is recognized from the movement of the ball whether there is a try in a scene that is considered to be a try from the images of a plurality of cameras.

Next, FIG. 19(B) shows a try/non-try determining flow from the referee motion.
Analyze images from multiple cameras, and when there is a play that seems to have resulted in a try, analyze the images from multiple cameras and analyze the behavior of referees around the player of interest according to the rules afterward. It may be recognized whether or not it was a try from the action of the umpire. The server sends information (results of motion recognition) on analysis results as to whether or not a try was scored based on the recognized motions of the referee, together with player information, to terminals such as cameras owned by professional photographers and general spectators.

In FIG. 19(B), in S1301, the server detects a referee's motion near a try-deciding motion from images of a plurality of cameras. In S1302, it is recognized from the actions of the referee whether there is a try in a scene that is considered to be a try from the images of a plurality of cameras.
FIG. 20 shows the refereeing operation for determining whether or not there is a try. In FIG. 20, FIG. 20(A) shows the referee's action when a try is established. FIG. 20(B) shows the referee's action when a try is not established.

Further, a flow for recognizing the presence or absence of a try based on the information displayed on the large screen in the stadium will be described with reference to FIG.
Analyze the images from multiple cameras, and when there is a play that seems to have been a try, input the information projected on the large screen of the stadium with multiple cameras, and use the information on this screen to make a try recognize whether or not it was Based on the information on the recognized screen, the server sends information as to whether or not the try was scored, together with player information, to terminals such as cameras owned by professional photographers and general spectators.

FIG. 21(A) shows a flow for judging whether or not there is a try based on the screen display judgment result on the server side. In FIG. 21A, in S1401, the server detects determination result information displayed on the screen after a try-like movement from images from a plurality of cameras. In S1402, it is recognized from the judgment result displayed on the screen whether there is a try in a scene that is considered to be a try from images of a plurality of cameras.

Next, the flow for recognizing the presence or absence of a try based on the score information displayed on the screen will be described with reference to FIG. 21(B).
Analyze the video from multiple cameras, and when there is a play that seems to have been a try, enter the score information displayed on the screen based on the images from multiple cameras, and use the score information on this screen Recognize whether it was a try or not. Based on the score information on the recognized screen, the server sends information as to whether or not it was a try along with player information to terminals such as cameras owned by professional photographers and general spectators. A try scores 5 points, and a successful conversion kick scores 2 points.

Also, if a penalty kick or a drop goal is successfully scored, 3 points are scored. Comparing the score before the try was supposed to be scored with the score after the try was scored allows us to recognize whether the try was successful.
FIG. 21(B) shows a flow in which the server determines the presence or absence of a try based on the score information on the screen.

In FIG. 21B, in S1501, the server detects score information displayed on the screen after a try-like movement from the images of a plurality of cameras. In S1502, it is recognized from the difference in score information displayed on the screen whether there is a try in a scene that is considered to be a try from the images of a plurality of cameras. In S1503, a try, a conversion kick, a penalty kick, or a drop goal is recognized from the difference in score information displayed on the screen as to whether a try was scored in a scene that is considered to be a try based on images from a plurality of cameras.

Next, a flow for recognizing the presence or absence of a try based on voice information announced in the field will be described with reference to FIG.
Analyze audio information input from microphones attached to multiple cameras (fixed cameras and moving cameras), and when there is a play that seems to have been a try, use the audio information from the above microphones to determine whether it was a try or not. recognize. Based on the recognized voice information, the server sends information as to whether or not it was a try, together with player information, to terminals such as cameras owned by professional photographers and general spectators.
FIG. 16 shows a specific flow for judging whether or not there is a try from voice information by the server. In FIG. 22, in S1601, the server detects audio information after a try-like movement from microphones of a plurality of cameras. In S1602, it is recognized from the microphones of a plurality of cameras whether there is a try in a scene considered to be a try based on audio information.

In the above embodiment, the determination of the score of a try has been described, but in addition to the score of a try, it is also possible to consider the score of a conversion after a try and the score of a penalty kick.
FIG. 23 shows the server-side detection flow of whether or not a try has been made, but the control of a terminal such as a camera regarding whether or not a try has been made will be described.
FIG. 23 shows a try determination control flow on the side of a terminal such as a camera. In FIG. 23, steps S101 to S107-2 and S110 are the same as the steps in FIG. 9, so description thereof will be omitted.

At S1620, it is determined whether or not it is OK (successful) to continue tracking the player of interest. In S1621, it is determined whether or not the result of the try decision has been sent from the server. If the result of the try decision has been sent from the server, the process proceeds to S1622. Continue tracking players. In S1622, the try determination result is displayed on the display unit of the camera.

If it is determined in S1620 that the player of interest has not been successfully continued to be tracked, the process advances to S109 to determine whether or not the shooting of the player of interest has ended. If not, the process returns to S105. If so, the process advances to S1623 to determine whether or not the try decision result has been sent from the server.If the try decision result has been sent from the server, the process advances to S1624 to display the try decision result on the display unit of the camera terminal. do. If the try determination result has not been sent from the server, the process returns to S101.
As described above, it is possible to display whether or not the try is successful on the camera terminal side when a try-like state occurs. As a result, general spectators or photographers, for example, can correctly recognize the evaluation of the photographs taken. Since the cameraman can recognize the judgment just by looking at the display unit of the camera, he can appropriately select photographs to be sent to the news media, etc., and quickly prepare for the next photographing.

When a foul occurs, multiple cameras (fixed cameras and mobile cameras) recognize the foul situation, but this information is also sent to terminals such as cameras owned by professional photographers and general spectators and displayed on the above terminals. be done.
Here, FIG. 24 shows the foul determination control flow on the camera side. FIG. 24 is based on the try determination control flow on the camera side, and is for displaying foul situations on the camera side.
24 differs from FIG. 23 only in S7421 to S7424. That is, S1621, S1622, S1623, and S1624 in FIG. 23 are replaced with S7421, S7422, S7423, and S7424, and the only difference is that the "try decision" is changed to "foul decision".

As described above, the state of foul play can be displayed on a terminal such as a camera owned by a professional photographer or general spectator, so that the professional photographer or general spectator can correctly recognize the situation of the photographed photograph. In particular, professional photographers need to send photos to the press during the game, which is useful information for photo selection.
As described above, according to this embodiment, the server analyzes the surrounding information of players other than the player of interest (other than the specific target), and transmits the analysis result to an image processing device such as a camera. , you will be able to grasp the real-time situation of the competition such as tries, goals, and fouls. Therefore, in particular, professional photographers and the like can obtain very useful information when selecting photographs and sending them to the news media in a timely manner during a game.

It should be noted that the player's movements may be stored in the server as big data, and AI may be used to predict the player's movements based on this big data. Although an example in which only one player is designated as the player to be noticed has been described, a plurality of players may be designated as the player to be noticed. In addition, it is assumed that the player to be noticed can be switched in the middle. All the players who are participating in the game may be selected as featured players. Also, video and images include not only moving images but also still images. In addition, I have explained mainly about chasing and chasing attention players.

However, instead of chasing only the player of interest, information about the player who has the ball or who receives the ball may be transmitted and displayed to the professional cameraman or spectators. In addition, although the example of tracking a player has been described in the embodiment, it goes without saying that the present invention can be applied to a system that tracks a person such as a criminal using a plurality of monitoring cameras. Alternatively, the present invention can be applied not only to a system for tracking a specific vehicle in car racing or the like, but also to a system for tracking a horse in a horse race or the like. In the embodiment, an example in which a player of interest is specified using a camera terminal or the like has been described, but the player of interest may be specified on the server side.

In the above embodiment, rugby was used as an example, but it goes without saying that the present invention can also be applied to other sports and field work.
On the server side, the photograph sent from the terminal device may be sent to, for example, a broadcasting station via a network (not shown).
As described in the above embodiments, photographers want to add various data as metadata when taking pictures, but there is a risk that this may prevent them from concentrating on taking pictures. However, according to this embodiment, by sending a photograph to the server, the server can add or rewrite various metadata for the photograph. This allows the photographer to concentrate on shooting. In addition, the server side can add information that could not be obtained in the past, such as general information related to the photo and game judgment information. Therefore, the added value of each photograph can be increased, and efficiency can be improved when analyzing, sorting, and editing the photographs later.

Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist.
Also, a program (software) that implements the functions of the above-described embodiments for part or all of the control in the present invention may be supplied to the imaging device or the information processing device via a network or various storage media. Then, the computer (or CPU, MPU, etc.) in the imaging device or information processing device may read and execute the program. In that case, the program and the storage medium storing the program constitute the present invention.

101, 102, 103... Cameras 401, 402, 403... Terminal 110... Server 371... Tracking unit 380... Image display unit

Claims (16)

display means for displaying an image;
selection means for selecting a specific target from among the images displayed on the display means;
Designation information generating means for generating designation information designating the specific target selected by the selection means;
transmitting means for transmitting the specified information generated by the specified information generating means to a server;
Acquisition means for acquiring position information indicating the relative position or absolute position of the specific target in the image displayed on the display means from the server and predetermined metadata of the specific target created in the server;
Control means for displaying additional information based on the position information of the specific target acquired by the acquisition means or character information based on the predetermined metadata on the display means;
An image processing device comprising: 2. An image processing apparatus according to claim 1, wherein said image processing apparatus transmits the photographed image to said server by said transmitting means. 3. The image processing apparatus according to claim 2, wherein said server adds said predetermined metadata to said image transmitted by said transmitting means. 4. The image processing apparatus according to claim 3, wherein said server transmits said image to which said predetermined metadata is added to a predetermined destination. 5. The image processing apparatus according to claim 4, wherein said predetermined destination includes said image processing apparatus. 5. The image processing apparatus according to claim 4, wherein said predetermined destination includes a destination other than said image processing apparatus. 6. The image processing apparatus according to any one of claims 1 to 5, further comprising input means for inputting metadata, wherein said transmission means transmits the metadata input by said input means to said server. 1. The image processing device according to item 1. 8. The image processing apparatus according to claim 7, wherein said server has metadata creation means for creating said predetermined metadata based on the metadata transmitted from said image processing apparatus by said transmission means. 9. The image processing apparatus according to claim 8, wherein said server replaces said predetermined metadata created by said metadata creating means with metadata transmitted from said image processing apparatus. 8. The image processing apparatus according to claim 7, wherein the metadata input by said input means includes metadata relating to audio. 11. The image processing apparatus according to claim 10, wherein said input means converts said voice into characters and uses them as metadata. Receiving means for receiving designation information for designating a specific object sent from the image processing device;
relative position or absolute position of the specific target in the image displayed on the display means of the image processing device by searching for the specific target from images of a plurality of cameras based on the designation information received by the receiving means; generating means for generating position information indicating a position ;
metadata creation means for creating predetermined metadata of the specific target ;
An image processing server, comprising a transmission unit for transmitting the position information of the specific object and the predetermined metadata generated by the generation unit to the image processing apparatus. A computer program for causing a computer to function as each means of the image processing apparatus according to any one of claims 1 to 11. 14. A computer readable storage medium storing the computer program according to claim 13. A computer program for causing a computer to function as each means of the image processing server according to claim 12. 16. A computer readable storage medium storing the computer program according to claim 15.

Images