JP2003264827A - Image processing apparatus - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus for composing images picked up with a plurality of cameras installed on a vehicle and avoiding a sense of incongruity from being caused on a displayed image even when the state of moving parts of the vehicle is changed. <P>SOLUTION: A pixel composing section 21 receives a camera image outputted from an imaging section 1 and references a mapping table MPT to produce a synthesis image. A display switching section 23 receives a signal S1 indicative a state of the moving parts such as a door and a bonnet and rewrites mapping data so as not use the camera image for the synthesis image when the state change in the moving parts effects the camera image. <P>COPYRIGHT: (C)2003,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing technique for generating a composite image by using captured images of a plurality of cameras installed in a vehicle, and particularly to assist in safety confirmation when driving a vehicle. It belongs to the technology effective for the monitoring system used.

[0002]

2. Description of the Related Art Conventionally, as a device for monitoring the surroundings of a vehicle by using a camera, a vehicle image providing device disclosed in Japanese Patent Laid-Open No. 11-78692 is known. With this device,
Attaching multiple cameras for vehicle image provision around the vehicle, transforming the images taken by each camera and synthesizing while maintaining the continuity of boundaries, create a new image,
It is displayed on the monitor.

[0003]

In the vehicle surroundings monitoring system as described above, image processing is executed on the assumption that the camera once installed does not normally change its installation position or orientation. ,it is conceivable that. This is because it is extremely difficult in practice from the viewpoint of processing time and device scale to perform image processing for generating a composite image in real time while considering the installation position and orientation of the camera each time.

On the other hand, considering the aerodynamic surface and the design surface of the vehicle, it is unlikely that the camera used in such a surveillance system will be installed in the vicinity of the mounting portion of the door mirror or in a movable portion such as a trunk or a hood. , Can be enough.

At this time, the following problems occur.

That is, a change in the state of the movable part to which the camera is attached may affect the image picked up by the camera and cause a shift or the like in the combined image. For example, when the camera is attached near the door mirror, the direction of the camera changes when the door is opened, so that only the portion using the camera image on the composite image is displaced. As a result, the user feels uncomfortable on the display screen.

The same problem occurs when the camera is mounted on the fixed portion. For example, when the door is reflected in the captured image of a specific camera due to the opening of the door, a shift occurs only in the portion using the camera image on the composite image. The shift on the composite image results in a sense of discomfort to the user, which is not preferable.

In view of the above-mentioned problems, the present invention is an image processing apparatus for generating a composite image by using captured images of a plurality of cameras installed in a vehicle. It is an object to prevent the user from feeling uncomfortable in the displayed image even when affected.

[0009]

[Means for Solving the Problems] In order to solve the above-mentioned problems, a solution means provided by the invention of claim 1 is a plurality of cameras installed in a vehicle as an image processing device and photographing the surroundings of the vehicle. Of the plurality of captured images, the composite image is generated from these captured images, and the combined image is output to a display device. The image processing unit is configured to change the state of a movable part of the vehicle. When the installation position or orientation of at least one of the cameras changes and the composite image becomes abnormal, the display mode of the composite image is switched from the normal mode to the abnormal mode, and in the abnormal mode, instead of the composite image, or A message is output together with the composite image to inform the user of the state change of the movable part that causes the abnormal mode, and the message is a message indicating that the state change has occurred. It is different depending min type.

According to the first aspect of the present invention, by switching the display mode of the composite image, it is possible to promptly and accurately inform the user that the state of the movable portion is not desirable for running the vehicle. become.

Further, in the invention of claim 2, the message in the image processing apparatus of claim 1 includes a content for specifying a movable part in which a state change has occurred.

[0012]

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment of the present invention will be described below with reference to the drawings.

(First Embodiment) FIG. 1 is a block diagram showing the arrangement of a monitoring system according to the first embodiment of the present invention. In the monitoring system shown in FIG. 1, the image processing unit 2
Receives a plurality of camera images output from the image pickup unit 1 and synthesizes them to generate a new image. This composite image is displayed by the display device 3. Image processing unit 2
The image processing apparatus according to the present invention is configured by the above.

FIG. 2 is a diagram showing an example of camera arrangement in the present embodiment, and FIG. 3 is a diagram showing an example of a captured image of each camera. Here, it is assumed that eight cameras are arranged as shown in FIGS. 2 and 3. In the camera arrangements shown in FIGS. 2 and 3, in order to eliminate blind spots in the surroundings,
It is preferable to determine the position and orientation of each camera so that there is an overlapping area in the images of the cameras adjacent to each other.

The image processing unit 2 transforms / combines the eight camera images shown in FIG. 3 to generate a composite image as viewed from above the vehicle vertically downward. Image generation processing and synthesis processing (including boundary processing) of the deformed partial images are required to generate the composite image. However, in the configuration of FIG. 1, the mapping table reference unit 22 is provided and the captured image The mapping table MPT is used to perform the processing in one step. The mapping table will be described later.

The mapping table selection unit 41 selects the display content of the composite image and gives an instruction to the mapping table MPT according to the selected display content. This selection may be performed artificially or automatically depending on the vehicle position such as gear position, steering wheel steering angle or winker. A signal for selecting the mapping table MPT is sent to the message output unit 26 together with the mapping table reference unit 22.

The display switching section 23 receives the status signal S1 indicating the status of the movable portion of the vehicle and switches the display mode of the composite image. For example, when the installation position or orientation of any of the plurality of cameras changes due to a change in the state of the movable part and the combined image becomes abnormal, the display mode of the combined image is switched from the normal mode to the abnormal mode. In the abnormal mode, for example, a synthetic image is generated without using a captured image of a camera whose installation position or orientation has changed.

FIG. 4 is an example of a composite image that is no longer normal due to a change in the state of the movable portion of the vehicle. As can be seen from the composite image in FIG. 4, the images captured by the cameras 1 and 2 changed due to the opening of the right door of the vehicle, so the images captured by the cameras 1 and 2 were used. The part is misaligned.

Therefore, in this embodiment, the open / closed state of the right door, the left door, the trunk, and the hood on which the camera is installed is used as the state of the movable portion of the vehicle. That is, the display switching unit 23 is, for example, "the right door is open" and "the hood is closed" according to the status signal S1.
Information such as is entered. The display switching unit 23 previously has data indicating the correspondence between the movable parts of the vehicle and the camera numbers as shown in FIG. 5, and this correspondence and the status signal S.
The information shown by 1 is used to switch the display mode of the composite image.

FIG. 6 is a diagram showing an example of switching the display mode of the composite image according to the present invention. In the figure, (a) is a composite image in the normal mode. When all the movable parts of the vehicle are in the closed state, the composite image becomes normal, and therefore, it is displayed as shown in FIG. 6 (a). FIG. 6B is an example of a composite image in the abnormal mode, and it is assumed that the right door of the vehicle is open. In the composite image of FIG. 6B, the captured images of the camera 1 and the camera 2 installed on the right door are not used for image composition, but instead are painted in a predetermined color. Instead of painting with a predetermined color, another pattern may be used, or the color may be switched at appropriate time intervals.

In the present embodiment, such switching of the display mode of the composite image is realized by rewriting the mapping data of the mapping table MPT.

In the abnormal mode, the output of the composite image itself may be stopped, or the message output unit 26 may output a message.

In FIG. 6C, in the abnormal mode, the composite image is not output, and only the message indicating that the right door is open is output. Further, in FIG. 6D, the composite image shown in FIG. 6B is displayed, and
The same message as in (c) is also output together. Figure 6
As shown in (d), not only the camera image affected by the change in the state of the movable part is not used for generating the composite image, but also the change in the state of the movable part (for example, the door is open) that causes the change. By also outputting a message notifying the user, it becomes possible to know the state of the vehicle such as the driver more quickly and accurately.

The message output unit 26 stores the message to be displayed in advance in the form of, for example, an image, and the status signal S1 indicating the state of the movable part of the vehicle and the mapping table output from the mapping table selection unit 41. MP
In response to the signal for selecting T, the message image to be displayed in the composite image is selected and sent to the video signal generation unit 24.

FIG. 7 is a diagram showing an example of a message image stored in the message output unit 26. In the example of FIG. 7, the message part is expressed in white ((255, 255, 255) when expressed in 256 gradations for each of RGB, and the other parts are expressed in black (0, 0, 0). The size of the message image is the same as the size of the composite image.

The content and display position of the message in the message image differ depending on the mapping table used to generate the composite image and the type of the movable part in which the state change has occurred. For this reason, it is desirable that a message image be prepared for each assumed condition and that the message image be selected and used according to the situation.

FIG. 8 is a diagram showing the correspondence relationship between the mapping table used for the composite image and the state of the movable portion of the vehicle, and the message image to be displayed. For example, 2 in FIG.
The second line is the message image No. 1 when the currently selected mapping table is for creating a "looking down" image and the "right door" of the moving parts of the vehicle is open. This means selecting an image. FIG. 7A is an example of the first message image. When the mapping table is "looking down" and the "right door" is open, the image captured by the camera installed on the right door is not used for image composition, so it is installed on the right door on the composite image. To the position where the camera image is supposed to be used,
The message "The right door is open" is displayed.

The message output unit 26 stores in advance the information as shown in FIG. 8 and receives and displays the status signal S1 indicating the status of the movable part of the vehicle and the signal for selecting the mapping table MPT. Select the message image that should be output and output it. The video signal generation unit 24 receives the message image output from the message output unit 26, and displays the message image superimposed on the combined image output from the pixel combining unit 21.

For superimposing the message images, for example, for each pixel, the RGB values in the composite image and the message image are respectively taken out and the one with the larger value may be output. For example, when this method is applied to the message image of FIG. 7A, the RGB value of the composite image is used for the black pixel portion of the message image, while the RGB value of the message image is always used for the white pixel portion of the message image. Will be used, resulting in an image in which only part of the message is superimposed on the composite image. In this case, it is desirable to fill a portion of the composite image where the camera image is not used with a dark color so that the white message does not become hard to see as a result of being superimposed.

Further, by using this superposition method, all movable parts are closed (that is, normal mode).
In the case of displaying the composite image of No. 2, an image in which the whole is painted in black (0, 0, 0) as shown in FIG. 7B may be prepared as the message image. That is, as the seventh message image shown in the bottom line of FIG.
An image as shown in (b) may be prepared. of course,
In the normal mode, the message output unit 26 may not output the message image.

Further, when a plurality of movable parts are in the open state, the following measures can be taken. For example, when "looking down" is selected as the mapping table and both the right door and the left door are open, the message output unit 26 firstly outputs the first message image corresponding to each condition. (For example, in FIG.
The images of (a)) and No. 2 (for example, FIG. 7C) are selected. Then, for each pixel, a new message image (for example, FIG. 7D) is generated by extracting the RGB values in the first image and the second image, respectively, and selecting the one with the larger value. It may be sent to the video signal generator 24.

Although the character is used as the message here, the message is not limited to the character and, for example, a simplified drawing or an illustration may be used as the message. Needless to say, the message output unit 26 may be omitted when the message is not output in the abnormal mode.

The rewriting of the mapping data will be described in detail below.

The "mapping table" refers to a table in which the correspondence between the pixels of the composite image and the pixel data of each camera image is described, and is used to perform the composite image generation processing at high speed.

The pixel of the camera image and the pixel of the composite image are
There may be a one-to-one correspondence, or a plurality of one-to-one correspondence. Whether one-to-one correspondence or multiple-to-one correspondence depends on the installation position of the camera, the type of synthesized image to be generated, and the like. For example, when the boundary processing is performed at the boundary portion of the camera image, the pixels of the camera image and the pixels of the combined image often correspond to each other in a plural-to-one manner.

FIG. 9 is a schematic diagram showing an example of the mapping table. In the mapping table shown in FIG. 9, mapping data is provided corresponding to each pixel coordinate (i, j) of the composite image.

For example, mapping data MP1 is provided corresponding to the pixel coordinates (i1, j1). This mapping data MP1 has the coordinates (i1, j
This indicates that only pixel data at coordinates (12, 45) of the image captured by the camera 1 is used for generating the pixel 1). Further, since the necessity degree is “1”, in this case, the pixel data of the coordinates (12, 45) of the captured image of the camera 1 is used as it is as the pixel value of the coordinates (i1, j1) of the combined image.

Further, mapping data MP2 is provided corresponding to the pixel coordinates (i2, j2). This mapping data MP2 has coordinates (i2, j
It is shown that the pixel data of the coordinates (10, 10) of the captured image of the camera 1 and the pixel data of the coordinates (56, 80) of the captured image of the camera 0 are used for the generation of the pixel of 2). There is. In this case, the coordinates (i2, j2) of the composite image
The pixel value of is, for example, pixel value = {(pixel value of coordinates (56,80) of camera 1)
× 0.5 + (pixel value of coordinates (10,10) of camera 0)
X0.3} / (0.5 + 0.3) can be determined.

By creating such a mapping table in advance by calculation using geometrical transformation or the like or by manual work, it is possible to represent a desired composite image, for example, the entire periphery of the vehicle or the rear or side. A composite image can be generated at high speed. Then, by switching the mapping table used for generating the composite image, the display mode of the composite image can be easily switched. The switching of the display modes can be performed manually or automatically according to the running state of the vehicle.

The mapping table is specifically, for example, a ROM (writable / erasable ROM such as EEPROM).
Stored in RAM). The storage of the mapping table may be realized, for example, by writing the mapping data calculated by the processor in the image processing unit in the ROM or the RAM, or the mapping table data provided as firmware may be stored in the communication line or the disk drive. R using data transfer means such as
You may make it write in AM or ROM.

Next, the operation of generating a composite image using the mapping table MPT and the display switching section 23 will be described.

FIG. 10 is a flow chart showing the overall flow of the image synthesizing operation according to this embodiment. First, in step S11, the frame memory 12a of the imaging unit 1,
12b is switched. The timing generation unit 25 generates a timing signal for generating a moving image series of synthetic images. In response to the frame start timing signal output from the timing generation unit 25, the imaging unit 1 causes the cameras 11 to
A frame memory for writing the captured image of the
Switch to the frame memory referenced from. As will be described later, this is because the positions of the pixels of the camera image referred to by the image processing unit 2 are scattered irrespective of the order of writing from the camera 11, so that writing and reference do not interfere with each other. This is because.

Next, in step S12, the timing generation section 25 sets the pixels of the composite image to be output in synchronization with the composite output timing.

Then, in step S 13, the mapping table reference section 22 reads the mapping data corresponding to the pixel set by the timing generation section 25 from the set mapping table MPT and outputs it to the display switching section 23. In step S20, the display switching unit 23 receives the status signal S1 and, if necessary,
The mapping data read from the mapping table reference unit 22 is rewritten. Then, it outputs to the pixel composition unit 21. Details of step S20 will be described later.

In step S30, the pixel composition unit 21
Composes the pixel values set by the timing generation unit 25 according to the input mapping data and outputs them to the video signal generation unit 24. Then, in step S14, the video signal generation unit 24 causes the timing generation unit 25 to operate.
The pixel value of the input composite image is converted into a video signal in accordance with the timing signal from and output to the display device 3.

The timing generating section 25 has a step S12.
The following processing is executed for all pixels of the frame (S1
5, S16), when the processing of the last pixel of the frame is completed,
Start processing the next frame.

Of course, it is also possible to execute processing in field units.

FIG. 11 shows step S of the flow shown in FIG.
7 is a flowchart showing the operation of the display switching unit 23 in FIG.

First, in step S21, one of the movable parts of the vehicle, that is, the open / closed state of the right door, the left door, the trunk or the hood is read from the input state signal S1. Then, when the movable part is in the open state (YES in S22), the process proceeds to step S23, and the number of the camera installed in the movable part is obtained from the data stored in advance as shown in FIG. on the other hand,
In the closed state (NO in S22), the open / closed states of other movable parts are read (S27, S21).

In step S25, it is checked whether or not the camera number obtained in step S23 matches the camera number written in the mapping data. When they match (S25
If YES, the process proceeds to step S26, and the data regarding the camera number in the mapping data is rewritten to the content indicating the camera unusable state. In the example of FIG. 11,
The values of the camera number, the X coordinate, and the Y coordinate are all rewritten to "-1". On the other hand, when the camera numbers do not match (NO in S25), the open / closed states of other movable parts are read (S27, S21).

When the above process is completed for all the movable parts (YES in S27), the process proceeds to step S28, the necessity degree of the mapping data is recalculated, and the process is completed.

The operation of the display switching unit 23 in step S20 will be specifically described by taking the case where the mapping data MP2 shown in FIG. 9 is input to the display switching unit 23 as an example. For the sake of explanation, it is assumed here that only the right door of the vehicle is open.

First, since it is known that the right door of the vehicle is in the open state (S21, S22), referring to the correspondence shown in FIG. 5, "1" and "2" are set as the camera numbers of the cameras installed in the right door. Is obtained (S23).

Next, the camera numbers "1" and "2" are
It is checked whether or not the camera number described in the mapping data MP2 matches (S25). Since the camera number “1” is described in the mapping data MP2, the camera 1
Is rewritten to the content indicating the camera unusable state (S26). On the other hand, since the camera number “2” is not described in the mapping data MP2,
Do not rewrite.

Since all the other movable parts are in the closed state, the process proceeds to step S28, and the necessity degree is recalculated.
Here, the necessity level of the camera 1 that has become unusable is set to “0”, and the necessity level of the camera 0 is set to “1”.

FIG. 12 is a flow chart showing details of the pixel synthesizing process in step S30 in the flow shown in FIG.

First, in step S31, the value of the pixel to be combined is initialized to "0".

Then, in step S32, the camera number, the X and Y coordinates of the pixel, and the necessity degree are read from the corresponding mapping data. When the read camera number is usable, the pixel values corresponding to the X and Y coordinates are read and held from the frame memory in which the corresponding camera image is stored (S34). On the other hand, when the read camera number is unusable, a predetermined pixel value (for example, black) indicating that the camera is unusable is held and output (S35, S3).
9).

In step S36, the held pixel value is weighted by the degree of necessity and added to the value of the pixel to be combined.

The above-mentioned processing is repeatedly executed for all camera numbers described in the mapping data (S
37, S38), when the processing is completed (YE in S37)
S), the value of the pixel to be combined is output (S39).

As described above, by switching the display mode of the composite image, the state of the movable portion of the vehicle changes,
Even when the composite image is not normal, it is possible to avoid making the user feel uncomfortable.

(Second Embodiment) FIG. 13 shows a second embodiment of the present invention.
2 is a block diagram showing the configuration of a monitoring system according to the embodiment of FIG. 13, constituent elements common to FIG. 1 are assigned the same reference numerals as those in FIG. 1, and detailed description thereof will be omitted here.

In the configuration of FIG. 13, the display switching unit 23A switches the presence / absence of a camera image input from the image pickup unit 1 to the pixel synthesizing unit 21A according to the state signal S1.
It is characterized in that the display mode of the composite image is switched. That is, the switch 31 is provided on the camera image input side of the pixel composition unit 21A, and the display switching unit 23A
When it is recognized that any of the movable parts is in the open state by the state signal S1, the camera that cannot be used for the composite image generation is identified from the information as shown in FIG. The switch 31 is switch-controlled so as not to be input.

Therefore, in the present embodiment, the image composition is performed using the mapping table as in the first embodiment, but the process S20 for changing the mapping data in the flow shown in FIG. 10 is omitted. This is different from the first embodiment. That is, since the input of the image captured by the camera is physically stopped, the process S20 of changing the mapping data according to the state of the movable part is naturally unnecessary. Along with this, steps S33 and S35 in the flow of FIG. 12 are also omitted.

The configuration of the second embodiment can be easily applied to an image processing unit that does not use a mapping table.

In each of the above-mentioned embodiments, the display device 3 is typically a liquid crystal display, but other display devices such as a plasma display may be used. In addition, a vehicle-mounted GPS terminal display,
It may be shared with the display of a so-called car navigation system. Further, the camera 11 used in the image pickup unit 1 is typically a color or monochrome digital camera having a solid-state image pickup device such as a CCD or CMOS device, but other image pickup devices may be used.

In each of the above-described embodiments, it is assumed that the camera installation position or orientation changes due to the change in the state of a movable part such as a door or bonnet, which causes the synthesized image to be abnormal. There is. However, other than this, there is a possibility that the area occupied by the movable portion in the image captured by the camera may change due to the change in the state of the movable portion, which may cause the synthesized image to become abnormal. For example, there is a case where when the door is opened, the opened door is reflected in the captured image of the camera.

Even in such a case, instead of the data as shown in FIG. 5, data indicating the correspondence between the movable portion and the camera whose state change of the movable portion influences the picked-up image is previously displayed. If it is prepared in advance, the same effect can be obtained by the same processing as each of the above-described embodiments.

Instead of the binary information of the open / closed state, it is possible to use information such as the opening degree. For example, the normal mode may be maintained when the degree of opening of the door is minute, and the normal mode may be switched to the abnormal mode when the degree of opening exceeds a predetermined threshold value. The movable parts are not limited to doors, bonnets, and trunks, but tires, antennas, movable mirrors, lights, etc. can be considered. For example, when the tire affects the composite image, the display mode may be switched according to the steering angle of the tire.

In the present invention, the vehicle includes ordinary cars, light cars, freight cars, buses and the like. Further, a special vehicle such as a crane truck or a shovel car can be the vehicle of the present invention as long as the technical idea of the present invention can be applied.

In the above description, the monitoring system and the image processing apparatus according to the present invention are applied to a vehicle, but the same applies to a moving body other than the vehicle, such as an airplane or a ship. can do. Further, the camera may be installed in a monitoring target other than the moving body, such as a store, a residence, or a showroom.

The installation positions and the number of cameras are as follows.
It is not limited to those shown here.

The functions of the image processing apparatus according to the present invention may be realized in whole or in part by using dedicated hardware, or by software. It is also possible to use a recording medium or a transmission medium that stores a program for causing a computer to execute all or some of the functions of the image processing apparatus according to the present invention.

[0074]

As described above, according to the present invention, even if an abnormality occurs in the composite image due to the change in the state of the movable portion of the vehicle, an image that makes the user feel uncomfortable is displayed. Can be avoided. Further, by switching the display mode of the composite image, it becomes possible to promptly and accurately inform the user that the movable portion is in a state not desirable for traveling of the vehicle, for example.

[Brief description of drawings]

FIG. 1 is a block diagram showing a configuration of a monitoring system according to a first embodiment of the present invention.

FIG. 2 is an example of camera arrangement in the first embodiment of the present invention.

FIG. 3 is a diagram showing an example of camera arrangement and a captured image of each camera according to the first embodiment of the present invention.

FIG. 4 is an example of a composite image that is no longer normal due to a change in the state of a movable portion of a vehicle.

FIG. 5 is data showing a correspondence relationship between a vehicle movable portion and a camera included in the display switching unit.

FIG. 6 is a diagram showing an example of switching a display mode of a composite image according to the present invention.

FIG. 7 is a diagram showing an example of a message image.

FIG. 8 is a diagram showing a correspondence relationship between a mapping table and states of movable parts of a vehicle and message images to be displayed.

FIG. 9 is a schematic diagram showing an example of a mapping table.

FIG. 10 is a flowchart showing the overall flow of an image combining operation.

FIG. 11 is a step S20 in the flow shown in FIG.
3 is a flowchart showing the details of FIG.

FIG. 12 is a step S30 in the flow shown in FIG.
3 is a flowchart showing the details of FIG.

FIG. 13 is a block diagram showing a configuration of a monitoring system according to a second embodiment of the present invention.

[Explanation of symbols]

2 Image processing unit 3 display devices 11 cameras

   ─────────────────────────────────────────────────── ─── Continued front page    (72) Inventor Hiroshi Ishii             1006 Kadoma, Kadoma-shi, Osaka Matsushita Electric             Sangyo Co., Ltd. (72) Inventor, Issei             1006 Kadoma, Kadoma-shi, Osaka Matsushita Electric             Sangyo Co., Ltd. (72) Inventor Jun Morimura             1006 Kadoma, Kadoma-shi, Osaka Matsushita Electric             Sangyo Co., Ltd. F-term (reference) 5C054 AA01 AA05 CA04 CC03 CF01                       CF05 FA00 FC12 FC13 FE11                       FE16 FE25 FE26 FE28 FF03                       HA30

Claims (2)

[Claims] 1. An image processing unit installed in a vehicle, which receives imaged images of a plurality of cameras for capturing the surroundings of the vehicle, generates a composite image from the imaged images, and outputs the image to a display device. The image processing unit displays the composite image when the installation position or orientation of at least one of the plurality of cameras changes due to the state change of the movable part of the vehicle and the composite image becomes abnormal. The mode is switched from the normal mode to the abnormal mode, and in the abnormal mode, instead of the synthetic image or together with the synthetic image, a message is output to inform the user of the state change of the movable part causing the abnormal mode. The image processing apparatus, wherein the message differs depending on the type of the movable part in which the state change has occurred. 2. The image processing apparatus according to claim 1, wherein the message includes content that specifies a movable part in which a state change has occurred.