picom: make sure --diagnostics works when another picom is running

Currently picom fails at register_cm(), which is before
print_diagnostics(), so we don't see anything.

This commit makes sure we keep going when there is another compositor
already running. However, in this case we need to be careful to not take
the overlay window, otherwise a poorly implemented compositor (for
example, picom) will stop working.

Also restructured the target window initialization logic and added some
comments explaining the logic.

Fixes #333

src/picom.h

@@ -61,6 +61,8 @@ void quit(session_t *ps);
 
 xcb_window_t session_get_target_window(session_t *);
 
+uint8_t session_redirection_mode(session_t *ps);
+
 /**
  * Set a <code>switch_t</code> array of all unset wintypes to true.
  */