Camera Movement Correction

Background

A common issue to deal with is that after a camera has been spatially calibrated, the camera then moves and needs to be re-calibrated. The calcam.movement module contains tools for dealing with this. The problem is framed as movement of the camera image (a “moved image”) with respect to a “reference image” e.g. an image that has already been calibrated. There are two approaches to deal with this:

  • Apply transformations to warped images, or pixel coordinates in warped images, to make them conform to a reference calibration.

  • Update the reference calibration to apply to the moved image.

Which approach is preferable will depend on how you are using calcam results. See the detailed documentation below for details of how these methods are implemented in calcam.movement, or see the Examples page for examples of how to use this functionality.

Determining Camera Movement

Calcam can try to detect the movement between two images automatically, or there is a GUI tool which allows manual feature matching by the user or automatic detection but user validation.

calcam.movement.detect_movement(ref, moved)

Attempt to auto-detect image movement between two images using sparse optical flow and return a MovementCorrection object representing the movement. If the movement cannot be successfully determined, raises calcam.movement.DetectionFailedError

Parameters

  • ref (np.ndarray or calcam.Calibration) – Reference image or calibration to align to. This can be either an array containg a refrence image, or a calcam calibrationn containing an image.

  • moved (np.ndarray or calcam.Calibration) – Moved image or calibration to align. This can be either an array containg a refrence image, or a calcam calibrationn containing an image.

Returns

Movement correction object representing the movement correction.

Return type

MovementCorrection

Raises

DetectionFailedError –

calcam.movement.phase_correlation_movement(ref, moved)

Attempt to auto-detect a simple rigid translation of an image relative to a reference image using Phase Correlation and return a MovementCorrection object representing the movement.

Parameters

  • ref (np.ndarray or calcam.Calibration) – Reference image or calibration to align to. This can be either an array containg a refrence image, or a calcam calibrationn containing an image.

  • moved (np.ndarray or calcam.Calibration) – Moved image or calibration to align. This can be either an array containg a refrence image, or a calcam calibrationn containing an image.

Returns

Movement correction object representing the movement correction.

Return type

MovementCorrection

Raises

DetectionFailedError –

calcam.movement.manual_movement(ref, moved, correction=None, parent_window=None)

Determine camera movement (semi-)manually using a GUI tool. See the Camera Movement Determination GUI doc page for the GUI user guide.

Paremeters:

ref (np.ndarray or calcam.Calibration) : Reference image or calibration to align to. This can be either an array containg a refrence image, or a calcam calibrationn containing an image.

mmoved (np.ndarray or calcam.Calibration) : Moved image or calibration to align. This can be either an array containg a refrence image, or a calcam calibrationn containing an image.

correction (MovementCorrection) : Existing movement correction object to start from.

parent_window (QWidget) : If being called from a QT GUI window class, a reference to the parent window must be passed. If it is not, the parent window might irrecoverably freeze when the movement dialog is closed.

Returns

Either the image transofmration, or None is the user does not define one.

Return type

MovementCorrection or NoneType

The MovementCorrection class

The above functions return movement correction objects, which represent the geometrical transform between the reference and moved images. These objects then have various methods for transforming between reference and moved images and coordinates.

class calcam.movement.MovementCorrection(matrix, im_shape, ref_points, moved_points, src)

Class to represent a geometric transform from a moved image to a reference image. This type of object is returned by image movement correction related functions.

Parameters

  • matrix (np.matrix) – 2x3 or 3x3 Affine or projective transform matrix

  • im_shape (tuple) – Image array dimensions (rows,cols) to which this transform applies

  • ref_points (np.ndarray) – Nx2 array containing coordinates of points on the reference image

  • moved_points (np.ndarray) – Nx2 array containing coordinates of corresponding points on the moved image

  • src (string) – Human-readable description of how the transform was created.

get_ddscore(ref_im, moved_im, tol=50)

Get the DDScore for this movement correction when applied to the given image pair. DDScore is a score developed by Van-Stroud et.al. which indicates how much better the alignment of two input images after application of the correction. The score ranges from -1 to 1, where negative values indicate the alignment gets worse, 0 is no change or undetermined, and positive values are an improvement.

Parameters

  • ref_im (np.ndarray) – Reference aligned image

  • moved_im (np.ndarray) – Moved image

  • tol (float) – Tolerance

Returns

DDscore in the range -1,1

Return type

float

classmethod load(filename)

Load a movement correction from a .cmc file on disk.

Parameters

filename (string) – File name to load from.

moved_to_ref_coords(x, y)

Given image coordinates in the moved image, return the corresponding coordinates in the reference image.

Parameters

  • x (np.ndarray) – Array of x (horizontal) pixel coordinates in the moved image

  • y (np.ndarray) – Array of y (vertucal) pixel coordinates in the moved image

Returns

Array of x (horizontal) pixel coordinates in the reference image np.ndarray : Array of y (vertical) pixel coordinates in the reference image

Return type

np.ndarray

ref_to_moved_coords(x, y)

Given image coordinates in the reference image, return the corresponding coordinates in the moved image.

Parameters

  • x (np.ndarray) – Array of x (horizontal) pixel coordinates in the reference image

  • y (np.ndarray) – Array of y (vertucal) pixel coordinates in the reference image

Returns

Array of x (horizontal) pixel coordinates in the moved image np.ndarray : Array of y (vertical) pixel coordinates in the moved image

Return type

np.ndarray

property rotation

Image rotation to go from the moved to reference image in degrees clockwise.

save(filename)

Save the correction to a given filename.

Paremeters:

filaneme (string) : Filename to which to save the correction. The file extension is .cmc; if this is not included in the given filename it is added.

property scale

Scale factor to go from moved to reference image.

property translation

x, y translation in pixels to go from the moved to reference image.

warp_moved_to_ref(image, interp='linear', fill_edges=False)

Warp a moved image to align with the reference one. Note: this can also be used on binned or englarged images, with respect to the one used for the original movement correction determination, provided they are scaled proportionally i.e. have the same aspect ratio as the originals.

Parameters

  • image (np.ndarray) – Moved image to warp

  • interp (string) – Interpolation method to use, can be ‘linear’ or ‘nearest’

  • fill_edges (bool) – Whether to fill the warped image edges with a repetition of the edge pixels from the image (if True), or leave un-filled images edges as 0 value (if False; this is the default).

Returns

Two ndarrays: the warped image, and a boolean mask the same shape as the image indicating which pixels contain valid image data (True) and which do not (False).

Return type

np.ndarray

warp_ref_to_moved(image, interp='linear', fill_edges=False)

Warp a reference-aligned image to align with the moved one. Note: this can also be used on binned or englarged images, with respect to the one used for the original movement correction determination, provided they are scaled proportionally i.e. have the same aspect ratio as the originals.

Parameters

  • image (np.ndarray) – Image to warp

  • interp (string) – Interpolation method to use, can be ‘linear’ or ‘nearest’

Returns

Two ndarrays: the warped image, and a boolean mask the same shape as the image indicating which pixels contain valid image data (True) and which do not (False).

Return type

np.ndarray

Adjusting calibrations for image movement

Having obtained a movement correction object describing the movement, you can update an exicting calcam calibration to apply to the new image:

calcam.movement.update_calibration(calibration, moved_image, mov_correction, image_src=None, coords='Display')

Update a given calibration to account for image movement. This currently only supports point pair fitting calibrations, not manual alignment calibrations.

Parameters

  • calibration (calcam.Calibration) – Calibration to update.

  • moved_image (np.ndarray) – Moved image

  • mov_correction (MovementCorrection) – Movement correction object representing the movement between the original calibrated image and the moved image.

  • image_src (string) – Human-readable description of where the moved image comes from, for data provenance tracking.

  • coords (string) – ‘Display’ or ‘Original’, whether the movement correction and moved image are in the calibration’s display or original image orientation.

Returns

Updated calibration object for the moved image

Return type

calcam.Calibration

Exceptions

class calcam.movement.DetectionFailedError

Exception raised if the automatic image movement detection fails to determine the image movement with sufficient confidence or quality.