Documentation

While this documentation aims to go beyond a simple listing of parameters and instead attempts to explain some of the principles behind the functions, please see the section “Usage” for more details and usage examples including code and flow field visualisations.

Using the Flow Class

This section documents the custom flow class and all its class methods. It is the recommended way of using oflibnumpy and makes the full range of functionality available to the user.

Flow Constructors and Operators

class oflibnumpy.Flow(flow_vectors: numpy.ndarray, ref: Optional[str] = None, mask: Optional[numpy.ndarray] = None)
__init__(flow_vectors: numpy.ndarray, ref: Optional[str] = None, mask: Optional[numpy.ndarray] = None)Flow

Flow object constructor. For a more detailed explanation of the arguments, see the class attributes vecs, ref, and mask.

Parameters
  • flow_vectors – Numpy array of shape \((H, W, 2)\) containing the flow vector in OpenCV convention: flow_vectors[..., 0] are the horizontal, flow_vectors[..., 1] are the vertical vector components, defined as positive when pointing to the right / down.

  • ref – Flow reference, either t for “target”, or s for “source”. Defaults to t

  • mask – Numpy array of shape \((H, W)\) containing a boolean mask indicating where the flow vectors are valid. Defaults to True everywhere.

property vecs

Flow vectors, a numpy array of shape \((H, W, 2)\). The last dimension contains the flow vectors. These are in the order horizontal component first, vertical component second (OpenCV convention). They are defined as positive towards the right and the bottom, meaning the origin is located in the left upper corner of the \(H \times W\) flow field area.

Returns

Flow vectors as numpy array of shape \((H, W, 2)\) and type float32

property ref

Flow reference, a string: either s for “source” or t for “target”. This determines whether the regular grid of shape \((H, W)\) associated with the flow vectors should be understood as the source of the vectors (which then point to any other position), or the target of the vectors (whose start point can then be any other position). The flow reference t is the default, meaning the regular grid refers to the coordinates the pixels whose motion is being recorded by the vectors end up at.

Applying a flow with reference s is known as “forward” warping, while reference t corresponds to what is termed “backward” or “reverse” warping.

Caution

The apply() method for warping an image is significantly faster with a flow in t reference. The reason is that this requires interpolating unstructured points from a regular grid, while reference s requires interpolating a regular grid from unstructured points. The former uses the fast OpenCV remap() function, the latter is much more operationally complex and relies on the SciPy griddata() function.

Caution

The track() method for tracking points is significantly faster with a flow in s reference, again due to not requiring a call to SciPy’s griddata() function.

Tip

If some algorithm get_flow() is set up to calculate a flow field with reference t (or s) as in flow_one_ref = get_flow(img1, img2), it is very simple to obtain the flow in reference s (or t) instead: simply call the algorithm with the images in the reversed order, and multiply the resulting flow vectors by -1: flow_other_ref = -1 * get_flow(img2, img1)

Returns

Flow reference, as string of value t or s

property mask

Flow mask as a numpy array of shape \((H, W)\) and type bool. This array indicates, for each flow vector, whether it is considered “valid”. As an example, this allows for masking of the flow based on object segmentations. It is also necessary to keep track of which flow vectors are valid when different flow fields are combined, as those operations often lead to undefined (partially or fully unknown) points in the given \(H \times W\) area where the flow vectors are either completely unknown, or will not have valid values.

Returns

Flow mask as numpy array of shape \((H, W)\) and type bool

property shape

Shape (resolution) \((H, W)\) of the flow, corresponding to the first two dimensions of the flow vector array of shape \((H, W, 2)\)

Returns

Tuple of the shape (resolution) \((H, W)\) of the flow object

classmethod zero(shape: Union[list, tuple], ref: Optional[str] = None, mask: Optional[numpy.ndarray] = None)Flow

Flow object constructor, zero everywhere

Parameters
  • shape – List or tuple of the shape \((H, W)\) of the flow field

  • ref – Flow reference, string of value t (“target”) or s (“source”). Defaults to t

  • mask – Numpy array of shape \((H, W)\) and type bool indicating where the flow vectors are valid. Defaults to True everywhere

Returns

Flow object

classmethod from_matrix(matrix: numpy.ndarray, shape: Union[list, tuple], ref: Optional[str] = None, mask: Optional[numpy.ndarray] = None)Flow

Flow object constructor, based on transformation matrix input

Parameters
  • matrix – Transformation matrix to be turned into a flow field, as numpy array of shape \((3, 3)\)

  • shape – List or tuple of the shape \((H, W)\) of the flow field

  • ref – Flow reference, string of value t (“target”) or s (“source”). Defaults to t

  • mask – Numpy array of shape \((H, W)\) and type bool indicating where the flow vectors are valid. Defaults to True everywhere

Returns

Flow object

classmethod from_transforms(transform_list: list, shape: Union[list, tuple], ref: Optional[str] = None, mask: Optional[numpy.ndarray] = None)Flow

Flow object constructor, based on list of transforms

Parameters
  • transform_list

    List of transforms to be turned into a flow field, where each transform is expressed as a list of [transform name, transform value 1, … , transform value n]. Supported options:

    • Transform translation, with values horizontal shift in px, vertical shift in px

    • Transform rotation, with values horizontal centre in px, vertical centre in px, angle in degrees, counter-clockwise

    • Transform scaling, with values horizontal centre in px, vertical centre in px, scaling fraction

  • shape – List or tuple of the shape \((H, W)\) of the flow field

  • ref – Flow reference, string of value t (“target”) or s (“source”). Defaults to t

  • mask – Numpy array of shape \((H, W)\) and type bool indicating where the flow vectors are valid. Defaults to True everywhere

Returns

Flow object

classmethod from_kitti(path: str, load_valid: Optional[bool] = None)Flow

Loads the flow field contained in KITTI uint16 png images files, optionally including the valid pixels. Follows the official instructions on how to read the provided .png files

Parameters
  • path – String containing the path to the KITTI flow data (uint16, .png file)

  • load_valid – Boolean determining whether the valid pixels are loaded as the flow mask. Defaults to True

Returns

A flow object corresponding to the KITTI flow data, with flow reference ref s.

classmethod from_sintel(path: str, inv_path: Optional[str] = None)Flow

Loads the flow field contained in Sintel .flo byte files, including the invalid pixels if required. Follows the official instructions provided alongside the .flo data.

Parameters
  • path – String containing the path to the Sintel flow data (.flo byte file, little Endian)

  • inv_path – String containing the path to the Sintel invalid pixel data (.png, black and white)

Returns

A flow object corresponding to the Sintel flow data, with flow reference ref s

copy()Flow

Copy a flow object by constructing a new one with the same vectors vecs, reference ref, and mask mask

Returns

Copy of the flow object

__str__()str

Enhanced string representation of the flow object, containing the flow reference ref and shape shape

Returns

String representation

__getitem__(item: Union[int, list, slice])Flow

Mimics __getitem__ of a numpy array, returning a new flow object cut accordingly

Will throw an error if mask.__getitem__(item) or vecs.__getitem__(item) (corresponding to mask[item] and vecs[item]) throw an error. Also throws an error if sliced vecs or mask are not suitable to construct a new flow object with, e.g. if the number of dimensions is too low.

Parameters

item – Slice used to select a part of the flow

Returns

New flow object cut as a corresponding numpy array would be cut

__add__(other: Union[numpy.ndarray, Flow])Flow

Adds a flow object or a numpy array to a flow object

Caution

This is not equal to applying the two flows sequentially. For that, use combine_with() with mode set to 3.

Caution

If this method is used to add two flow objects, there is no check on whether they have the same reference ref.

Parameters

other – Flow object or numpy array corresponding to the addend. Adding a flow object will adjust the mask of the resulting flow object to correspond to the logical union of the augend / addend masks

Returns

New flow object corresponding to the sum

__sub__(other: Union[numpy.ndarray, Flow])Flow

Subtracts a flow object or a numpy array from a flow object

Caution

This is not equal to subtracting the effects of applying flow fields to an image. For that, use combine_with() with mode set to 1 or 2.

Caution

If this method is used to subtract two flow objects, there is no check on whether they have the same reference ref.

Parameters

other – Flow object or numpy array corresponding to the subtrahend. Subtracting a flow object will adjust the mask of the resulting flow object to correspond to the logical union of the minuend / subtrahend masks

Returns

New flow object corresponding to the difference

__mul__(other: Union[float, int, list, numpy.ndarray])Flow

Multiplies a flow object with a single number, a list, or a numpy array

Parameters

other

Multiplier, options:

  • can be converted to a float

  • a list of shape \((2)\)

  • an array of the same shape \((H, W)\) as the flow object

  • an array of the same shape \((H, W, 2)\) as the flow vectors

Returns

New flow object corresponding to the product

__truediv__(other: Union[float, int, list, numpy.ndarray])Flow

Divides a flow object by a single number, a list, or a numpy array

Parameters

other

Divisor, options:

  • can be converted to a float

  • a list of shape \((2)\)

  • an array of the same shape \((H, W)\) as the flow object

  • an array of the same shape \((H, W, 2)\) as the flow vectors

Returns

New flow object corresponding to the quotient

__pow__(other: Union[float, int, bool, list, numpy.ndarray])Flow

Exponentiates a flow object by a single number, a list, or a numpy array

Parameters

other

Exponent, options:

  • can be converted to a float

  • a list of shape \((2)\)

  • an array of the same shape \((H, W)\) as the flow object

  • an array of the same shape \((H, W, 2)\) as the flow vectors

Returns

New flow object corresponding to the power

__neg__()Flow

Returns a new flow object with all the flow vectors inverted

Caution

This is not equal to inverting the transformation a flow field corresponds to! For that, use invert().

Returns

New flow object with inverted flow vectors

Manipulating the Flow

Flow.resize(scale: Union[float, int, list, tuple])Flow

Resize a flow field, scaling the flow vectors values vecs accordingly.

Parameters

scale

Scale used for resizing, options:

  • Integer or float of value scaling applied both vertically and horizontally

  • List or tuple of shape \((2)\) with values [vertical scaling, horizontal scaling]

Returns

New flow object scaled as desired

Flow.pad(padding: Optional[Union[list, tuple]] = None, mode: Optional[str] = None)Flow

Pad the flow with the given padding. Padded flow vecs values are either constant (set to 0), correspond to the edge values of the flow field, or are a symmetric mirroring of the existing flow values. Padded mask values are set to False.

Parameters
  • padding – List or tuple of shape \((4)\) with padding values [top, bot, left, right]

  • mode – String of the numpy padding mode for the flow vectors, with options constant (fill value 0), edge, symmetric (see documentation for numpy.pad()). Defaults to constant

Returns

New flow object with the padded flow field

Flow.invert(ref: Optional[str] = None)Flow

Inverting a flow: img1f –> img2 becomes img1 <– fimg2. The smaller the input flow, the closer the inverse is to simply multiplying the flow by -1.

Parameters

ref – Desired reference of the output field, defaults to the reference of original flow field

Returns

New flow object, inverse of the original

Flow.switch_ref(mode: Optional[str] = None)Flow

Switch the reference ref between s (“source”) and t (“target”)

Caution

Do not use mode=invalid if avoidable: it does not actually change any flow values, and the resulting flow object, when applied to an image, will no longer yield the correct result.

Parameters

mode

Mode used for switching, available options:

  • invalid: just the flow reference attribute is switched without any flow values being changed. This is functionally equivalent to simply assigning flow.ref = 't' for a “source” flow or flow.ref = 's' for a “target” flow

  • valid (default): the flow field is switched to the other coordinate reference, with flow vectors recalculated accordingly

Returns

New flow object with switched coordinate reference

Flow.combine_with(flow: Flow, mode: int, thresholded: Optional[bool] = None)Flow

Function that returns the result of the combination of two flow objects of the same shape shape and reference ref

Tip

All of the flow field combinations in this function rely on some combination of the apply(), invert(), and combine_with() methods, and can be very slow (several seconds) due to calling scipy.interpolate.griddata() multiple times. The table below aids decision-making with regards to which reference a flow field should be provided in to obtain the fastest result.

Calls to scipy.interpolate.griddata()

mode

ref = 's'

ref = 't'

1

1

3

2

1

1

3

0

0

All formulas used in this function have been derived from first principles. The base formula is \(flow_1 ⊕ flow_2 = flow_3\), where \(⊕\) is a non-commutative flow composition operation. This can be visualised with the start / end points of the flows as follows:

S = Start point    S1 = S3 ─────── f3 ────────────┐
E = End point       │                             │
f = flow           f1                             v
                    └───> E1 = S2 ── f2 ──> E2 = E3

The main difficulty in combining flow fields is that it would be incorrect to simply add up or subtract flow vectors at one location in the flow field area \(H \times W\). This appears to work given e.g. a translation to the right, and a translation downwards: the result will be the linear combination of the two vectors, or a translation towards the bottom right. However, looking more closely, it becomes evident that this approach isn’t actually correct: A pixel that has been moved from S1 to E1 by the first flow field f1 is then moved from that location by the flow vector of the flow field f2 that corresponds to the new pixel location E1, not the original location S1. If the flow vectors are the same everywhere in the field, the difference will not be noticeable. However, if the flow vectors of f2 vary throughout the field, such as with a rotation around some point, it will!

In this case (corresponding to calling f1.combine_with(f2, mode=3)), and if the flow reference ref is s (“source”), the solution is to first apply the inverse of f1 to f2, essentially linking up each location E1 back to S1, and then to add up the flow vectors. Analogous observations apply for the other permutations of flow combinations and reference ref values.

Note

This is consistent with the observation that two translations are commutative in their application - the order does not matter, and the vectors can simply be added up at every pixel location -, while a translation followed by a rotation is not the same as a rotation followed by a translation: adding up vectors at each pixel cannot be the correct solution as there wouldn’t be a difference based on the order of vector addition.

Parameters
  • flow – Flow object to combine with

  • mode

    Integer determining how the input flows are combined, where the number corresponds to the position in the formula \(flow_1 ⊕ flow_2 = flow_3\):

    • Mode 1: self corresponds to \(flow_2\), flow corresponds to \(flow_3\), the result will be \(flow_1\)

    • Mode 2: self corresponds to \(flow_1\), flow corresponds to \(flow_3\), the result will be \(flow_2\)

    • Mode 3: self corresponds to \(flow_1\), flow corresponds to \(flow_2\), the result will be \(flow_3\)

  • thresholded – Boolean determining whether flows are thresholded during an internal call to is_zero(), defaults to False

Returns

New flow object

Applying the Flow

Flow.apply(target: Union[numpy.ndarray, Flow], target_mask: Optional[numpy.ndarray] = None, return_valid_area: Optional[bool] = None, consider_mask: Optional[bool] = None, padding: Optional[Union[list, tuple]] = None, cut: Optional[bool] = None)Union[numpy.ndarray, Flow, Tuple[Union[numpy.ndarray, Flow], numpy.ndarray]]

Apply the flow to a target, which can be a numpy array or a Flow object itself. If the flow shape \((H_{flow}, W_{flow})\) is smaller than the target shape \((H_{target}, W_{target})\), a list of padding values needs to be passed to localise the flow in the larger \(H_{target} \times W_{target}\) area.

The valid image area that can optionally be returned is True where the image values in the function output:

  1. have been affected by flow vectors. If the flow has a reference ref value of t (“target”), this is always True as the target image by default has a corresponding flow vector at each pixel location in \(H \times W\). If the flow has a reference ref value of s (“source”), this is only True for some parts of the image: some target image pixel locations in \(H \times W\) would only be reachable by flow vectors originating outside of the source image area, which is impossible by definition

  2. have been affected by flow vectors that were themselves valid, as determined by the flow mask

Caution

The parameter consider_mask relates to whether the invalid flow vectors in a flow field with reference s are removed before application (default behaviour) or not. Doing so results in a smoother flow field, but can cause artefacts to arise where the outline of the area returned by valid_target() is not a convex hull. For a more detailed explanation with an illustrative example, see the section “Applying a Flow” in the usage documentation.

Parameters
  • target – Numpy array of shape \((H, W)\) or \((H, W, C)\), or a flow object of shape \((H, W)\) to which the flow should be applied, where \(H\) and \(W\) are equal or larger than the corresponding dimensions of the flow itself

  • target_mask – Optional numpy array of shape \((H, W)\) that indicates which part of the target is valid (only relevant if target is a numpy array). Only impacts the valid area returned when return_valid_area = True. Defaults to True everywhere

  • return_valid_area – Boolean determining whether the valid image area is returned (only if the target is a numpy array), defaults to False. The valid image area is returned as a boolean numpy array of shape \((H, W)\).

  • consider_mask – Boolean determining whether the flow vectors are masked before application (only relevant for flows with reference ref = 's'). Results in smoother outputs, but more artefacts. Defaults to True

  • padding – List or tuple of shape \((4)\) with padding values [top, bottom, left, right]. Required if the flow and the target don’t have the same shape. Defaults to None, which means no padding needed

  • cut – Boolean determining whether the warped target is returned cut from \((H_{target}, W_{target})\) to \((H_{flow}, W_{flow})\), in the case that the shapes are not the same. Defaults to True

Returns

The warped target of the same shape \((C, H, W)\) and type as the input (rounded if necessary), and optionally the valid area of the flow as a boolean array of shape \((H, W)\)

Flow.track(pts: numpy.ndarray, int_out: Optional[bool] = None, get_valid_status: Optional[bool] = None, s_exact_mode: Optional[bool] = None)numpy.ndarray

Warp input points with the flow field, returning the warped point coordinates as integers if required

Tip

Calling track() on a flow field with reference ref s (“source”) is significantly faster (as long as s_exact_mode is not set to True), as this does not require a call to scipy.interpolate.griddata().

Parameters
  • pts – Numpy array of shape \((N, 2)\) containing the point coordinates. pts[:, 0] corresponds to the vertical coordinate, pts[:, 1] to the horizontal coordinate

  • int_out – Boolean determining whether output points are returned as rounded integers, defaults to False

  • get_valid_status – Boolean determining whether an array of shape \((N, 2)\) is returned, which contains the status of each point. This corresponds to applying valid_source() to the point positions, and returns True for the points that 1) tracked by valid flow vectors, and 2) end up inside the flow area of \(H \times W\). Defaults to False

  • s_exact_mode – Boolean determining whether the necessary flow interpolation will be done using scipy.interpolate.griddata(), if the flow has the reference ref value of s (“source”). Defaults to False, which means a less exact, but around 2 orders of magnitude faster bilinear interpolation method will be used. This is recommended for normal point tracking applications.

Returns

Numpy array of warped (‘tracked’) points, and optionally a numpy array of the point tracking status

Evaluating the Flow

Flow.is_zero(thresholded: Optional[bool] = None, masked: Optional[bool] = None)bool

Check whether all flow vectors (where mask is True) are zero. Optionally, a threshold flow magnitude value of 1e-3 is used. This can be useful to filter out motions that are equal to very small fractions of a pixel, which might just be a computational artefact to begin with.

Parameters
  • thresholded – Boolean determining whether the flow is thresholded, defaults to True

  • masked – Boolean determining whether the flow is masked with mask, defaults to True

Returns

True if the flow field is zero everywhere, otherwise False

Flow.matrix(dof: Optional[int] = None, method: Optional[str] = None, masked: Optional[bool] = None)numpy.ndarray

Fit a transformation matrix to the flow field using OpenCV functions

Parameters
  • dof

    Integer describing the degrees of freedom in the transformation matrix to be fitted, defaults to 8. Options are:

    • 4: Partial affine transform with rotation, translation, scaling

    • 6: Affine transform with rotation, translation, scaling, shearing

    • 8: Projective transform, i.e estimation of a homography

  • method

    String describing the method used to fit the transformations matrix by OpenCV, defaults to ransac. Options are:

    • lms: Least mean squares

    • ransac: RANSAC-based robust method

    • lmeds: Least-Median robust method

  • masked – Boolean determining whether the flow mask is used to ignore flow locations where the mask mask is False. Defaults to True

Returns

Numpy array of shape \((3, 3)\) containing the transformation matrix

Flow.valid_target(consider_mask: Optional[bool] = None)numpy.ndarray

Find the valid area in the target domain

Given a source image and a flow, both of shape \((H, W)\), the target image is created by warping the source with the flow. The valid area is then a boolean numpy array of shape \((H, W)\) that is True wherever the value in the target img stems from warping a value from the source, and False where no valid information is known.

Pixels that are False will often be black (or ‘empty’) in the warped target image - but not necessarily, due to warping artefacts etc. The valid area also allows a distinction between pixels that are black due to no actual information being available at this position (validity False), and pixels that are black due to black pixel values having been warped to that (valid) location by the flow.

Parameters

consider_mask – Boolean determining whether the flow vectors are masked before application (only relevant for flows with reference ref = 's', analogous to apply()). Results in smoother outputs, but more artefacts. Defaults to True

Returns

Boolean numpy array of the same shape \((H, W)\) as the flow

Flow.valid_source(consider_mask: Optional[bool] = None)numpy.ndarray

Finds the area in the source domain that will end up being valid in the target domain (see valid_target()) after warping

Given a source image and a flow, both of shape \((H, W)\), the target image is created by warping the source with the flow. The source area is then a boolean numpy array of shape \((H, W)\) that is True wherever the value in the source will end up somewhere inside the valid target area, and False where the value in the source will either be warped outside of the target image, or not be warped at all due to a lack of valid flow vectors connecting to this position.

Parameters

consider_mask – Boolean determining whether the flow vectors are masked before application (only relevant for flows with reference ref = 't' as their inverse flow will be applied, using the reference s; analogous to apply()). Results in smoother outputs, but more artefacts. Defaults to True

Returns

Boolean numpy array of the same shape \((H, W)\) as the flow

Flow.get_padding()list

Determine necessary padding from the flow field:

  • When the flow reference ref has the value t (“target”), this corresponds to the padding needed in a source image which ensures that every flow vector in vecs marked as valid by the mask mask will find a value in the source domain to warp towards the target domain. I.e. any invalid locations in the area \(H \times W\) of the target domain (see valid_target()) are purely due to no valid flow vector being available to pull a source value to this target location, rather than no source value being available in the first place.

  • When the flow reference ref has the value s (“source”), this corresponds to the padding needed for the flow itself, so that applying it to a source image will result in no input image information being lost in the warped output, i.e each input image pixel will come to lie inside the padded area.

Returns

A list of shape \((4)\) with the values [top, bottom, left, right]

Visualising the Flow

Flow.visualise(mode: str, show_mask: Optional[bool] = None, show_mask_borders: Optional[bool] = None, range_max: Optional[float] = None)numpy.ndarray

Visualises the flow as an rgb / bgr / hsv image, optionally showing the outline of the flow mask mask as a black line, and the invalid areas greyed out.

Parameters
  • mode – Output mode, options: rgb, bgr, hsv

  • show_mask – Boolean determining whether the flow mask is visualised, defaults to False

  • show_mask_borders – Boolean determining whether the flow mask border is visualised, defaults to False

  • range_max – Maximum vector magnitude expected, corresponding to the HSV maximum Value of 255 when scaling the flow magnitudes. Defaults to the 99th percentile of the flow field magnitudes

Returns

Numpy array of shape \((H, W, 3)\) containing the flow visualisation

Flow.visualise_arrows(grid_dist: Optional[int] = None, img: Optional[numpy.ndarray] = None, scaling: Optional[Union[float, int]] = None, show_mask: Optional[bool] = None, show_mask_borders: Optional[bool] = None, colour: Optional[tuple] = None, thickness: Optional[int] = None)numpy.ndarray

Visualises the flow as arrowed lines, optionally showing the outline of the flow mask mask as a black line, and the invalid areas greyed out.

Parameters
  • grid_dist – Integer of the distance of the flow points to be used for the visualisation, defaults to 20

  • img – Numpy array with the background image to use (in BGR mode), defaults to white

  • scaling – Float or int of the flow line scaling, defaults to scaling the 99th percentile of arrowed line lengths to be equal to twice the grid distance (empirical value)

  • show_mask – Boolean determining whether the flow mask is visualised, defaults to False

  • show_mask_borders – Boolean determining whether the flow mask border is visualised, defaults to False

  • colour – Tuple of the flow arrow colour, defaults to hue based on flow direction as in visualise()

  • thickness – Integer of the flow arrow thickness, larger than zero. Defaults to 1

Returns

Numpy array of shape \((H, W, 3)\) containing the flow visualisation, in bgr colour space

Flow.show(wait: Optional[int] = None, show_mask: Optional[bool] = None, show_mask_borders: Optional[bool] = None)

Shows the flow in an OpenCV window using visualise()

Parameters
  • wait – Integer determining how long to show the flow for, in milliseconds. Defaults to 0, which means it will be shown until the window is closed, or the process is terminated

  • show_mask – Boolean determining whether the flow mask is visualised, defaults to False

  • show_mask_borders – Boolean determining whether flow mask border is visualised, defaults to False

Flow.show_arrows(wait: Optional[int] = None, grid_dist: Optional[int] = None, img: Optional[numpy.ndarray] = None, scaling: Optional[Union[float, int]] = None, show_mask: Optional[bool] = None, show_mask_borders: Optional[bool] = None, colour: Optional[tuple] = None)

Shows the flow in an OpenCV window using visualise_arrows()

Parameters
  • wait – Integer determining how long to show the flow for, in milliseconds. Defaults to 0, which means it will be shown until the window is closed, or the process is terminated

  • grid_dist – Integer of the distance of the flow points to be used for the visualisation, defaults to 20

  • img – Numpy array with the background image to use (in BGR colour space), defaults to black

  • scaling – Float or int of the flow line scaling, defaults to scaling the 99th percentile of arrowed line lengths to be equal to twice the grid distance (empirical value)

  • show_mask – Boolean determining whether the flow mask is visualised, defaults to False

  • show_mask_borders – Boolean determining whether the flow mask border is visualised, defaults to False

  • colour – Tuple of the flow arrow colour, defaults to hue based on flow direction as in visualise()

oflibnumpy.visualise_definition(mode: str, shape: Optional[Union[list, tuple]] = None, insert_text: Optional[bool] = None)numpy.ndarray

Return an image that shows the definition of the flow visualisation.

Parameters
  • mode – Desired output colour space: rgb, bgr, or hsv

  • shape – List or tuple of shape \((2)\) containing the desired image shape as values (H, W). Defaults to (601, 601) - do not change if you leave insert_text as True as otherwise the text will appear in the wrong location

  • insert_text – Boolean determining whether explanatory text is put on the image (using cv2.putText()), defaults to True

Returns

Numpy array of shape \((H, W, 3)\) and type uint8 showing the colour definition of the flow visualisation

Using NumPy Arrays

This section contains functions that take NumPy arrays as inputs, instead of making use of the custom flow class. On the one hand, this avoids having to define flow objects. On the other hand, it requires keeping track of flow attributes manually, and it does not avail itself of the full scope of functionality oflibnumpy can offer: most importantly, flow masks are not considered or tracked.

Flow Loading

oflibnumpy.from_matrix(matrix: numpy.ndarray, shape: Union[list, tuple], ref: str)numpy.ndarray

Flow field array calculated from a transformation matrix

Parameters
  • matrix – Transformation matrix to be turned into a flow field, as numpy array of shape \((3, 3)\)

  • shape – List or tuple of the shape \((H, W)\) of the flow field

  • ref – Flow reference, string of value t (“target”) or s (“source”)

Returns

Flow field, as numpy array

oflibnumpy.from_transforms(transform_list: list, shape: Union[list, tuple], ref: str)numpy.ndarray

Flow field array calculated from a list of transforms

Parameters
  • transform_list

    List of transforms to be turned into a flow field, where each transform is expressed as a list of [transform name, transform value 1, … , transform value n]. Supported options:

    • Transform translation, with values horizontal shift in px, vertical shift in px

    • Transform rotation, with values horizontal centre in px, vertical centre in px, angle in degrees, counter-clockwise

    • Transform scaling, with values horizontal centre in px, vertical centre in px, scaling fraction

  • shape – List or tuple of the shape \((H, W)\) of the flow field

  • ref – Flow reference, string of value t (“target”) or s (“source”)

Returns

Flow field as a numpy array

oflibnumpy.load_kitti(path: str)Union[List[numpy.ndarray], numpy.ndarray]

Loads the flow field contained in KITTI uint16 png images files, including the valid pixels. Follows the official instructions on how to read the provided .png files

Parameters

path – String containing the path to the KITTI flow data (uint16, .png file)

Returns

A numpy array with the KITTI flow data (including valid pixels)

oflibnumpy.load_sintel(path: str)numpy.ndarray

Loads the flow field contained in Sintel .flo byte files. Follows the official instructions provided with the Sintel .flo data.

Parameters

path – String containing the path to the Sintel flow data (.flo byte file, little Endian)

Returns

A numpy array containing the Sintel flow data

oflibnumpy.load_sintel_mask(path: str)numpy.ndarray

Loads the invalid pixels contained in Sintel .png mask files. Follows the official instructions provided with the .flo data.

Parameters

path – String containing the path to the Sintel invalid pixel data (.png, black and white)

Returns

A numpy array containing the Sintel invalid pixels (mask) data

Flow Manipulation

oflibnumpy.resize_flow(flow: numpy.ndarray, scale: Union[float, int, list, tuple])numpy.ndarray

Resize a flow field array, scaling the flow vectors values accordingly

Parameters
  • flow – Numpy array containing the flow vectors to be resized, shape \((H, W, 2)\)

  • scale

    Scale used for resizing, options:

    • Integer or float of value scaling applied both vertically and horizontally

    • List or tuple of shape \((2)\) with values [vertical scaling, horizontal scaling]

Returns

Scaled flow field as a numpy array

oflibnumpy.invert_flow(flow: numpy.ndarray, input_ref: str, output_ref: Optional[str] = None)numpy.ndarray

Inverting a flow: img1f –> img2 becomes img1 <– fimg2. The smaller the input flow, the closer the inverse is to simply multiplying the flow by -1.

Parameters
  • flow – The flow field as a numpy array of shape \((H, W, 2)\)

  • input_ref – Reference of the input flow field, either s or t

  • output_ref – Desired reference of the output field, either s or t. Defaults to input_ref

Returns

Flow field as a numpy array of shape \((H, W, 2)\)

oflibnumpy.switch_flow_ref(flow: numpy.ndarray, input_ref: str)numpy.ndarray

Recalculate flow vectors to correspond to a switched flow reference (see Flow reference ref)

Parameters
  • flow – The flow field as a numpy array of shape \((H, W, 2)\)

  • input_ref – The reference of the input flow field, either s or t

Returns

Flow field as a numpy array of shape \((H, W, 2)\)

oflibnumpy.combine_flows(input_1: Union[Flow, numpy.ndarray], input_2: Union[Flow, numpy.ndarray], mode: int, ref: Optional[str] = None, thresholded: Optional[bool] = None)Union[Flow, numpy.ndarray]

Returns the result of the combination of two flow fields of the same shape and reference

Tip

All of the flow field combinations in this function rely on some combination of the apply(), invert(), and combine_with() methods, and can be very slow (several seconds) due to calling scipy.interpolate.griddata() multiple times. The table below aids decision-making with regards to which reference a flow field should be provided in to obtain the fastest result.

Calls to scipy.interpolate.griddata()

mode

ref = 's'

ref = 't'

1

1

3

2

1

1

3

0

0

All formulas used in this function have been derived from first principles. The base formula is \(flow_1 ⊕ flow_2 = flow_3\), where \(⊕\) is a non-commutative flow composition operation. This can be visualised with the start / end points of the flows as follows:

S = Start point    S1 = S3 ─────── f3 ────────────┐
E = End point       │                             │
f = flow           f1                             v
                    └───> E1 = S2 ── f2 ──> E2 = E3

The main difficulty in combining flow fields is that it would be incorrect to simply add up or subtract flow vectors at one location in the flow field area \(H \times W\). This appears to work given e.g. a translation to the right, and a translation downwards: the result will be the linear combination of the two vectors, or a translation towards the bottom right. However, looking more closely, it becomes evident that this approach isn’t actually correct: A pixel that has been moved from S1 to E1 by the first flow field f1 is then moved from that location by the flow vector of the flow field f2 that corresponds to the new pixel location E1, not the original location S1. If the flow vectors are the same everywhere in the field, the difference will not be noticeable. However, if the flow vectors of f2 vary throughout the field, such as with a rotation around some point, it will!

In this case (corresponding to calling combine_flows(f1, f2, mode=3)), and if the flow reference is s (“source”), the solution is to first apply the inverse of f1 to f2, essentially linking up each location E1 back to S1, and then to add up the flow vectors. Analogous observations apply for the other permutations of flow combinations and references.

Note

This is consistent with the observation that two translations are commutative in their application - the order does not matter, and the vectors can simply be added up at every pixel location -, while a translation followed by a rotation is not the same as a rotation followed by a translation: adding up vectors at each pixel cannot be the correct solution as there wouldn’t be a difference based on the order of vector addition.

Parameters
  • input_1 – First input flow as a numpy array. Can also take a flow object, but this will be deprecated soon

  • input_2 – Second input flow, same type as input_1

  • mode

    Integer determining how the input flows are combined, where the number corresponds to the position in the formula \(flow_1 ⊕ flow_2 = flow_3\):

    • Mode 1: input_1 corresponds to \(flow_2\), input_2 corresponds to \(flow_3\), the result will be \(flow_1\)

    • Mode 2: input_1 corresponds to \(flow_1\), input_2 corresponds to \(flow_3\), the result will be \(flow_2\)

    • Mode 3: input_1 corresponds to \(flow_1\), input_2 corresponds to \(flow_2\), the result will be \(flow_3\)

  • ref – The reference of the input flow fields, either s or t

  • thresholded – Boolean determining whether flows are thresholded during an internal call to is_zero(), defaults to False

Returns

Flow object if inputs are flow objects (deprecated in future, avoid), Numpy array as standard

Flow Application

oflibnumpy.apply_flow(flow: numpy.ndarray, target: numpy.ndarray, ref: str, mask: Optional[numpy.ndarray] = None)numpy.ndarray

Uses a given flow to warp a target. The flow reference, if not given, is assumed to be t. Optionally, a mask can be passed which (only for flows in s reference) masks undesired (e.g. undefined or invalid) flow vectors.

Parameters
  • flow – Numpy array containing the flow vectors in cv2 convention (1st channel hor, 2nd channel ver), with shape \((H, W, 2)\)

  • target – Numpy array containing the content to be warped, with shape \((H, W)\) or \((H, W, C)\)

  • ref – Reference of the flow, t or s

  • mask – Boolean numpy array containing the flow mask, with shape \((H, W)\). Only relevant for s flows. Defaults to True everywhere

Returns

Numpy array of the same shape \((H, W)\) as the target, with the content warped by the flow

oflibnumpy.track_pts(flow: numpy.ndarray, ref: str, pts: numpy.ndarray, int_out: Optional[bool] = None, s_exact_mode: Optional[bool] = None)numpy.ndarray

Warp input points with the flow field, returning the warped point coordinates as integers if required

Tip

Calling track_pts() on a flow field with reference s (“source”) is significantly faster (as long as s_exact_mode is not set to True), as this does not require a call to scipy.interpolate.griddata().

Parameters
  • flow – Flow field as a numpy array of shape \((H, W, 2)\)

  • ref – Flow field reference, either s or t

  • pts – Numpy array of shape \((N, 2)\) containing the point coordinates. pts[:, 0] corresponds to the vertical coordinate, pts[:, 1] to the horizontal coordinate

  • int_out – Boolean determining whether output points are returned as rounded integers, defaults to False

  • s_exact_mode – Boolean determining whether the necessary flow interpolation will be done using scipy.interpolate.griddata(), if the flow has the reference ref value of s (“source”). Defaults to False, which means a less exact, but around 2 orders of magnitude faster bilinear interpolation method will be used. This is recommended for normal point tracking applications.

Returns

Numpy array of warped (‘tracked’) points, and optionally a numpy array of the point tracking status

Flow Evaluation

oflibnumpy.is_zero_flow(flow: numpy.ndarray, thresholded: Optional[bool] = None)bool

Check whether all flow vectors are zero. Optionally, a threshold flow magnitude value of 1e-3 is used. This can be useful to filter out motions that are equal to very small fractions of a pixel, which might just be a computational artefact to begin with.

Parameters
  • flow – Flow field as a numpy array of shape \((H, W, 2)\)

  • thresholded – Boolean determining whether the flow is thresholded, defaults to True

Returns

True if the flow field is zero everywhere, otherwise False

oflibnumpy.get_flow_matrix(flow: numpy.ndarray, ref: str, dof: Optional[int] = None, method: Optional[str] = None)numpy.ndarray

Fit a transformation matrix to the flow field using OpenCV functions

Parameters
  • flow – Flow field as a numpy array of shape \((H, W, 2)\)

  • ref – Reference of the flow field, s or t

  • dof

    Integer describing the degrees of freedom in the transformation matrix to be fitted, defaults to 8. Options are:

    • 4: Partial affine transform with rotation, translation, scaling

    • 6: Affine transform with rotation, translation, scaling, shearing

    • 8: Projective transform, i.e estimation of a homography

  • method

    String describing the method used to fit the transformations matrix by OpenCV, defaults to ransac. Options are:

    • lms: Least mean squares

    • ransac: RANSAC-based robust method

    • lmeds: Least-Median robust method

Returns

Numpy array of shape \((3, 3)\) containing the transformation matrix

oflibnumpy.valid_target(flow: numpy.ndarray, ref: str)numpy.ndarray

Find the valid area in the target domain

Given a source image and a flow, both of shape \((H, W)\), the target image is created by warping the source with the flow. The valid area is then a boolean numpy array of shape \((H, W)\) that is True wherever the value in the target img stems from warping a value from the source, and False where no valid information is known.

Pixels that are False will often be black (or ‘empty’) in the warped target image - but not necessarily, due to warping artefacts etc. The valid area also allows a distinction between pixels that are black due to no actual information being available at this position (validity False), and pixels that are black due to black pixel values having been warped to that (valid) location by the flow.

Parameters
  • flow – Flow field as a numpy array of shape \((H, W, 2)\)

  • ref – Reference of the flow field, s or t

Returns

Boolean numpy array of the same shape \((H, W)\) as the flow

oflibnumpy.valid_source(flow: numpy.ndarray, ref: str)numpy.ndarray

Finds the area in the source domain that will end up being valid in the target domain (see valid_target()) after warping

Given a source image and a flow, both of shape \((H, W)\), the target image is created by warping the source with the flow. The source area is then a boolean numpy array of shape \((H, W)\) that is True wherever the value in the source will end up somewhere inside the valid target area, and False where the value in the source will either be warped outside of the target image, or not be warped at all due to a lack of valid flow vectors connecting to this position.

Parameters
  • flow – Flow field as a numpy array of shape \((H, W, 2)\)

  • ref – Reference of the flow field, s or t

Returns

Boolean numpy array of the same shape \((H, W)\) as the flow

oflibnumpy.get_flow_padding(flow: numpy.ndarray, ref: str)numpy.ndarray

Determine necessary padding from the flow field:

  • When the flow reference is t (“target”), this corresponds to the padding needed in a source image which ensures that every flow vector will find a value in the source domain to warp towards the target domain. I.e. any invalid locations in the area \(H \times W\) of the target domain (see valid_target()) are purely due to no valid flow vector being available to pull a source value to this target location, rather than no source value being available in the first place.

  • When the flow reference is s (“source”), this corresponds to the padding needed for the flow itself, so that applying it to a source image will result in no input image information being lost in the warped output, i.e each input image pixel will come to lie inside the padded area.

Parameters
  • flow – Flow field as a numpy array of shape \((H, W, 2)\)

  • ref – Reference of the flow field, s or t

Returns

A list of shape \((4)\) with the values [top, bottom, left, right]

Flow Visualisation

oflibnumpy.visualise_flow(flow: numpy.ndarray, mode: str, range_max: Optional[float] = None)numpy.ndarray

Visualises the flow as an rgb / bgr / hsv image

Parameters
  • flow – Flow field as a numpy array of shape \((H, W, 2)\)

  • mode – Output mode, options: rgb, bgr, hsv

  • range_max – Maximum vector magnitude expected, corresponding to the HSV maximum Value of 255 when scaling the flow magnitudes. Defaults to the 99th percentile of the flow field magnitudes

Returns

Numpy array of shape \((H, W, 3)\) containing the flow visualisation

oflibnumpy.visualise_flow_arrows(flow: numpy.ndarray, ref: str, grid_dist: Optional[int] = None, img: Optional[numpy.ndarray] = None, scaling: Optional[Union[float, int]] = None, colour: Optional[tuple] = None, thickness: Optional[int] = None)numpy.ndarray

Visualises the flow as arrowed lines

Parameters
  • flow – Flow field as a numpy array of shape \((H, W, 2)\)

  • ref – Reference of the flow field, s or t

  • grid_dist – Integer of the distance of the flow points to be used for the visualisation, defaults to 20

  • img – Numpy array with the background image to use (in BGR mode), defaults to white

  • scaling – Float or int of the flow line scaling, defaults to scaling the 99th percentile of arrowed line lengths to be equal to twice the grid distance (empirical value)

  • colour – Tuple of the flow arrow colour, defaults to hue based on flow direction as in visualise()

  • thickness – Integer of the flow arrow thickness, larger than zero. Defaults to 1

Returns

Numpy array of shape \((H, W, 3)\) containing the flow visualisation, in bgr colour space

oflibnumpy.show_flow(flow: numpy.ndarray, wait: Optional[int] = None)

Shows the flow in an OpenCV window using visualise()

Parameters
  • flow – Flow field as a numpy array of shape \((H, W, 2)\)

  • wait – Integer determining how long to show the flow for, in milliseconds. Defaults to 0, which means it will be shown until the window is closed, or the process is terminated

oflibnumpy.show_flow_arrows(flow: numpy.ndarray, ref: str, wait: Optional[int] = None, grid_dist: Optional[int] = None, img: Optional[numpy.ndarray] = None, scaling: Optional[Union[float, int]] = None, colour: Optional[tuple] = None)

Shows the flow in an OpenCV window using visualise_arrows()

Parameters
  • flow – Flow field as a numpy array of shape \((H, W, 2)\)

  • ref – Reference of the flow field, s or t

  • wait – Integer determining how long to show the flow for, in milliseconds. Defaults to 0, which means it will be shown until the window is closed, or the process is terminated

  • grid_dist – Integer of the distance of the flow points to be used for the visualisation, defaults to 20

  • img – Numpy array with the background image to use (in BGR colour space), defaults to black

  • scaling – Float or int of the flow line scaling, defaults to scaling the 99th percentile of arrowed line lengths to be equal to twice the grid distance (empirical value)

  • colour – Tuple of the flow arrow colour, defaults to hue based on flow direction as in visualise()