UP42 Recipes: Stereo data ordering 101

This article is the second in a series of articles which explores the UP42 data platform using simple recipes.

In the first article, we went through the basics of ordering data and getting notified via webhooks when the order was delivered. In this article, we will explore how to order stereo imagery and how to get notified of that delivery via webhooks.

The recipe we follow is:

Searching for stereo and tri-stereo data.
Placing an order to acquire the data selected in step 1.
Getting notified via a webhook when the data ordered in step 2 is delivered.
Repeat from step 1 to order more stereo/tri-stereo data.

Technical skill level required: beginner to medium.

Requirements (Ingredients)

A Pipedream account: there are several ways to create an account. One of the easiest options is to authorize the application in your Github account. Your github identity would then be used throughout Pipedream.
An ngrok account: similar to Pipedream, there are several waysto create an account. Connecting your ngrok account to your Github account is one of the easiest ways to get started.
Minimal knowledge of Linux containers in general and Docker in particular.
Minimal knowledge of nginx.

Stereo and tri-stereo in a nutshell

Stereo basic concept

Stereo in the context of satellite imaging refers to the technique of capturing two (or three for tri-stereo) image captures that are shifted by a certain viewing angle among them over the area of interest in the same orbital pass. The value of this shift determines the level of detail that can be reconstructed from combining the two images. Usually these images are used to create 3D models, for example, Digital Elevation Models (DEMs). Since the two images when properly combined provide depth information, the relief of the scene can be determined up to a certain degree of certainty.

Tri-stereo basic concept

The tri-stereo mode of acquisition adds a third additional image that is captured midway between the two extreme positions of a stereo capture. This corresponds to the satellite being (nearly) vertically aligned with the scene being imaged. The (nearly) vertical acquisition makes visible key relief details that would be hidden when the photo is taken obliquely, which is the case for stereo. For this reason, tri-stereo acquisitions are generally recommended for mountainous or densely packed urban areas.

Satellite orbits and stereo capture

In this notebook, and in fact for most modern earth observation satellites, these two (or three) modes of image captures are done either ascending or descending in a single orbit. This is called along-track stereo/tri-stereo.

Stereo and tri-stereo: Further explanation [optional]

Sun-synchronous orbit basics

Earth observation satellites usually have a Sun-synchronous orbit (SSO), also called a heliosynchronous orbit,[1], a nearly polar orbit around the planet in which the satellite passes over any given point of the planet's surface at the same local mean solar time.

The orbital period T is given by:

T = 2\pi \sqrt{\frac{a^3}{\mu}}

where:

a is the semi-major axis of the orbit (altitude).
μ is the standard gravitational parameter of the planet. For Earth, the value is approximately 3.986e14 cubic meters per squared seconds.

This translates into an orbital period within the 90 minute range. I.e., it takes roughly 90 minutes for the satellite to complete a sun-synchronous orbit around the Earth. The exact value depends on the altitude, for Earth observation the satellites value is usually in the 600-1000 km range.

Attitude adjustment

When capturing stereo/tri-stereo imagery, the orientation of the satellite in 3D space needs to be adjusted so that the appropriate incidence angle is obtained for each image capture. This adjustment can be achieved through attitude control.

However, since satellites are built to be in orbit for a long time, and therefore cannot rely on the use of thrusters for attitude control, this is usually done through reaction wheels.

The mechanical principle behind reaction wheels is the conservation of angular momentum.

Along-track stereo and tri-stereo

As mentioned above we deal exclusively with along-track stereo acquisitions. This means that the acquisition times of the stereo/tri-stereo images are within a 90-100 seconds range. Across-track stereo/tri-stereo image acquisitions would differ by more than an orbital period.

Below is a figure depicting how stereo/tri-stereo image captures are made. It shows clearly the advantages of tri-stereo over stereo in areas where the relief varies considerably.

Stereo and tri-stereo image captures (in Pléaides NEO User Manual - Airbus Defense and Space, 2021)

The defining characteristic for the level of detail in a stereo/tri-stereo capture is the trigonometric tangent of the angle of the triangle formed by the distance (B) between the extreme points of the stereo/tri-stereo capture and the altitude (H). Referred to as B/H, as illustrated in the figure below.

Geometric characteristics of stereo/tri-stereo image acquisition (in Pléaides NEO User Manual - Airbus Defense and Space, 2021)

The exact values of B/H are specific to each satellite and they vary with the length of the scene being acquired.

Pléaides 1A/1B and SPOT 6/7 B/H ranges

stereo B/H range	tri-stereo B/H range
0.15 - 0.6	0.3 - 0.8m

Pléiades NEO B/H ranges

stereo B/H range	tri-stereo B/H range
0.15 - 0.5	0.3 - 0.7

The smaller the B/H, the more details are able to be captured in a satellite stereo/tri-stereo image capture.

Heuristic for finding stereo/tri-stereo images

Based on the above discussion, we can establish the following heuristic for finding stereo image pairs and tri-stereo image triples. Given an AOI, the criteria are the following:

Look for images that have the acquisition time within a 90 second range.
The incidence angle along track is such that the B/H ratio is within the range for a given satellite.

From a programming point of view we build a predicate that translates these two criteria and we filter of the search results, for the search parameters we have specified.

Installation and webhook set up

We refer to the installation section of the previous article to set up:

nginx
ngrok
Pipedream

and for further details on the webhook set up. We are going to (re)use the webhook we have set up in that article.

You will need to be in the data-ordering101 directory to launch nginx. The only differences in terms of code between the previous article and this are on the Jupyter notebook. Everything else stays the same. We are using the Makefile in the data-ordering101 directory.

Algorithm for finding stereo pairs/tri-stereo triples

The algoritm for finding stereo pairs/tri-stereo triples is based on the heuristic rule set:

Acquisition time differs by 90 seconds at most.
Incidence angle along track is such that the B/H values are within the range for stereo/tri-stereo.

The geometry used is simplified and we approximate the arc length of the capture to have zero curvature. With this simplification we can use the incidence angles to compute B/H. Bear in mind that B/H is used as a validating rule after we select the acquisition dates. Also considering the radius of the Earth (ellipsoid semi-axis length) the approximation is good enough.

The algorith consists in a conjunction of two predicates. One for the acquistion dates and one for the angles.

Predicates

Let us look at the code for the predicates. It is quite easy to follow since is a direct implementation of the first rule above, i.e., checking if the acquisition dates are within a 90 second range. There is a small preprocessing of the dates, since they are given in milliseconds and Python date strings only accept microseconds. To deal with that we just multiply the milliseconds value by 1000.

After that we convert the dates to a UNIX timestamp and we compute the difference between the two numbers.

For the angles we compute an estimate of B/H.

def is_stereo_dates(*acquisition_dates: str) -> bool:
    """Checks to see if the acquisition dates are within the range allowed for a stereo/tri-stereo capture.

    Parameters
    ----------
    *acquisition_dates : str
        pair/triple of acquisition dates

    Returns
    -------
    bool
        True if the dates are within a stereo/tri-stereo range, False
        if not.

    Examples
    --------
    is_stereo_dates(("2021-10-13T10:59:01.624Z", "2021-10-13T10:58:42.874Z"))
    is_stereo_dates(("2021-10-07T11:21:34.555Z", "2021-10-07T11:21:20.305Z", "2021-10-07T11:21:06.180Z"))

    """

    # Convert the milliseconds to microseconds. Python doesn't have a
    # milliseconds date format descriptor. See:
    # https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes. Also
    # create a UNIX timestamp from the given acquisition dates.
    return reduce(minus_op, map(lambda d: dt.timestamp(
        re.sub(r"(?P<ms>\.\d{3})Z$", r"\g<ms>000Z", d),
        "%Y-%m-%dT%H:%M:%S.%fZ"), acquisition_dates)) < MAX_ACQ_TIME_DELTA

def is_stereo_angles(*incidence_angles: float, sensor: str, tri-stereo: bool=False) -> bool:
    """Checks is the incidence angles along the track is within the B/H bounds defined for a given satellite when doing stereo/tri-stereo captures.

    Parameters
    ----------
    *incidence_angles : float
        pair/triple of incidence angles along the track
    sensor : str
        the name of the sensor (satellite)
    tri-stereo : bool
        True if we are looking for tri-stereo, False if looking for
        stereo.

    Returns
    -------
    bool
        True if the angles are such that we are within the B/H range
        for stereo/tri-stereo. False if not.

    Examples
    --------
    is_stereo_angles((12.458, -1.567))
    is_stereo_angles((12.458, -1.567, 4.674))

    """

    # Compute the sin of each incidence angle along track. Assuming
    # the curvature of the Earth is negligible for the maximum length
    # of stereo/tri-stereo captures. Up to 100 km. The curvature of the
    # Earth will make it so that the "drop" between, the beginning and
    # the end of the capture is roughly 800 m for a 100 km length.
    b_over_h = reduce(lambda x, y: sin(radians(x)) + sin(radians(y)),
                      incidence_angles)
    # Get the acquisition mode.
    mode = "tri-stereo" if tri-stereo else "stereo"
    # Check if B/H is within the respective sensor bounds.
    return B_H_RANGE_SENSORS[sensor][mode]["lower"] <= b_over_h <= B_H_RANGE_SENSORS[sensor][mode]["upper"]

Search results metadata

The search results return metadata that includes the two values we are interested in:

Acquisition time
Incidence angle along track

What the algorithm needs to do is to filter the search results according to the predicates. To do that we make use of the extensive support Python provides for functional programming.

Filtering the search results

Both for stereo and tri-stereo, we create a set of parallel iterators, where one or two will be shifted relative the others. For stereo we advance the second iterator by one element, and for tri-stereo we additionally advance the third iterator by two elements. This allows us to build lists of pairs and triples respectively.

def is_stereo_dates(*acquisition_dates: str) -> bool:
    """Checks to see if the acquisition dates are within the range allowed for a stereo/tri-stereo capture.

    Parameters
    ----------
    *acquisition_dates : str
        pair/triple of acquisition dates

    Returns
    -------
    bool
        True if the dates are within a stereo/tri-stereo range, False
        if not.

    Examples
    --------
    is_stereo_dates(("2021-10-13T10:59:01.624Z", "2021-10-13T10:58:42.874Z"))
    is_stereo_dates(("2021-10-07T11:21:34.555Z", "2021-10-07T11:21:20.305Z", "2021-10-07T11:21:06.180Z"))

    """

    # Convert the milliseconds to microseconds. Python doesn't have a
    # milliseconds date format descriptor. See:
    # https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes. Also
    # create a UNIX timestamp from the given acquisition dates.
    return reduce(minus_op, map(lambda d: dt.timestamp(
        re.sub(r"(?P<ms>\.\d{3})Z$", r"\g<ms>000Z", d),
        "%Y-%m-%dT%H:%M:%S.%fZ"), acquisition_dates)) < MAX_ACQ_TIME_DELTA

def is_stereo_angles(*incidence_angles: float, sensor: str, tri-stereo: bool=False) -> bool:
    """Checks if the incidence angles along the track are within the B/H bounds defined for a given satellite when doing stereo/tri-stereo captures.

    Parameters
    ----------
    *incidence_angles : float
        pair/triple of incidence angles along the track
    sensor : str
        the name of the sensor (satellite)
    tri-stereo : bool
        True if we are looking for tri-stereo, False if looking for
        stereo.

    Returns
    -------
    bool
        True if the angles are such that we are within the B/H range
        for stereo/tri-stereo. False if not.

    Examples
    --------
    is_stereo_angles((12.458, -1.567))
    is_stereo_angles((12.458, -1.567, 4.674))

    """

    # Compute the sin of each incidence angle along track. Assuming
    # the curvature of the Earth is negligible for the maximum length
    # of stereo/tri-stereo captures. Up to 100 km. The curvature of the
    # Earth will make it so that the "drop" between, the beginning and
    # the end of the capture is roughly 800 m for a 100 km length.
    b_over_h = reduce(lambda x, y: sin(radians(x)) + sin(radians(y)),
                      incidence_angles)
    # Get the acquisition mode.
    mode = "tri-stereo" if tri-stereo else "stereo"
    # Check if B/H is within the respective sensor bounds.
    return B_H_RANGE_SENSORS[sensor][mode]["lower"] <= b_over_h <= B_H_RANGE_SENSORS[sensor][mode]["upper"]

Of course we could have opted for not duplicating/triplicating the data and instead use, for example, using backtracking as we move through the given search results. But the algorithm and implementation is much more complex and we would lose the clarity that a purely functional approach usually provides.

Carrying on with the SDK and a Jupyter notebook

From here on, we continue this recipe exclusively on the Jupyter notebook.

António Almeida

Senior Tech Evangelist

Developer content

Product

UP42 recipes: Stereo data ordering 101

Requirements (Ingredients)