Skip to main content

Pipeline

Understanding the image pipeline​

The processing pipeline as at the heart of IPP. It provides a flexible way to configure the image treatment process to your particular needs, whether you only wish to compress images or make a full set of breakpoint-resized images.

To understand the pipeline, think of it as a tree-like structure of interconnetced pipes. Each pipeline as an entry point, through which the source image passes through. As it goes through each pipe, the image can be transformed in some way, such as being resized or converted into a different codec. The next pipe will then recieve a copy of the new image.

info

Each pipe can only have one input, but multiple outputs.

At different points you can tap into the pipeline to "leak out" a copy of the image buffer and save it to a file. This lets you save the result of a pipe and is generally what you would want to do at the very end of the pipeline (otherwise the processing happens for nothing!).

Image metadata​

The actual object that passes through the pipeline is a DataObject. It is an binary image buffer accompanied with a metadata object (a key-value map) that contains information about the image, such as the dimensions and format.

This makes it easy for each pipe to inspect the metadata, and modify it as needed along with the image buffer. For example, if the pipe was to convert the image to a different format, it should also update the format fieldsin the metadata to reflect this change.

Don't worry, this is handled by developer and not the end user. The metadat object is still useful for you, however. For example, when saving an image (or generating the manifest), you may use metadata values to dynamically generate the file name (or extract useful information about the format).

We'll touch more on this later.

An example pipeline​

The pipeline is defined in the configuration file under the pipeline property. Here's an example of how you might define one:

.ipprc.yml
pipeline:
  - pipe: resize
    options:
      resizeOptions:
        height: 1280
    save: "[source.name]-[width][ext]"
    then:
      - pipe: convert
        options:
          format: webp
        save: "[source.name]-[width][ext]"
tip

Both YAML and JSON formats are supported, we recommend using YAML, as it has a more concise and easier to read syntax.

If we provide a sufficiently large image file called parrot.jpg to the above pipeline, this process should yield two outputs, parrot-1280.jpg and parrot-1280.webp. Notice that we used template literals (using square brackets) to dynamically generate the filename using the metadata.

info

It can be hard to see what metadata is available, there is no solution for this yet. Check the reference for each pipe to see what metadata it provides. By default, basic properties such as width, height, format, hash, ext and name are always available.

Every pipeline chunk is an array of pipes. The pipeline property of the configuration file defines the entry point, the first pipes to receive the source image. Consecutive pipes can then be chained after each pipe using the then property to cascade transformations.

Here, each source image will flow into the resize pipe, where they get reduced to 1280 pixels in width, before being saved to a file called [originalName][ext]. As well as the saved file, the image object also then is passed to a convert pipe that creates a WebP variant, and saves that using the same name. The use of template literals ensures that a unique name is generated for each variant, as WebP uses a different file extension (.webp).

Metadata expansions​

To be able to dynamically create file names, IPP supports a template literals (you may also know this concept as template expansions). The square bracket notation tells IPP to replace that item with its corresponding metadata value when the file is being saved. The text inside of the brackets is used as the key for the metadata object.

Accessing metadata is done in three parts, using a selector, a key and a limit.

The selector is used to select the context, like a folder. For example, when the image is first loaded, most basic metadata (such as the name, size) is saved under the source selector, so that it is not overwritten during the pipeline (as the size may change, for example).

The key is used to select the actual metadata item. name usually refers to the filename including the extension.

The limit it used to limit the length of the replaced value, this is useful for limiting hashes which can be over 256 charcaters long!

Let's see a complete example.

pipeline:
  ...
  save: "[source.name]-[source.hash:8][ext]""

This could produce a filename such as "parrot-4e9004e6.jpeg", where the original file was called "parrot.jpeg" and the MD5 hash of the original file was "4e9004e6fe9d3ffd7933e5e4f9dd534f".

If you wish to learn more about the metadata object, see the architecture page.

Connecting more pipes​

As touched on in the example, pipes may be chained using the then property. IPP will correctly orchestrate the process to make sure that each pipe is able to carry out its task correctly.

To find out what other pipes are available, visit the pipe documentation.