doc/source/handling-files/overlaps.rst

.. _handling_files_overlaps:

Overlapping files
=================
In this chapter, we will discuss what happens when files from multiple
element artifacts conflict with eachother, and what strategies we can
use to resolve these situations.

.. note::

   This example is distributed with BuildStream
   in the `doc/examples/overlaps
   <https://github.com/apache/buildstream/tree/master/doc/examples/overlaps>`_
   subdirectory.


Overview
--------
This project builds on the previous chapter on :ref:`composition <handling_files_composition>`,
and as such we'll only go over what has changed from there, which is not much.


Project structure
-----------------
In this example we've just extended the ``libhello.bst`` and the ``hello.bst``
elements such that they both install an additional file: ``%{docdir}/hello.txt``.

We've updated the following Makefiles:


files/libhello/Makefile
~~~~~~~~~~~~~~~~~~~~~~~

.. literalinclude:: ../../examples/overlaps/files/libhello/Makefile
   :language: Makefile


files/hello/Makefile
~~~~~~~~~~~~~~~~~~~~

.. literalinclude:: ../../examples/overlaps/files/hello/Makefile
   :language: Makefile


As you can see, this now presents a conflict of sorts, where multiple
elements in the pipeline want to install the *same file*.


Using the project
-----------------
In this chapter, we're only going to present the warning and then
discuss how to mitigate this situation.

See what happens when we try to build the ``runtime-only.bst``
:mod:`compose <elements.compose>` element:

.. raw:: html
   :file: ../sessions/overlaps-build.html

Notice the warning message about the conflicting file, it is there to
inform the user about which files are *overlapping*, and also which
elements are being staged in which order.

Note also that BuildStream does not discover the overlap until the
moment that you build a reverse dependency which will require staging
of both artifacts.

.. tip::

   The ``overlaps`` warning discussed here can be configured to be
   a :ref:`fatal warning <configurable_warnings>`. This is useful
   in the case that you want to be strict about avoiding overlapping
   files in your project.


Mitigating overlapping files
----------------------------
Since we recently discussed :ref:`filtering of artifacts <handling_files_filtering>`,
we should note that it is of course possible to handle this case by
having ``hello.bst`` depend on a *filtered* version of ``libhello.bst`` with the
offending file excluded.

However, working with :mod:`filter <elements.filter>` elements just for the
sake of handling a conflicting artifact would be quite inconvenient, so we
have other means.


Whitelisting overlapping files
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
BuildStream allows explicitly ignoring such errors by adding the files
to an :ref:`overlap whitelist <public_overlap_whitelist>`, you could achieve
this in the given example by adding the following to the ``hello.bst`` element:

.. code:: yaml

  public:
    bst:
      overlap-whitelist:
      - |
        %{docdir}/hello.txt

.. note::

   Note that :func:`glob patterns <buildstream.utils.glob>` are also
   supported in the whitelist.


Artifact munging
~~~~~~~~~~~~~~~~
Another way around this situation is the *munge* the artifacts at install
time such that there is no conflict.

This is the easiest approach in the case that you might want to keep the
underlying ``%{docdir}/hello.txt`` from ``libhello.bst`` and discard the
same file from ``hello.bst``.

In this case, we might modify the ``hello.bst`` file so that it's install
command contain an ``rm`` statement, as such:

.. code:: yaml

  install-commands:
  - make -j1 PREFIX="%{prefix}" DESTDIR="%{install-root}" install

  - |
    # Rid ourselves of the unwanted file at install time
    rm -f %{install-root}%{docdir}/hello.txt

This would cause later builds of ``runtime-only.bst`` to no longer
conflict on the given file.


Summary
-------
In this chapter we've presented a situation where an artifact
can *conflict* with another artifact by way of providing the
same files.

We've presented the :ref:`overlap whitelist <public_overlap_whitelist>`
public data which is the typical solution for silencing the
error when the outcome is desired, and also presented a strategy
to deal with cases where you want to keep files from the
*overlapped* artifact instead.