Theory of Operations
--------------------

Newt has a fairly smart package manager that can read a directory tree,
build a dependency tree, and emit the right build artifacts.

Building dependencies
~~~~~~~~~~~~~~~~~~~~~

Newt can read a directory tree, build a dependency tree, and emit the
right build artifacts. An example newt source tree is in
mynewt-blinky/develop:

.. code-block:: console

  $ tree -L 3
  .
  ├── LICENSE
  ├── NOTICE
  ├── README.md
  ├── apps
  │   └── blinky
  │       ├── pkg.yml
  │       └── src
  ├── project.yml
  └── targets
      ├── my_blinky_sim
      │   ├── pkg.yml
      │   └── target.yml
      └── unittest
          ├── pkg.yml
          └── target.yml

  6 directories, 9 files

When newt sees a directory tree that contains a "project.yml" file it knows that it is in the base directory of a project,
and automatically builds a package tree. You can see that there are two essential package directories, "apps" and "targets."

"apps" Package Directory
^^^^^^^^^^^^^^^^^^^^^^^^

``apps`` is where applications are stored, and applications are where the main() function is contained.  The base project
directory comes with one simple app called ``blinky`` in the ``apps`` directory. The core repository ``@apache-mynewt-core``
comes with many additional sample apps in its ``apps`` directory. At the time of this writing, there are several example BLE
apps, the boot app, slinky app for using newt manager protocol, and more in that directory.

.. code-block:: console

  $ ls repos/apache-mynewt-core/apps
  blecent  blehr          blemesh_shell  blesplit  boot     btshell     lora_app_shell  ocf_sample    slinky      splitty    trng_test
  blecsc   blemesh        bleprph        bletest   bsncent  ffs2native  loraping        pwm_test      slinky_oic  testbench
  blehci   blemesh_light  bleprph_oic    bleuart   bsnprph  iptest      lorashell       sensors_test  spitest     timtest

Along with the ``targets`` directory, ``apps`` represents the top-level of the build tree for the particular project, and
define the dependencies for the rest of the system. Mynewt users and developers can add their own apps to the project's ``apps`` directory.

The app definition is contained in a ``pkg.yml`` file. For example, blinky's ``pkg.yml`` file is:

.. code-block:: console

  $ more apps/blinky/pkg.yml
  pkg.name: apps/blinky
  pkg.type: app
  pkg.description: Basic example application which blinks an LED.
  pkg.author: "Apache Mynewt <dev@mynewt.apache.org>"
  pkg.homepage: "http://mynewt.apache.org/"
  pkg.keywords:

  pkg.deps:
      - "@apache-mynewt-core/kernel/os"
      - "@apache-mynewt-core/hw/hal"
      - "@apache-mynewt-core/sys/console/stub"

This file says that the name of the package is apps/blinky, and it depends on the ``kernel/os``, ``hw/hal`` and ``sys/console/stub`` packages.

**NOTE:** @apache-mynewt-core is a repository descriptor, and this will be
covered in the "repository" section.

"targets" Package Directory
^^^^^^^^^^^^^^^^^^^^^^^^^^^

``targets`` is where targets are stored, and each target is a collection of parameters that must be passed to newt in
order to generate a reproducible build. Along with the ``apps`` directory, ``targets`` represents the top of the build
tree. Any packages or parameters specified at the target level cascades down to all dependencies.

Most targets consist of:

* app: The application to build
* bsp: The board support package to combine with that application
* build_profile: Either debug or optimized.

The ``my_blinky_sim`` target that is included by default has the following settings:

.. code-block:: console

  $ newt target show
  targets/my_blinky_sim
      app=apps/blinky
      bsp=@apache-mynewt-core/hw/bsp/native
      build_profile=debug
  $ ls targets/my_blinky_sim/
  pkg.yml		target.yml

There are helper functions to aid the developer specify parameters for a target.

* **vals**: Displays all valid values for the specified parameter type (e.g. bsp for a target)
* **target show**: Displays the variable values for either a specific target or all targets defined for the project
* **target set**: Sets values for target variables

In general, the three basic parameters of a target (``app``, ``bsp``, and ``build_profile``) are stored in the
target's ``target.yml`` file in the targets/<target-name> directory, where ``target-name`` is the name of the target. You
will also see a ``pkg.yml`` file in the same directory. Since targets are packages, a ``pkg.yml`` is expected. It contains
typical package descriptors, dependencies, and additional parameters such as the following:

* Cflags: Any additional compiler flags you might want to specify to the build
* Aflags: Any additional assembler flags you might want to specify to the build
* Lflags: Any additional linker flags you might want to specify to the build

You can also override the values of the system configuration settings that are defined by the packages that your target
includes. You override the values in your target's ``syscfg.yml`` file (stored in the targets/<target-name> directory).
You can use the ``newt target config show`` command to see the configuration settings and values for your target, and use
the ``newt target set`` command to set the ``syscfg`` variable and override the configuration setting values.  You can
also use an editor to create your target's ``syscfg.yml`` file and add the setting values to the file.
See :doc:`../os/modules/sysinitconfig/sysinitconfig` for more information on system configuration settings.

Resolving dependencies
~~~~~~~~~~~~~~~~~~~~~~

When newt builds a project, it will:

* find the top-level project.yml file
* recurse the packages in the package tree, and build a list of all source packages

Newt then looks at the target that the user set, for example, blinky_sim:

.. code-block:: console

  $ more targets/my_blinky_sim/target.yml
  ### Target: targets/my_blinky_sim
  target.app: "apps/blinky"
  target.bsp: "@apache-mynewt-core/hw/bsp/native"
  target.build_profile: "debug"

The target specifies two major things:

* Application (target.app): The application to build
* Board Support Package (target.bsp): The board support package to build along with that application.

Newt builds the dependency tree specified by all the packages. While building this tree, it does a few other things:

* Sets up the include paths for each package. Any package that depends on another package, automatically gets the include
  directories from the package it includes.  Include directories in the
  newt structure must always be prefixed by the package name. For example, kernel/os has the following include tree and its
  include directory files contains the package name "os" before any header files.  This is so in order to avoid any header file conflicts.

.. code-block:: console

  $ tree repos/apache-mynewt-core/kernel/os/include/
  repos/apache-mynewt-core/kernel/os/include/
  └── os
      ├── arch
      │   ├── cortex_m0
      │   │   └── os
      │   │       └── os_arch.h
      │   ├── cortex_m4
      │   │   └── os
      │   │       └── os_arch.h
      │   ├── mips
      │   │   └── os
      │   │       └── os_arch.h
      │   ├── sim
      │   │   └── os
      │   │       └── os_arch.h
      │   └── sim-mips
      │       └── os
      │           └── os_arch.h
      ├── endian.h
      ├── os.h
      ├── os_callout.h
      ├── os_cfg.h
      ├── os_cputime.h
      ├── os_dev.h
      ├── os_eventq.h
      ├── os_fault.h
      ├── os_heap.h
      ├── os_malloc.h
      ├── os_mbuf.h
      ├── os_mempool.h
      ├── os_mutex.h
      ├── os_sanity.h
      ├── os_sched.h
      ├── os_sem.h
      ├── os_task.h
      ├── os_test.h
      ├── os_time.h
      └── queue.h

  12 directories, 25 files

* Validates API requirements.  Packages can export APIs they
  implement, (i.e. pkg.api: hw-hal-impl), and other packages can require
  those APIs (i.e. pkg.req_api: hw-hal-impl).

* Reads and validates the configuration setting definitions and values from the package ``syscfg.yml`` files.
  It generates a ``syscfg.h`` header file that packages include in the source files in order to access the settings.
  It also generates a system initialization function to initialize the packages.
  See :doc:`../os/modules/sysinitconfig/sysinitconfig` for more information.

In order to properly resolve all dependencies in the build system, newt recursively processes the package dependencies
until there are no new dependencies.  And it builds a big list of all the packages that need to be build.

Newt then goes through this package list, and builds every package into
an archive file.

**NOTE:** The newt tool generates compiler dependencies for all of these packages, and only rebuilds the packages whose
dependencies have changed. Changes in package & project dependencies are also taken into account. It is smart, after all!

Producing artifacts
~~~~~~~~~~~~~~~~~~~

Once newt has built all the archive files, it then links the archive files together.  The linkerscript to use is specified
by the board support package (BSP.)

The newt tool creates a bin directory under the base project directory, and places a target's build artifacts into the
bin/targets/<target-name>/app/apps/<app-name> directory, where ``target-name`` is the name of the target and ``app-name``
is the name of the application. As an example, the ``blinky.elf`` executable for the ``blinky`` application defined by
the ``my_blinky_sim`` target is stored in the bin/targets/my_blinky_sim/app/apps/blinky directory as shown in the following source tree:

.. code-block:: console

  $tree -L 9 bin/
  bin/
  └── targets
      ├── my_blinky_sim
      │   ├── app
      │   │   ├── apps
      │   │   │   └── blinky
      │   │   │       ├── apps
      │   │   │       │   └── blinky
      │   │   │       │       └── src
      │   │   │       │           ├── main.d
      │   │   │       │           ├── main.o
      │   │   │       │           └── main.o.cmd
      │   │   │       ├── apps_blinky.a
      │   │   │       ├── apps_blinky.a.cmd
      │   │   │       ├── blinky.elf
      │   │   │       ├── blinky.elf.cmd
      │   │   │       ├── blinky.elf.dSYM
      │   │   │       │   └── Contents
      │   │   │       │       ├── Info.plist
      │   │   │       │       └── Resources
      │   │   │       │           └── DWARF
      │   │   │       ├── blinky.elf.lst
      │   │   │       └── manifest.json
      │   │   ├── hw
      │   │   │   ├── bsp
      │   │   │   │   └── native
      │   │   │   │       ├── hw_bsp_native.a
      │   │   │   │       ├── hw_bsp_native.a.cmd
      │   │   │   │       └── repos
      │   │   │   │           └── apache-mynewt-core
      │   │   │   │               └── hw

  <snip>

As you can see, a number of files are generated:

- Archive File
- \*.cmd: The command use to generate the object or archive file
- \*.lst: The list file where symbols are located

Note: The \*.o object files that get put into the archive file are stored in the bin/targets/my_blinky_sim/app/apps/blinky/apps/blinky/src directory.

Download/Debug Support
~~~~~~~~~~~~~~~~~~~~~~

Once a target has been built, there are a number of helper functions
that work on the target.  These are:

* **load**     Download built target to board
* **debug**        Open debugger session to target
* **size**         Size of target components
* **create-image**  Add image header to target binary
* **run**  The equivalent of build, create-image, load, and debug on specified target
* **target** Create, delete, configure, and query a target

``load`` and ``debug`` handles driving GDB and the system debugger.  These
commands call out to scripts that are defined by the BSP.

.. code-block:: console

  $ more repos/apache-mynewt-core/hw/bsp/nrf52dk/nrf52dk_debug.sh
  <snip>
  . $CORE_PATH/hw/scripts/jlink.sh

  FILE_NAME=$BIN_BASENAME.elf

  if [ $# -gt 2 ]; then
      SPLIT_ELF_NAME=$3.elf
      # TODO -- this magic number 0x42000 is the location of the second image
      # slot. we should either get this from a flash map file or somehow learn
      # this from the image itself
      EXTRA_GDB_CMDS="add-symbol-file $SPLIT_ELF_NAME 0x8000 -readnow"
  fi

  JLINK_DEV="nRF52"

  jlink_debug

The idea is that every BSP will add support for the debugger environment
for that board. That way common tools can be used across various
development boards and kits.