From 7097e1785df60c43007bfeeea4e85d2bcb041201 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?Mart=C3=AD=20Bol=C3=ADvar?= Date: Wed, 6 Jan 2021 00:30:12 -0800 Subject: [PATCH] doc: improve west's repo-tool.rst MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Various improvements to the west repo-tool.rst page and pages that link to it: Rename the page title to "Basics", since it documents, well the basics, including built-in commands like "help" an "config" that are not directly related to multiple repositories. The "multi-repo" term was invented before we started using "workspace". Drop it from the text and rework things using words like "workspace", "basics", or "built-ins" instead, which read better. We've been using west for a long enough time that justifying its existence prominently at the start of this page is no longer necessary; move that to the "dustbin of history" page (why.rst). Try harder to clarify exactly what a "project" is, along with other workspace related clarifications. Improve the documentation for the west init and west update commands, and create :ref: targets for linking directly to them for convenience elsewhere in the docs. Slim down the usages for other built-in commands, so we don't have to keep those up to date as carefully each release. This is about to be important for west 0.9, which is going to change the detailed semantics for the "[PROJECT ...]" part of each command synopsis in a way that will be inconvenient to duplicate for each of these. Move the "Topologies supported" content lower down. Try to help the reading flow by letting people get familiar with a workspace and what you can do with it using the Zephyr GSG workspace that they probably already have before overwhelming them with other possibilities and choices. Signed-off-by: Martí Bolívar --- doc/guides/west/config.rst | 2 +- doc/guides/west/release-notes.rst | 2 +- doc/guides/west/repo-tool.rst | 464 +++++++++++++++++------------- doc/guides/west/why.rst | 4 + 4 files changed, 268 insertions(+), 204 deletions(-) diff --git a/doc/guides/west/config.rst b/doc/guides/west/config.rst index 5a9ee86d7fe..3079d7dcf7a 100644 --- a/doc/guides/west/config.rst +++ b/doc/guides/west/config.rst @@ -142,7 +142,7 @@ commands are documented in the pages for those commands. * - ``update.fetch`` - String, one of ``"smart"`` (the default behavior starting in v0.6.1) or ``"always"`` (the previous behavior). If set to ``"smart"``, the - :ref:`west update ` command will skip fetching + :ref:`west-update` command will skip fetching from project remotes when those projects' revisions in the manifest file are SHAs or tags which are already available locally. The ``"always"`` behavior is to unconditionally fetch from the remote. diff --git a/doc/guides/west/release-notes.rst b/doc/guides/west/release-notes.rst index b2eb7355854..5062b0ad732 100644 --- a/doc/guides/west/release-notes.rst +++ b/doc/guides/west/release-notes.rst @@ -149,7 +149,7 @@ v0.6.1 The user-visible features in this point release are: -- The `west update ` command has a new ``--fetch`` +- The :ref:`west-update` command has a new ``--fetch`` command line flag and ``update.fetch`` :ref:`configuration option `. The default value, "smart", skips fetching SHAs and tags which are available locally. diff --git a/doc/guides/west/repo-tool.rst b/doc/guides/west/repo-tool.rst index 0f5fd26cf1a..de6fa4522f4 100644 --- a/doc/guides/west/repo-tool.rst +++ b/doc/guides/west/repo-tool.rst @@ -1,79 +1,289 @@ .. _west-multi-repo: -Multiple Repository Management -############################## +Basics +###### -This page introduces basic concepts related to west and its multiple repository -management features, and gives an overview of the associated commands. See -:ref:`west-history` and `Zephyr issue #6770`_ for additional discussion, -rationale, and motivation. - -.. note:: - - West's multi-repo commands are meant to augment Git in minor ways for - multi-repo work, not to replace it. For tasks that only operate on one - repository, just use plain Git commands. +This page introduces west's basic concepts and built-in commands along with +references to further reading. .. _west-workspace: Introduction ************ -West's built-in commands allow you to work with projects composed of multiple -Git repositories installed under a common parent directory, which we call a -*west workspace* (before west 0.7, this was called a *west installation*). This -works similarly to `Git Submodules -`_ and Google's `repo -`_. +West's built-in commands allow you to work with *projects* (Git +repositories) under a common *workspace* directory. You can create a +workspace using the :ref:`west-init` command. -A west workspace is the result of running the ``west init`` command, which -is described in more detail below. For upstream Zephyr, the workspace looks -like this: +If you've followed the upstream Zephyr getting started guide, your +workspace looks like this: .. code-block:: none zephyrproject/ # west topdir - ├── .west/ - │ └── config - ├── zephyr/ # .git/ │ - │ ├── west.yml # manifest │ never modified by west - │ └── [... other files ...] │ + ├── .west/ # marks the location of the topdir + │ └── config # per-workspace local configuration file + ├── zephyr/ # .git/ repo │ the manifest repository, + │ ├── west.yml # manifest file │ never modified by west + │ └── [... other files ...] │ after creation ├── modules/ │ └── lib/ │ └── tinycbor/ # .git/ project ├── net-tools/ # .git/ project └── [ ... other projects ...] -Above, :file:`zephyrproject` is the name of the west workspace's root -directory. This name is just an example -- it could be anything, like ``z``, -``my-zephyr-workspace``, etc. The file :file:`.west/config` is the -workspace's :ref:`local configuration file `. +Above, :file:`zephyrproject` is the name of the workspace's top level +directory, or *topdir*. (The name :file:`zephyrproject` is just an example +-- it could be anything, like ``z``, ``my-zephyr-workspace``, etc.) + +The topdir contains the :file:`.west` directory. When west needs to find +the topdir, it searches for :file:`.west`, and uses its parent directory. +The search starts from the current working directory (and starts again from +the location in the :envvar:`ZEPHYR_BASE` environment variable as a +fallback if that fails). The file :file:`.west/config` is the workspace's +:ref:`local configuration file `. Every west workspace contains exactly one *manifest repository*, which is a -Git repository containing a file named :file:`west.yml`, which is the *west -manifest*. The location of the manifest repository is given by the -:ref:`manifest.path configuration option ` in the local -configuration file. The manifest file, along with west's configuration files, -controls the workspace's behavior. For upstream Zephyr, :file:`zephyr` is -the manifest repository, but you can configure west to use any Git repository -in the workspace as the manifest repository. The only requirement is that it -contains a valid manifest file. See :ref:`west-manifests` for more details on -what this means. +Git repository containing a *manifest file*. The location of the manifest +repository is given by the :ref:`manifest.path configuration option +` in the local configuration file. -Both of the :file:`tinycbor` and :file:`net-tools` directories are *projects* -managed by west, and configured in the manifest file. A west workspace can -contain arbitrarily many projects. As shown above, projects can be located -anywhere in the workspace. They don't have to be subdirectories of the -manifest directory, and they can be inside of arbitrary subdirectories inside -the workspace's root directory. By default, the Zephyr build system uses -west to get the locations of all the projects in the workspace, so any code -they contain can be used by applications. This behavior can be overridden using -the ``ZEPHYR_MODULES`` CMake variable; see :ref:`modules` for details. +For upstream Zephyr, :file:`zephyr` is the manifest repository, but you can +configure west to use any Git repository in the workspace as the manifest +repository. The only requirement is that it contains a valid manifest file. +See :ref:`west-topologies` for information on other options, and +:ref:`west-manifests` for details on the manifest file format. -Finally, any repository managed by a west workspace can contain -:ref:`extension commands `, which are extra west commands -provided by that project. This includes the manifest repository and any project -repository. +The manifest file is a YAML file that defines *projects*, which are the +additional Git repositories in the workspace managed by west. The manifest +file is named :file:`west.yml` by default; this can be overridden using the +``manifest.file`` local configuration option. + +You use the :ref:`west-update` command to update the workspace's projects +based on the contents of the manifest file. The manifest file controls +things like where the project should be placed within the workspace, what +URL to clone it from if it's missing, and what Git revision should be +checked out locally. + +Projects can be located anywhere inside the workspace, but they may not +"escape" it. Project repositories need not be located in subdirectories of +the manifest repository or as immediate subdirectories of the topdir. +However, projects must have paths inside the workspace. (You may replace a +project's repository directory within the workspace with a symbolic link to +elsewhere on your computer, but west will not do this for you.) + +A workspace can contain additional Git repositories or other files and +directories not managed by west. West basically ignores anything in the +workspace except :file:`.west`, the manifest repository, and the projects +specified in the manifest file. + +For upstream Zephyr, ``tinycbor`` and ``net-tools`` are projects. They are +specified in the manifest file :file:`zephyr/west.yml`. This file specifies +that ``tinycbor`` is located in the :file:`modules/lib/tinycbor` directory +beneath the workspace topdir. By default, the Zephyr :ref:`build system +` uses west to get the locations of all the projects in the +workspace, so any code they contain can be used as :ref:`modules`. + +Finally, any repository managed by a west workspace (either the manifest +repository or any project repository) can define :ref:`west-extensions`. +Extensions are extra commands not built into west that you can run when +using that workspace. + +The zephyr repository uses this feature to provide Zephyr-specific commands +like :ref:`west build `. Defining these as extensions keeps +west's core agnostic to the specifics of any workspace's Zephyr version, +etc. + +.. _west-struct: + +Structure +********* + +West's code is distributed via PyPI in a Python package named ``west``. +This distribution includes a launcher executable, which is also named +``west`` (or ``west.exe`` on Windows). + +When west is installed, the launcher is placed by :file:`pip3` somewhere in +the user's filesystem (exactly where depends on the operating system, but +should be on the ``PATH`` :ref:`environment variable `). This +launcher is the command-line entry point to running both built-in commmands +like ``west init``, ``west update``, along with any extensions discovered +in the workspace. + +In addition to its command-line interface, you can also use west's Python +APIs directly. See :ref:`west-apis` for details. + +.. _west-manifest-rev: + +The ``manifest-rev`` branch +*************************** + +West creates and controls a Git branch named ``manifest-rev`` in each +project. This branch points to the revision that the manifest file +specified for the project at the time :ref:`west-update` was last run. +Other workspace management commands may use ``manifest-rev`` as a reference +point for the upstream revision as of this latest update. Among other +purposes, the ``manifest-rev`` branch allows the manifest file to use SHAs +as project revisions. + +Although ``manifest-rev`` is a normal Git branch, west will recreate and/or +reset it on the next update. For this reason, it is **dangerous** +to check it out or otherwise modify it yourself. For instance, any commits +you manually add to this branch may be lost the next time you run ``west +update``. Instead, check out a local branch with another name, and either +rebase it on top of a new ``manifest-rev``, or merge ``manifest-rev`` into +it. + +.. note:: + + West does not create a ``manifest-rev`` branch in the manifest repository, + since west does not manage the manifest repository's branches or revisions. + +.. _west-built-in-cmds: + +Built-in Commands +***************** + +This section gives an overview of west's built-in commands. + +Some commands are related to Git commands with the same name, but operate +on the entire workspace. For example, ``west diff`` shows local changes in +multiple Git repositories in the workspace. + +Some commands take projects as arguments. These arguments can be project +names as specified in the manifest file, or (as a fallback) paths to them +on the local file system. Omitting project arguments to commands which +accept them (such as ``west list``, ``west forall``, etc.) usually defaults +to using all projects in the manifest file plus the manifest repository +itself. + +The following documentation does not exhaustively describe all commands. +For additional help, run ``west -h`` (e.g. ``west init -h``). + +.. _west-init: + +``west init`` +============= + +This command creates a west workspace. It can be used in two ways: + +1. Cloning a new manifest repository from a remote URL +2. Creating a workspace around an existing local manifest repository + +**Option 1**: to clone a new manifest repository from a remote URL, use: + +.. code-block:: none + + west init [-m URL] [--mr REVISION] [--mf FILE] [directory] + +The new workspace is created in the given :file:`directory`, creating a new +:file:`.west` inside this directory. You can give the manifest URL using +the ``-m`` switch, the initial revision to check out using ``--mr``, and +the location of the manifest file within the repository using ``--mf``. + +For example, running: + +.. code-block:: shell + + west init -m https://github.com/zephyrproject-rtos/zephyr --mr v1.14.0 zp + +would clone the upstream official zephyr repository into :file:`zp/zephyr`, +and check out the ``v1.14.0`` release. This command creates +:file:`zp/.west`, and set the ``manifest.path`` :ref:`configuration option +` to ``zephyr`` to record the location of the manifest +repository in the workspace. The default manifest file location is used. + +The ``-m`` option defaults to +``https://github.com/zephyrproject-rtos/zephyr``. The ``--mr`` option +defaults to ``master``. The ``--mf`` option defaults to ``west.yml``. If no +``directory`` is given, the current working directory is used. + +**Option 2**: to create a workspace around an existing local manifest +repository, use: + +.. code-block:: none + + west init -l [--mf FILE] directory + +This creates :file:`.west` **next to** :file:`directory` in the file +system, and sets ``manifest.path`` to ``directory``. + +As above, ``--mf`` defaults to ``west.yml``. + +The ``west init`` command does not clone any of the projects defined in the +manifest file, regardless of whether ``-l`` is given. To do that, use +``west update``. + +.. _west-update: + +``west update`` +--------------- + +This command clones and updates the projects specified in the :term:`west +manifest` file. + +.. code-block:: none + + west update [-h] [--stats] [-f {always,smart}] [-k] [-r] [PROJECT ...] + +By default, this command: + +#. Parses the manifest file, usually :file:`west.yml` +#. Clones any project repositories that are not already present locally +#. Fetches any project revisions which are not already pulled from the + remote +#. Sets each project's :ref:`manifest-rev ` branch to the + revision specified for that project in the manifest file +#. Checks out each ``manifest-rev`` in local working trees, as `detached + HEADs `_ + +To operate on a subset of projects only, specify them using the ``PROJECT`` +positional arguments, which can be either project names as given in the +manifest file, or paths to the local project clones. + +To force this command to fetch from project remotes even if the revisions +appear to be available locally, either use ``--fetch always`` or set the +``update.fetch`` :ref:`configuration option ` to ``"always"``. + +For safety, ``west update`` uses ``git checkout --detach`` to check out a +detached ``HEAD`` at the manifest revision for each updated project, +leaving behind any branches which were already checked out. This is +typically a safe operation that will not modify any of your local branches. + +If you would rather rebase any locally checked out branches instead, use +the ``-r`` (``--rebase``) option. + +If you would like ``west update`` to keep local branches checked out as +long as they point to commits that are descendants of the new +``manifest-rev``, use the ``-k`` (``--keep-descendants``) option. + +.. _west-multi-repo-misc: + +Other Repository Management Commands +==================================== + +West has a few more commands for managing the repositories in the +workspace, which are summarized here. Run ``west -h`` for +detailed help. + +- ``west list``: print a line of information about each project in the + manifest, according to a format string +- ``west manifest``: manage the manifest file. See :ref:`west-manifest-cmd`. +- ``west diff``: run ``git diff`` in local project repositories +- ``west status``: run ``git status`` in local project repositories +- ``west forall``: run an arbitrary command in local project repositories + +Additional Commands +=================== + +Finally, here is a summary of other built-in commands. + +- ``west config``: get or set :ref:`configuration options ` +- ``west topdir``: print the top level directory of the west workspace +- ``west help``: get help about a command, or print information about all + commands in the workspace, including :ref:`west-extensions` + +.. _west-topologies: Topologies supported ******************** @@ -250,156 +460,6 @@ v2.2.0 and its modules, then add the ``app1`` and ``app2`` projects: You can also do this "by hand" by copy/pasting :file:`zephyr/west.yml` as shown :ref:`above ` for the T2 topology, with the same caveats. -.. _west-struct: - -West Structure -************** - -West's code is distributed via PyPI in a Python package named ``west``. See -:ref:`west-apis` for API documentation. - -This distribution also includes a launcher executable, also named ``west``. -When west is installed, the launcher is placed by :file:`pip3` somewhere in the -user's filesystem (exactly where depends on the operating system, but should be -on the ``PATH`` :ref:`environment variable `). This launcher is the -command-line entry point. - -.. _west-manifest-rev: - -The ``manifest-rev`` branch -*************************** - -West creates a branch named ``manifest-rev`` in each project, pointing to the -commit the project's revision resolves to. The branch is updated whenever -project data is fetched by ``west update``. Other multi-repo commands also use -``manifest-rev`` as a reference for the upstream revision as of the most recent -update. See :ref:`west-multi-repo-cmds`, below, for more information. - -``manifest-rev`` is a normal Git branch, but if you delete or otherwise modify -it, west will recreate and/or reset it as if with ``git reset --hard`` on the -next update (though ``git update-ref`` is used internally). For this reason, it -is normally a **bad idea to modify it yourself**. ``manifest-rev`` was added to -allow SHAs as project revisions in the manifest, and to give a consistent -reference for the current upstream revision regardless of how the manifest -changes over time. - -.. note:: - - West does not create a ``manifest-rev`` branch in the manifest repository, - since west does not manage the manifest repository's branches or revisions. - -.. _west-multi-repo-cmds: - -Multi-Repo Commands -******************* - -This section gives a quick overview of the multi-repo commands, split up by -functionality. Some commands loosely mimic the corresponding Git command, but -in a multi-repo context (e.g. ``west diff`` shows local changes on all -repositories). - -Project arguments can be the names of projects in the manifest, or (as -fallback) paths to them. Omitting project arguments to commands which accept a -list of projects (such as ``west list``, ``west forall``, etc.) usually -defaults to using all projects in the manifest file plus the manifest -repository itself. - -For help on individual commands, run ``west -h`` (e.g. ``west diff --h``). - -Main Commands -============= - -The ``west init`` and ``west update`` multi-repo commands are the most -important to understand. - -- ``west init [-l] [-m URL] [--mr REVISION] [PATH]``: create a west - workspace in directory :file:`PATH` (i.e. :file:`.west` etc. will be - created there). If the ``PATH`` argument is not given, the current working - directory is used. This command does not clone any of the projects in the - manifest; that is done the next time ``west update`` is run. - - This command can be invoked in two ways: - - 1. If you already have a local clone of the zephyr repository and want to - create a west workspace around it, you can use the ``-l`` switch to - pass its path to west, as in: ``west init -l path/to/zephyr``. This is - the only reason to use ``-l``. - - 2. Otherwise, omit ``-l`` to create a new workspace from a remote manifest - repository. You can give the manifest URL using the ``-m`` switch, and its - revision using ``--mr``. For example, invoking west with: ``west init -m - https://github.com/zephyrproject-rtos/zephyr --mr v1.15.0`` would clone - the upstream official zephyr repository at the tagged release v1.15.0 - (``-m`` defaults to https://github.com/zephyrproject-rtos/zephyr, and - the ``-mr`` default is overridden to ``v1.15.0``). - -- ``west update [--fetch {always,smart}] [--rebase] [--keep-descendants] - [PROJECT ...]``: clone and update the specified projects based - on the current :term:`west manifest`. - - By default, this command: - - #. Parses the manifest file, :file:`west.yml` - #. Clones any project repositories that are not already present locally - #. Fetches any project revisions in the manifest file which are not already - pulled from the remote - #. Sets each project's :ref:`manifest-rev ` branch to the - current manifest revision - #. Checks out those revisions in local working trees - - To operate on a subset of projects only, specify them using the ``PROJECT`` - positional arguments, which can be either project names as given in the - manifest file, or paths to the local project clones. - - To force this command to fetch from project remotes even if the revisions - appear to be available locally, either use ``--fetch always`` or set the - ``update.fetch`` :ref:`configuration option ` to ``"always"``. - - For safety, ``west update`` uses ``git checkout --detach`` to check out a - detached ``HEAD`` at the manifest revision for each updated project, leaving - behind any branches which were already checked out. This is typically a safe - operation that will not modify any of your local branches. See the help for - the ``--rebase`` / ``-r`` and ``--keep-descendants`` / ``-k`` options for - ways to influence this. - -.. _west-multi-repo-misc: - -Miscellaneous Commands -====================== - -West has a few more commands for managing the multi-repo, which are briefly -discussed here. Run ``west -h`` for detailed help. - -- ``west forall -c COMMAND [PROJECT ...]``: Runs the shell command ``COMMAND`` - within the top-level repository directory of each of the specified projects - (default: all cloned projects). If ``COMMAND`` consists of more than one - word, it must be quoted to prevent it from being split up by the shell. - - To run an arbitrary Git command in each project, use something like ``west - forall -c 'git --options'``. Note that ``west forall`` can be used - to run any command, though, not just Git commands. - -- ``west help ``: this is equivalent to ``west -h``. - -- ``west list [-f FORMAT] [PROJECT ...]``: Lists project information from the - manifest file, such as URL, revision, path, etc. The printed information can - be controlled using the ``-f`` option. - -- ``west manifest``: Manipulates manifest files. See :ref:`west-manifest-cmd`. - -- ``west manifest --validate``: Ensure the current manifest file is - well-formed. Print information about what's wrong and fail the process in - case of error. - -- ``west diff [PROJECT ...]``: Runs a multi-repo ``git diff`` - for the specified projects. - -- ``west status [PROJECT ...]``: Like ``west diff``, for - running ``git status``. - -- ``west topdir``: Prints the top directory of the west workspace. - Private repositories ******************** diff --git a/doc/guides/west/why.rst b/doc/guides/west/why.rst index e690b757fc9..994ab3a5180 100644 --- a/doc/guides/west/why.rst +++ b/doc/guides/west/why.rst @@ -36,6 +36,10 @@ The basic requirements are: Rationale for a custom tool *************************** +Some of west's features are similar to those provided by +`Git Submodules `_ and +Google's `repo `_. + Existing tools were considered during west's initial design and development. None were found suitable for Zephyr's requirements. In particular, these were examined in detail: