1.. SPDX-License-Identifier: CC-BY-2.5 2 3===================== 4File Download Support 5===================== 6 7| 8 9BitBake's fetch module is a standalone piece of library code that deals 10with the intricacies of downloading source code and files from remote 11systems. Fetching source code is one of the cornerstones of building 12software. As such, this module forms an important part of BitBake. 13 14The current fetch module is called "fetch2" and refers to the fact that 15it is the second major version of the API. The original version is 16obsolete and has been removed from the codebase. Thus, in all cases, 17"fetch" refers to "fetch2" in this manual. 18 19The Download (Fetch) 20==================== 21 22BitBake takes several steps when fetching source code or files. The 23fetcher codebase deals with two distinct processes in order: obtaining 24the files from somewhere (cached or otherwise) and then unpacking those 25files into a specific location and perhaps in a specific way. Getting 26and unpacking the files is often optionally followed by patching. 27Patching, however, is not covered by this module. 28 29The code to execute the first part of this process, a fetch, looks 30something like the following:: 31 32 src_uri = (d.getVar('SRC_URI') or "").split() 33 fetcher = bb.fetch2.Fetch(src_uri, d) 34 fetcher.download() 35 36This code sets up an instance of the fetch class. The instance uses a 37space-separated list of URLs from the :term:`SRC_URI` 38variable and then calls the ``download`` method to download the files. 39 40The instantiation of the fetch class is usually followed by:: 41 42 rootdir = l.getVar('WORKDIR') 43 fetcher.unpack(rootdir) 44 45This code unpacks the downloaded files to the specified by ``WORKDIR``. 46 47.. note:: 48 49 For convenience, the naming in these examples matches the variables 50 used by OpenEmbedded. If you want to see the above code in action, 51 examine the OpenEmbedded class file ``base.bbclass`` 52 . 53 54The :term:`SRC_URI` and ``WORKDIR`` variables are not hardcoded into the 55fetcher, since those fetcher methods can be (and are) called with 56different variable names. In OpenEmbedded for example, the shared state 57(sstate) code uses the fetch module to fetch the sstate files. 58 59When the ``download()`` method is called, BitBake tries to resolve the 60URLs by looking for source files in a specific search order: 61 62- *Pre-mirror Sites:* BitBake first uses pre-mirrors to try and find 63 source files. These locations are defined using the 64 :term:`PREMIRRORS` variable. 65 66- *Source URI:* If pre-mirrors fail, BitBake uses the original URL (e.g 67 from :term:`SRC_URI`). 68 69- *Mirror Sites:* If fetch failures occur, BitBake next uses mirror 70 locations as defined by the :term:`MIRRORS` variable. 71 72For each URL passed to the fetcher, the fetcher calls the submodule that 73handles that particular URL type. This behavior can be the source of 74some confusion when you are providing URLs for the :term:`SRC_URI` variable. 75Consider the following two URLs:: 76 77 https://git.yoctoproject.org/git/poky;protocol=git 78 git://git.yoctoproject.org/git/poky;protocol=http 79 80In the former case, the URL is passed to the ``wget`` fetcher, which does not 81understand "git". Therefore, the latter case is the correct form since the Git 82fetcher does know how to use HTTP as a transport. 83 84Here are some examples that show commonly used mirror definitions:: 85 86 PREMIRRORS ?= "\ 87 bzr://.*/.\* http://somemirror.org/sources/ \ 88 cvs://.*/.\* http://somemirror.org/sources/ \ 89 git://.*/.\* http://somemirror.org/sources/ \ 90 hg://.*/.\* http://somemirror.org/sources/ \ 91 osc://.*/.\* http://somemirror.org/sources/ \ 92 p4://.*/.\* http://somemirror.org/sources/ \ 93 svn://.*/.\* http://somemirror.org/sources/" 94 95 MIRRORS =+ "\ 96 ftp://.*/.\* http://somemirror.org/sources/ \ 97 http://.*/.\* http://somemirror.org/sources/ \ 98 https://.*/.\* http://somemirror.org/sources/" 99 100It is useful to note that BitBake 101supports cross-URLs. It is possible to mirror a Git repository on an 102HTTP server as a tarball. This is what the ``git://`` mapping in the 103previous example does. 104 105Since network accesses are slow, BitBake maintains a cache of files 106downloaded from the network. Any source files that are not local (i.e. 107downloaded from the Internet) are placed into the download directory, 108which is specified by the :term:`DL_DIR` variable. 109 110File integrity is of key importance for reproducing builds. For 111non-local archive downloads, the fetcher code can verify SHA-256 and MD5 112checksums to ensure the archives have been downloaded correctly. You can 113specify these checksums by using the :term:`SRC_URI` variable with the 114appropriate varflags as follows:: 115 116 SRC_URI[md5sum] = "value" 117 SRC_URI[sha256sum] = "value" 118 119You can also specify the checksums as 120parameters on the :term:`SRC_URI` as shown below:: 121 122 SRC_URI = "http://example.com/foobar.tar.bz2;md5sum=4a8e0f237e961fd7785d19d07fdb994d" 123 124If multiple URIs exist, you can specify the checksums either directly as 125in the previous example, or you can name the URLs. The following syntax 126shows how you name the URIs:: 127 128 SRC_URI = "http://example.com/foobar.tar.bz2;name=foo" 129 SRC_URI[foo.md5sum] = 4a8e0f237e961fd7785d19d07fdb994d 130 131After a file has been downloaded and 132has had its checksum checked, a ".done" stamp is placed in :term:`DL_DIR`. 133BitBake uses this stamp during subsequent builds to avoid downloading or 134comparing a checksum for the file again. 135 136.. note:: 137 138 It is assumed that local storage is safe from data corruption. If 139 this were not the case, there would be bigger issues to worry about. 140 141If :term:`BB_STRICT_CHECKSUM` is set, any 142download without a checksum triggers an error message. The 143:term:`BB_NO_NETWORK` variable can be used to 144make any attempted network access a fatal error, which is useful for 145checking that mirrors are complete as well as other things. 146 147If :term:`BB_CHECK_SSL_CERTS` is set to ``0`` then SSL certificate checking will 148be disabled. This variable defaults to ``1`` so SSL certificates are normally 149checked. 150 151.. _bb-the-unpack: 152 153The Unpack 154========== 155 156The unpack process usually immediately follows the download. For all 157URLs except Git URLs, BitBake uses the common ``unpack`` method. 158 159A number of parameters exist that you can specify within the URL to 160govern the behavior of the unpack stage: 161 162- *unpack:* Controls whether the URL components are unpacked. If set to 163 "1", which is the default, the components are unpacked. If set to 164 "0", the unpack stage leaves the file alone. This parameter is useful 165 when you want an archive to be copied in and not be unpacked. 166 167- *dos:* Applies to ``.zip`` and ``.jar`` files and specifies whether 168 to use DOS line ending conversion on text files. 169 170- *striplevel:* Strip specified number of leading components (levels) 171 from file names on extraction 172 173- *subdir:* Unpacks the specific URL to the specified subdirectory 174 within the root directory. 175 176The unpack call automatically decompresses and extracts files with ".Z", 177".z", ".gz", ".xz", ".zip", ".jar", ".ipk", ".rpm". ".srpm", ".deb" and 178".bz2" extensions as well as various combinations of tarball extensions. 179 180As mentioned, the Git fetcher has its own unpack method that is 181optimized to work with Git trees. Basically, this method works by 182cloning the tree into the final directory. The process is completed 183using references so that there is only one central copy of the Git 184metadata needed. 185 186.. _bb-fetchers: 187 188Fetchers 189======== 190 191As mentioned earlier, the URL prefix determines which fetcher submodule 192BitBake uses. Each submodule can support different URL parameters, which 193are described in the following sections. 194 195.. _local-file-fetcher: 196 197Local file fetcher (``file://``) 198-------------------------------- 199 200This submodule handles URLs that begin with ``file://``. The filename 201you specify within the URL can be either an absolute or relative path to 202a file. If the filename is relative, the contents of the 203:term:`FILESPATH` variable is used in the same way 204``PATH`` is used to find executables. If the file cannot be found, it is 205assumed that it is available in :term:`DL_DIR` by the 206time the ``download()`` method is called. 207 208If you specify a directory, the entire directory is unpacked. 209 210Here are a couple of example URLs, the first relative and the second 211absolute:: 212 213 SRC_URI = "file://relativefile.patch" 214 SRC_URI = "file:///Users/ich/very_important_software" 215 216.. _http-ftp-fetcher: 217 218HTTP/FTP wget fetcher (``http://``, ``ftp://``, ``https://``) 219------------------------------------------------------------- 220 221This fetcher obtains files from web and FTP servers. Internally, the 222fetcher uses the wget utility. 223 224The executable and parameters used are specified by the 225``FETCHCMD_wget`` variable, which defaults to sensible values. The 226fetcher supports a parameter "downloadfilename" that allows the name of 227the downloaded file to be specified. Specifying the name of the 228downloaded file is useful for avoiding collisions in 229:term:`DL_DIR` when dealing with multiple files that 230have the same name. 231 232If a username and password are specified in the ``SRC_URI``, a Basic 233Authorization header will be added to each request, including across redirects. 234To instead limit the Authorization header to the first request, add 235"redirectauth=0" to the list of parameters. 236 237Some example URLs are as follows:: 238 239 SRC_URI = "http://oe.handhelds.org/not_there.aac" 240 SRC_URI = "ftp://oe.handhelds.org/not_there_as_well.aac" 241 SRC_URI = "ftp://you@oe.handhelds.org/home/you/secret.plan" 242 243.. note:: 244 245 Because URL parameters are delimited by semi-colons, this can 246 introduce ambiguity when parsing URLs that also contain semi-colons, 247 for example:: 248 249 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git;a=snapshot;h=a5dd47" 250 251 252 Such URLs should should be modified by replacing semi-colons with '&' 253 characters:: 254 255 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47" 256 257 258 In most cases this should work. Treating semi-colons and '&' in 259 queries identically is recommended by the World Wide Web Consortium 260 (W3C). Note that due to the nature of the URL, you may have to 261 specify the name of the downloaded file as well:: 262 263 SRC_URI = "http://abc123.org/git/?p=gcc/gcc.git&a=snapshot&h=a5dd47;downloadfilename=myfile.bz2" 264 265 266.. _cvs-fetcher: 267 268CVS fetcher (``(cvs://``) 269------------------------- 270 271This submodule handles checking out files from the CVS version control 272system. You can configure it using a number of different variables: 273 274- :term:`FETCHCMD_cvs <FETCHCMD>`: The name of the executable to use when running 275 the ``cvs`` command. This name is usually "cvs". 276 277- :term:`SRCDATE`: The date to use when fetching the CVS source code. A 278 special value of "now" causes the checkout to be updated on every 279 build. 280 281- :term:`CVSDIR`: Specifies where a temporary 282 checkout is saved. The location is often ``DL_DIR/cvs``. 283 284- CVS_PROXY_HOST: The name to use as a "proxy=" parameter to the 285 ``cvs`` command. 286 287- CVS_PROXY_PORT: The port number to use as a "proxyport=" 288 parameter to the ``cvs`` command. 289 290As well as the standard username and password URL syntax, you can also 291configure the fetcher with various URL parameters: 292 293The supported parameters are as follows: 294 295- *"method":* The protocol over which to communicate with the CVS 296 server. By default, this protocol is "pserver". If "method" is set to 297 "ext", BitBake examines the "rsh" parameter and sets ``CVS_RSH``. You 298 can use "dir" for local directories. 299 300- *"module":* Specifies the module to check out. You must supply this 301 parameter. 302 303- *"tag":* Describes which CVS TAG should be used for the checkout. By 304 default, the TAG is empty. 305 306- *"date":* Specifies a date. If no "date" is specified, the 307 :term:`SRCDATE` of the configuration is used to 308 checkout a specific date. The special value of "now" causes the 309 checkout to be updated on every build. 310 311- *"localdir":* Used to rename the module. Effectively, you are 312 renaming the output directory to which the module is unpacked. You 313 are forcing the module into a special directory relative to 314 :term:`CVSDIR`. 315 316- *"rsh":* Used in conjunction with the "method" parameter. 317 318- *"scmdata":* Causes the CVS metadata to be maintained in the tarball 319 the fetcher creates when set to "keep". The tarball is expanded into 320 the work directory. By default, the CVS metadata is removed. 321 322- *"fullpath":* Controls whether the resulting checkout is at the 323 module level, which is the default, or is at deeper paths. 324 325- *"norecurse":* Causes the fetcher to only checkout the specified 326 directory with no recurse into any subdirectories. 327 328- *"port":* The port to which the CVS server connects. 329 330Some example URLs are as follows:: 331 332 SRC_URI = "cvs://CVSROOT;module=mymodule;tag=some-version;method=ext" 333 SRC_URI = "cvs://CVSROOT;module=mymodule;date=20060126;localdir=usethat" 334 335.. _svn-fetcher: 336 337Subversion (SVN) Fetcher (``svn://``) 338------------------------------------- 339 340This fetcher submodule fetches code from the Subversion source control 341system. The executable used is specified by ``FETCHCMD_svn``, which 342defaults to "svn". The fetcher's temporary working directory is set by 343:term:`SVNDIR`, which is usually ``DL_DIR/svn``. 344 345The supported parameters are as follows: 346 347- *"module":* The name of the svn module to checkout. You must provide 348 this parameter. You can think of this parameter as the top-level 349 directory of the repository data you want. 350 351- *"path_spec":* A specific directory in which to checkout the 352 specified svn module. 353 354- *"protocol":* The protocol to use, which defaults to "svn". If 355 "protocol" is set to "svn+ssh", the "ssh" parameter is also used. 356 357- *"rev":* The revision of the source code to checkout. 358 359- *"scmdata":* Causes the ".svn" directories to be available during 360 compile-time when set to "keep". By default, these directories are 361 removed. 362 363- *"ssh":* An optional parameter used when "protocol" is set to 364 "svn+ssh". You can use this parameter to specify the ssh program used 365 by svn. 366 367- *"transportuser":* When required, sets the username for the 368 transport. By default, this parameter is empty. The transport 369 username is different than the username used in the main URL, which 370 is passed to the subversion command. 371 372Following are three examples using svn:: 373 374 SRC_URI = "svn://myrepos/proj1;module=vip;protocol=http;rev=667" 375 SRC_URI = "svn://myrepos/proj1;module=opie;protocol=svn+ssh" 376 SRC_URI = "svn://myrepos/proj1;module=trunk;protocol=http;path_spec=${MY_DIR}/proj1" 377 378.. _git-fetcher: 379 380Git Fetcher (``git://``) 381------------------------ 382 383This fetcher submodule fetches code from the Git source control system. 384The fetcher works by creating a bare clone of the remote into 385:term:`GITDIR`, which is usually ``DL_DIR/git2``. This 386bare clone is then cloned into the work directory during the unpack 387stage when a specific tree is checked out. This is done using alternates 388and by reference to minimize the amount of duplicate data on the disk 389and make the unpack process fast. The executable used can be set with 390``FETCHCMD_git``. 391 392This fetcher supports the following parameters: 393 394- *"protocol":* The protocol used to fetch the files. The default is 395 "git" when a hostname is set. If a hostname is not set, the Git 396 protocol is "file". You can also use "http", "https", "ssh" and 397 "rsync". 398 399 .. note:: 400 401 When ``protocol`` is "ssh", the URL expected in :term:`SRC_URI` differs 402 from the one that is typically passed to ``git clone`` command and provided 403 by the Git server to fetch from. For example, the URL returned by GitLab 404 server for ``mesa`` when cloning over SSH is 405 ``git@gitlab.freedesktop.org:mesa/mesa.git``, however the expected URL in 406 :term:`SRC_URI` is the following:: 407 408 SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." 409 410 Note the ``:`` character changed for a ``/`` before the path to the project. 411 412- *"nocheckout":* Tells the fetcher to not checkout source code when 413 unpacking when set to "1". Set this option for the URL where there is 414 a custom routine to checkout code. The default is "0". 415 416- *"rebaseable":* Indicates that the upstream Git repository can be 417 rebased. You should set this parameter to "1" if revisions can become 418 detached from branches. In this case, the source mirror tarball is 419 done per revision, which has a loss of efficiency. Rebasing the 420 upstream Git repository could cause the current revision to disappear 421 from the upstream repository. This option reminds the fetcher to 422 preserve the local cache carefully for future use. The default value 423 for this parameter is "0". 424 425- *"nobranch":* Tells the fetcher to not check the SHA validation for 426 the branch when set to "1". The default is "0". Set this option for 427 the recipe that refers to the commit that is valid for any namespace 428 (branch, tag, ...) instead of the branch. 429 430- *"bareclone":* Tells the fetcher to clone a bare clone into the 431 destination directory without checking out a working tree. Only the 432 raw Git metadata is provided. This parameter implies the "nocheckout" 433 parameter as well. 434 435- *"branch":* The branch(es) of the Git tree to clone. Unless 436 "nobranch" is set to "1", this is a mandatory parameter. The number of 437 branch parameters must match the number of name parameters. 438 439- *"rev":* The revision to use for the checkout. The default is 440 "master". 441 442- *"tag":* Specifies a tag to use for the checkout. To correctly 443 resolve tags, BitBake must access the network. For that reason, tags 444 are often not used. As far as Git is concerned, the "tag" parameter 445 behaves effectively the same as the "rev" parameter. 446 447- *"subpath":* Limits the checkout to a specific subpath of the tree. 448 By default, the whole tree is checked out. 449 450- *"destsuffix":* The name of the path in which to place the checkout. 451 By default, the path is ``git/``. 452 453- *"usehead":* Enables local ``git://`` URLs to use the current branch 454 HEAD as the revision for use with ``AUTOREV``. The "usehead" 455 parameter implies no branch and only works when the transfer protocol 456 is ``file://``. 457 458Here are some example URLs:: 459 460 SRC_URI = "git://github.com/fronteed/icheck.git;protocol=https;branch=${PV};tag=${PV}" 461 SRC_URI = "git://github.com/asciidoc/asciidoc-py;protocol=https;branch=main" 462 SRC_URI = "git://git@gitlab.freedesktop.org/mesa/mesa.git;branch=main;protocol=ssh;..." 463 464.. note:: 465 466 When using ``git`` as the fetcher of the main source code of your software, 467 ``S`` should be set accordingly:: 468 469 S = "${WORKDIR}/git" 470 471.. note:: 472 473 Specifying passwords directly in ``git://`` urls is not supported. 474 There are several reasons: :term:`SRC_URI` is often written out to logs and 475 other places, and that could easily leak passwords; it is also all too 476 easy to share metadata without removing passwords. SSH keys, ``~/.netrc`` 477 and ``~/.ssh/config`` files can be used as alternatives. 478 479Using tags with the git fetcher may cause surprising behaviour. Bitbake needs to 480resolve the tag to a specific revision and to do that, it has to connect to and use 481the upstream repository. This is because the revision the tags point at can change and 482we've seen cases of this happening in well known public repositories. This can mean 483many more network connections than expected and recipes may be reparsed at every build. 484Source mirrors will also be bypassed as the upstream repository is the only source 485of truth to resolve the revision accurately. For these reasons, whilst the fetcher 486can support tags, we recommend being specific about revisions in recipes. 487 488.. _gitsm-fetcher: 489 490Git Submodule Fetcher (``gitsm://``) 491------------------------------------ 492 493This fetcher submodule inherits from the :ref:`Git 494fetcher<bitbake-user-manual/bitbake-user-manual-fetching:git fetcher 495(\`\`git://\`\`)>` and extends that fetcher's behavior by fetching a 496repository's submodules. :term:`SRC_URI` is passed to the Git fetcher as 497described in the :ref:`bitbake-user-manual/bitbake-user-manual-fetching:git 498fetcher (\`\`git://\`\`)` section. 499 500.. note:: 501 502 You must clean a recipe when switching between '``git://``' and 503 '``gitsm://``' URLs. 504 505 The Git Submodules fetcher is not a complete fetcher implementation. 506 The fetcher has known issues where it does not use the normal source 507 mirroring infrastructure properly. Further, the submodule sources it 508 fetches are not visible to the licensing and source archiving 509 infrastructures. 510 511.. _clearcase-fetcher: 512 513ClearCase Fetcher (``ccrc://``) 514------------------------------- 515 516This fetcher submodule fetches code from a 517`ClearCase <http://en.wikipedia.org/wiki/Rational_ClearCase>`__ 518repository. 519 520To use this fetcher, make sure your recipe has proper 521:term:`SRC_URI`, :term:`SRCREV`, and 522:term:`PV` settings. Here is an example:: 523 524 SRC_URI = "ccrc://cc.example.org/ccrc;vob=/example_vob;module=/example_module" 525 SRCREV = "EXAMPLE_CLEARCASE_TAG" 526 PV = "${@d.getVar("SRCREV", False).replace("/", "+")}" 527 528The fetcher uses the ``rcleartool`` or 529``cleartool`` remote client, depending on which one is available. 530 531Following are options for the :term:`SRC_URI` statement: 532 533- *vob*: The name, which must include the prepending "/" character, 534 of the ClearCase VOB. This option is required. 535 536- *module*: The module, which must include the prepending "/" 537 character, in the selected VOB. 538 539 .. note:: 540 541 The module and vob options are combined to create the load rule in the 542 view config spec. As an example, consider the vob and module values from 543 the SRC_URI statement at the start of this section. Combining those values 544 results in the following:: 545 546 load /example_vob/example_module 547 548- *proto*: The protocol, which can be either ``http`` or ``https``. 549 550By default, the fetcher creates a configuration specification. If you 551want this specification written to an area other than the default, use 552the ``CCASE_CUSTOM_CONFIG_SPEC`` variable in your recipe to define where 553the specification is written. 554 555.. note:: 556 557 the SRCREV loses its functionality if you specify this variable. However, 558 SRCREV is still used to label the archive after a fetch even though it does 559 not define what is fetched. 560 561Here are a couple of other behaviors worth mentioning: 562 563- When using ``cleartool``, the login of ``cleartool`` is handled by 564 the system. The login require no special steps. 565 566- In order to use ``rcleartool`` with authenticated users, an 567 "rcleartool login" is necessary before using the fetcher. 568 569.. _perforce-fetcher: 570 571Perforce Fetcher (``p4://``) 572---------------------------- 573 574This fetcher submodule fetches code from the 575`Perforce <https://www.perforce.com/>`__ source control system. The 576executable used is specified by ``FETCHCMD_p4``, which defaults to "p4". 577The fetcher's temporary working directory is set by 578:term:`P4DIR`, which defaults to "DL_DIR/p4". 579The fetcher does not make use of a perforce client, instead it 580relies on ``p4 files`` to retrieve a list of 581files and ``p4 print`` to transfer the content 582of those files locally. 583 584To use this fetcher, make sure your recipe has proper 585:term:`SRC_URI`, :term:`SRCREV`, and 586:term:`PV` values. The p4 executable is able to use the 587config file defined by your system's ``P4CONFIG`` environment variable 588in order to define the Perforce server URL and port, username, and 589password if you do not wish to keep those values in a recipe itself. If 590you choose not to use ``P4CONFIG``, or to explicitly set variables that 591``P4CONFIG`` can contain, you can specify the ``P4PORT`` value, which is 592the server's URL and port number, and you can specify a username and 593password directly in your recipe within :term:`SRC_URI`. 594 595Here is an example that relies on ``P4CONFIG`` to specify the server URL 596and port, username, and password, and fetches the Head Revision:: 597 598 SRC_URI = "p4://example-depot/main/source/..." 599 SRCREV = "${AUTOREV}" 600 PV = "p4-${SRCPV}" 601 S = "${WORKDIR}/p4" 602 603Here is an example that specifies the server URL and port, username, and 604password, and fetches a Revision based on a Label:: 605 606 P4PORT = "tcp:p4server.example.net:1666" 607 SRC_URI = "p4://user:passwd@example-depot/main/source/..." 608 SRCREV = "release-1.0" 609 PV = "p4-${SRCPV}" 610 S = "${WORKDIR}/p4" 611 612.. note:: 613 614 You should always set S to "${WORKDIR}/p4" in your recipe. 615 616By default, the fetcher strips the depot location from the local file paths. In 617the above example, the content of ``example-depot/main/source/`` will be placed 618in ``${WORKDIR}/p4``. For situations where preserving parts of the remote depot 619paths locally is desirable, the fetcher supports two parameters: 620 621- *"module":* 622 The top-level depot location or directory to fetch. The value of this 623 parameter can also point to a single file within the depot, in which case 624 the local file path will include the module path. 625- *"remotepath":* 626 When used with the value "``keep``", the fetcher will mirror the full depot 627 paths locally for the specified location, even in combination with the 628 ``module`` parameter. 629 630Here is an example use of the the ``module`` parameter:: 631 632 SRC_URI = "p4://user:passwd@example-depot/main;module=source/..." 633 634In this case, the content of the top-level directory ``source/`` will be fetched 635to ``${P4DIR}``, including the directory itself. The top-level directory will 636be accesible at ``${P4DIR}/source/``. 637 638Here is an example use of the the ``remotepath`` parameter:: 639 640 SRC_URI = "p4://user:passwd@example-depot/main;module=source/...;remotepath=keep" 641 642In this case, the content of the top-level directory ``source/`` will be fetched 643to ``${P4DIR}``, but the complete depot paths will be mirrored locally. The 644top-level directory will be accessible at 645``${P4DIR}/example-depot/main/source/``. 646 647.. _repo-fetcher: 648 649Repo Fetcher (``repo://``) 650-------------------------- 651 652This fetcher submodule fetches code from ``google-repo`` source control 653system. The fetcher works by initiating and syncing sources of the 654repository into :term:`REPODIR`, which is usually 655``${DL_DIR}/repo``. 656 657This fetcher supports the following parameters: 658 659- *"protocol":* Protocol to fetch the repository manifest (default: 660 git). 661 662- *"branch":* Branch or tag of repository to get (default: master). 663 664- *"manifest":* Name of the manifest file (default: ``default.xml``). 665 666Here are some example URLs:: 667 668 SRC_URI = "repo://REPOROOT;protocol=git;branch=some_branch;manifest=my_manifest.xml" 669 SRC_URI = "repo://REPOROOT;protocol=file;branch=some_branch;manifest=my_manifest.xml" 670 671.. _az-fetcher: 672 673Az Fetcher (``az://``) 674-------------------------- 675 676This submodule fetches data from an 677`Azure Storage account <https://docs.microsoft.com/en-us/azure/storage/>`__ , 678it inherits its functionality from the HTTP wget fetcher, but modifies its 679behavior to accomodate the usage of a 680`Shared Access Signature (SAS) <https://docs.microsoft.com/en-us/azure/storage/common/storage-sas-overview>`__ 681for non-public data. 682 683Such functionality is set by the variable: 684 685- :term:`AZ_SAS`: The Azure Storage Shared Access Signature provides secure 686 delegate access to resources, if this variable is set, the Az Fetcher will 687 use it when fetching artifacts from the cloud. 688 689You can specify the AZ_SAS variable as shown below:: 690 691 AZ_SAS = "se=2021-01-01&sp=r&sv=2018-11-09&sr=c&skoid=<skoid>&sig=<signature>" 692 693Here is an example URL:: 694 695 SRC_URI = "az://<azure-storage-account>.blob.core.windows.net/<foo_container>/<bar_file>" 696 697It can also be used when setting mirrors definitions using the :term:`PREMIRRORS` variable. 698 699.. _gcp-fetcher: 700 701GCP Fetcher (``gs://``) 702-------------------------- 703 704This submodule fetches data from a 705`Google Cloud Storage Bucket <https://cloud.google.com/storage/docs/buckets>`__. 706It uses the `Google Cloud Storage Python Client <https://cloud.google.com/python/docs/reference/storage/latest>`__ 707to check the status of objects in the bucket and download them. 708The use of the Python client makes it substantially faster than using command 709line tools such as gsutil. 710 711The fetcher requires the Google Cloud Storage Python Client to be installed, along 712with the gsutil tool. 713 714The fetcher requires that the machine has valid credentials for accessing the 715chosen bucket. Instructions for authentication can be found in the 716`Google Cloud documentation <https://cloud.google.com/docs/authentication/provide-credentials-adc#local-dev>`__. 717 718If it used from the OpenEmbedded build system, the fetcher can be used for 719fetching sstate artifacts from a GCS bucket by specifying the 720``SSTATE_MIRRORS`` variable as shown below:: 721 722 SSTATE_MIRRORS ?= "\ 723 file://.* gs://<bucket name>/PATH \ 724 " 725 726The fetcher can also be used in recipes:: 727 728 SRC_URI = "gs://<bucket name>/<foo_container>/<bar_file>" 729 730However, the checksum of the file should be also be provided:: 731 732 SRC_URI[sha256sum] = "<sha256 string>" 733 734.. _crate-fetcher: 735 736Crate Fetcher (``crate://``) 737---------------------------- 738 739This submodule fetches code for 740`Rust language "crates" <https://doc.rust-lang.org/reference/glossary.html?highlight=crate#crate>`__ 741corresponding to Rust libraries and programs to compile. Such crates are typically shared 742on https://crates.io/ but this fetcher supports other crate registries too. 743 744The format for the :term:`SRC_URI` setting must be:: 745 746 SRC_URI = "crate://REGISTRY/NAME/VERSION" 747 748Here is an example URL:: 749 750 SRC_URI = "crate://crates.io/glob/0.2.11" 751 752.. _npm-fetcher: 753 754NPM Fetcher (``npm://``) 755------------------------ 756 757This submodule fetches source code from an 758`NPM <https://en.wikipedia.org/wiki/Npm_(software)>`__ 759Javascript package registry. 760 761The format for the :term:`SRC_URI` setting must be:: 762 763 SRC_URI = "npm://some.registry.url;ParameterA=xxx;ParameterB=xxx;..." 764 765This fetcher supports the following parameters: 766 767- *"package":* The NPM package name. This is a mandatory parameter. 768 769- *"version":* The NPM package version. This is a mandatory parameter. 770 771- *"downloadfilename":* Specifies the filename used when storing the downloaded file. 772 773- *"destsuffix":* Specifies the directory to use to unpack the package (default: ``npm``). 774 775Note that NPM fetcher only fetches the package source itself. The dependencies 776can be fetched through the `npmsw-fetcher`_. 777 778Here is an example URL with both fetchers:: 779 780 SRC_URI = " \ 781 npm://registry.npmjs.org/;package=cute-files;version=${PV} \ 782 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \ 783 " 784 785See :yocto_docs:`Creating Node Package Manager (NPM) Packages 786</dev-manual/packages.html#creating-node-package-manager-npm-packages>` 787in the Yocto Project manual for details about using 788:yocto_docs:`devtool <https://docs.yoctoproject.org/ref-manual/devtool-reference.html>` 789to automatically create a recipe from an NPM URL. 790 791.. _npmsw-fetcher: 792 793NPM shrinkwrap Fetcher (``npmsw://``) 794------------------------------------- 795 796This submodule fetches source code from an 797`NPM shrinkwrap <https://docs.npmjs.com/cli/v8/commands/npm-shrinkwrap>`__ 798description file, which lists the dependencies 799of an NPM package while locking their versions. 800 801The format for the :term:`SRC_URI` setting must be:: 802 803 SRC_URI = "npmsw://some.registry.url;ParameterA=xxx;ParameterB=xxx;..." 804 805This fetcher supports the following parameters: 806 807- *"dev":* Set this parameter to ``1`` to install "devDependencies". 808 809- *"destsuffix":* Specifies the directory to use to unpack the dependencies 810 (``${S}`` by default). 811 812Note that the shrinkwrap file can also be provided by the recipe for 813the package which has such dependencies, for example:: 814 815 SRC_URI = " \ 816 npm://registry.npmjs.org/;package=cute-files;version=${PV} \ 817 npmsw://${THISDIR}/${BPN}/npm-shrinkwrap.json \ 818 " 819 820Such a file can automatically be generated using 821:yocto_docs:`devtool <https://docs.yoctoproject.org/ref-manual/devtool-reference.html>` 822as described in the :yocto_docs:`Creating Node Package Manager (NPM) Packages 823</dev-manual/packages.html#creating-node-package-manager-npm-packages>` 824section of the Yocto Project. 825 826Other Fetchers 827-------------- 828 829Fetch submodules also exist for the following: 830 831- Bazaar (``bzr://``) 832 833- Mercurial (``hg://``) 834 835- OSC (``osc://``) 836 837- S3 (``s3://``) 838 839- Secure FTP (``sftp://``) 840 841- Secure Shell (``ssh://``) 842 843- Trees using Git Annex (``gitannex://``) 844 845No documentation currently exists for these lesser used fetcher 846submodules. However, you might find the code helpful and readable. 847 848Auto Revisions 849============== 850 851We need to document ``AUTOREV`` and :term:`SRCREV_FORMAT` here. 852