dt-bindings: kbuild: Use DTB files for validation

Switch the DT validation to use DTB files directly instead of a DTS to
YAML conversion.

The original motivation for supporting validation on DTB fi

dt-bindings: kbuild: Use DTB files for validation

Switch the DT validation to use DTB files directly instead of a DTS to
YAML conversion.

The original motivation for supporting validation on DTB files was to
enable running validation on a running system (e.g. 'dt-validate
/sys/firmware/fdt') or other cases where the original source DTS is not
available.

The YAML format was not without issues. Using DTBs with the schema type
information solves some of those problems. The YAML format relies on the
DTS source level information including bracketing of properties, size
directives, and phandle tags all of which are lost in a DTB file. While
standardizing the bracketing is a good thing, it does cause a lot of
extra warnings and churn to fix them.

Another issue has been signed types are not validated correctly as sign
information is not propagated to YAML. Using the schema type information
allows for proper handling of signed types. YAML also can't represent
the full range of 64-bit integers as numbers are stored as floats by
most/all parsers.

The DTB validation works by decoding property values using the type
information in the schemas themselves. The main corner case this does
not work for is matrix types where neither dimension is fixed. For
now, checking the dimensions in these cases are skipped.

Signed-off-by: Rob Herring <robh@kernel.org>
Tested-by: Geert Uytterhoeven <geert+renesas@glider.be>
Link: https://lore.kernel.org/r/20220310160513.1708182-3-robh@kernel.org

show more ...

scripts/dtc: Call pkg-config POSIXly correct

Running with POSIXLY_CORRECT=1 in the environment the scripts/dtc build
fails, because pkg-config doesn't output anything when the flags come
after the a

scripts/dtc: Call pkg-config POSIXly correct

Running with POSIXLY_CORRECT=1 in the environment the scripts/dtc build
fails, because pkg-config doesn't output anything when the flags come
after the arguments.

Fixes: 067c650c456e ("dtc: Use pkg-config to locate libyaml")
Signed-off-by: Thomas Bracht Laumann Jespersen <t@laumann.xyz>
Signed-off-by: Rob Herring <robh@kernel.org>
Link: https://lore.kernel.org/r/20220131112028.7907-1-t@laumann.xyz

show more ...

scripts/dtc: Call pkg-config POSIXly correct

[ Upstream commit a8b309ce9760943486e0585285e0125588a31650 ]

Running with POSIXLY_CORRECT=1 in the environment the scripts/dtc build
fails, because pkg-

scripts/dtc: Call pkg-config POSIXly correct

[ Upstream commit a8b309ce9760943486e0585285e0125588a31650 ]

Running with POSIXLY_CORRECT=1 in the environment the scripts/dtc build
fails, because pkg-config doesn't output anything when the flags come
after the arguments.

Fixes: 067c650c456e ("dtc: Use pkg-config to locate libyaml")
Signed-off-by: Thomas Bracht Laumann Jespersen <t@laumann.xyz>
Signed-off-by: Rob Herring <robh@kernel.org>
Link: https://lore.kernel.org/r/20220131112028.7907-1-t@laumann.xyz
Signed-off-by: Sasha Levin <sashal@kernel.org>

show more ...

scripts/dtc: Add missing fdtoverlay to gitignore

Commit 0da6bcd9fcc0 ("scripts: dtc: Build fdtoverlay tool") enabled
building fdtoverlay, but failed to add it to .gitignore.

Also add a note to keep

scripts/dtc: Add missing fdtoverlay to gitignore

Commit 0da6bcd9fcc0 ("scripts: dtc: Build fdtoverlay tool") enabled
building fdtoverlay, but failed to add it to .gitignore.

Also add a note to keep hostprogs in sync with .gitignore.

Fixes: 0da6bcd9fcc0 ("scripts: dtc: Build fdtoverlay tool")
Reported-by: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Viresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: Rob Herring <robh@kernel.org>

show more ...

scripts: dtc: Build fdtoverlay tool

We will start building overlays for platforms soon in the kernel and
would need fdtoverlay going forward. Lets start building it.

The fdtoverlay program applies

scripts: dtc: Build fdtoverlay tool

We will start building overlays for platforms soon in the kernel and
would need fdtoverlay going forward. Lets start building it.

The fdtoverlay program applies one or more overlay dtb blobs to a base
dtb blob. The kernel build system would later use fdtoverlay to generate
the overlaid blobs based on platform specific configurations.

Signed-off-by: Viresh Kumar <viresh.kumar@linaro.org>
Signed-off-by: Rob Herring <robh@kernel.org>
Link: https://lore.kernel.org/r/4a201dea3ba11a00cab7e936dfc1140dac1a1ae3.1611904394.git.viresh.kumar@linaro.org

show more ...

scripts/dtc: only append to HOST_EXTRACFLAGS instead of overwriting

When building with

$ HOST_EXTRACFLAGS=-g make

the expectation is that host tools are built with debug informations.
This howeve

scripts/dtc: only append to HOST_EXTRACFLAGS instead of overwriting

When building with

$ HOST_EXTRACFLAGS=-g make

the expectation is that host tools are built with debug informations.
This however doesn't happen if the Makefile assigns a new value to the
HOST_EXTRACFLAGS instead of appending to it. So use += instead of := for
the first assignment.

Fixes: e3fd9b5384f3 ("scripts/dtc: consolidate include path options in Makefile")
Signed-off-by: Uwe Kleine-König <u.kleine-koenig@pengutronix.de>
Signed-off-by: Rob Herring <robh@kernel.org>

show more ...

kbuild: introduce hostprogs-always-y and userprogs-always-y

To build host programs, you need to add the program names to 'hostprogs'
to use the necessary build rule, but it is not enough to build th

kbuild: introduce hostprogs-always-y and userprogs-always-y

To build host programs, you need to add the program names to 'hostprogs'
to use the necessary build rule, but it is not enough to build them
because there is no dependency.

There are two types of host programs: built as the prerequisite of
another (e.g. gen_crc32table in lib/Makefile), or always built when
Kbuild visits the Makefile (e.g. genksyms in scripts/genksyms/Makefile).

The latter is typical in Makefiles under scripts/, which contains host
programs globally used during the kernel build. To build them, you need
to add them to both 'hostprogs' and 'always-y'.

This commit adds hostprogs-always-y as a shorthand.

The same applies to user programs. net/bpfilter/Makefile builds
bpfilter_umh on demand, hence always-y is unneeded. In contrast,
programs under samples/ are added to both 'userprogs' and 'always-y'
so they are always built when Kbuild visits the Makefiles.

userprogs-always-y works as a shorthand.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Acked-by: Miguel Ojeda <miguel.ojeda.sandonis@gmail.com>

show more ...

scripts/dtc: use pkg-config to include <yaml.h> in non-standard path

Commit 067c650c456e ("dtc: Use pkg-config to locate libyaml") added
'pkg-config --libs' to link libyaml installed in a non-standa

scripts/dtc: use pkg-config to include <yaml.h> in non-standard path

Commit 067c650c456e ("dtc: Use pkg-config to locate libyaml") added
'pkg-config --libs' to link libyaml installed in a non-standard
location.

yamltree.c includes <yaml.h>, but that commit did not add the search
path for <yaml.h>. If /usr/include/yaml.h does not exist, it fails to
build. A user can explicitly pass HOSTCFLAGS to work around it, but
the policy is not consistent.

There are two ways to deal with libraries in a non-default location.

[1] Use HOSTCFLAGS and HOSTLDFLAGS for additional search paths for
headers and libraries.
They are documented in Documentation/kbuild/kbuild.rst

$ make HOSTCFLAGS='-I <prefix>/include' HOSTLDFLAGS='-L <prefix>/lib'

[2] Use pkg-config

'pkg-config --cflags' for querying the header search path
'pkg-config --libs' for querying the lib and its path

If we go with pkg-config, use [2] consistently. Do not mix up
[1] and [2].

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Signed-off-by: Rob Herring <robh@kernel.org>

show more ...

kbuild: check libyaml installation for 'make dt_binding_check'

If you run 'make dtbs_check' without installing the libyaml package,
the error message "dtc needs libyaml ..." is shown.

This should b

kbuild: check libyaml installation for 'make dt_binding_check'

If you run 'make dtbs_check' without installing the libyaml package,
the error message "dtc needs libyaml ..." is shown.

This should be checked also for 'make dt_binding_check' because dtc
needs to validate *.example.dts extracted from *.yaml files.

It is missing since commit 4f0e3a57d6eb ("kbuild: Add support for DT
binding schema checks"), but this fix-up is applicable only after commit
e10c4321dc1e ("kbuild: allow to run dt_binding_check and dtbs_check
in a single command").

I gave the Fixes tag to the latter in case somebody is interested in
back-porting this.

Fixes: e10c4321dc1e ("kbuild: allow to run dt_binding_check and dtbs_check in a single command")
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Signed-off-by: Rob Herring <robh@kernel.org>

show more ...

kbuild: allow to run dt_binding_check without kernel configuration

The dt_binding_check target is located outside of the
'ifneq ($(dtstree),) ... endif' block.

So, you can run 'make dt_binding_chec

kbuild: allow to run dt_binding_check without kernel configuration

The dt_binding_check target is located outside of the
'ifneq ($(dtstree),) ... endif' block.

So, you can run 'make dt_binding_check' on any architecture.
This makes a perfect sense because the dt-schema is arch-agnostic.

The only one problem I see is that scripts/dtc/dtc is not always built.
For example, ARCH=x86 defconfig does not define CONFIG_DTC. Kbuild
descends into scripts/dtc/ with doing nothing. Then, it fails to build
*.example.dt.yaml files.

Let's build scripts/dtc/dtc forcibly when running dt_binding_check.

The dt-schema does not depend on any CONFIG option either, so you
should be able to run dt_binding_check without the .config file.

Going forward, you can directly run 'make dt_binding_check' in a
pristine source tree.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Rob Herring <robh@kernel.org>

show more ...

kbuild: rename hostprogs-y/always to hostprogs/always-y

In old days, the "host-progs" syntax was used for specifying host
programs. It was renamed to the current "hostprogs-y" in 2004.

It is typica

kbuild: rename hostprogs-y/always to hostprogs/always-y

In old days, the "host-progs" syntax was used for specifying host
programs. It was renamed to the current "hostprogs-y" in 2004.

It is typically useful in scripts/Makefile because it allows Kbuild to
selectively compile host programs based on the kernel configuration.

This commit renames like follows:

always -> always-y
hostprogs-y -> hostprogs

So, scripts/Makefile will look like this:

always-$(CONFIG_BUILD_BIN2C) += ...
always-$(CONFIG_KALLSYMS) += ...
...
hostprogs := $(always-y) $(always-m)

I think this makes more sense because a host program is always a host
program, irrespective of the kernel configuration. We want to specify
which ones to compile by CONFIG options, so always-y will be handier.

The "always", "hostprogs-y", "hostprogs-m" will be kept for backward
compatibility for a while.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

show more ...

dtc: Use pkg-config to locate libyaml

Using Makefile's wildcard with absolute path to detect
the presence of libyaml results in false-positive
detection when cross-compiling e.g. in yocto environmen

dtc: Use pkg-config to locate libyaml

Using Makefile's wildcard with absolute path to detect
the presence of libyaml results in false-positive
detection when cross-compiling e.g. in yocto environment.
The latter results in build error:
| scripts/dtc/yamltree.o: In function `yaml_propval_int':
| yamltree.c: undefined reference to `yaml_sequence_start_event_initialize'
| yamltree.c: undefined reference to `yaml_emitter_emit'
| yamltree.c: undefined reference to `yaml_scalar_event_initialize'
...
Use pkg-config to locate libyaml to address this scenario.

Signed-off-by: Pavel Modilaynen <pavel.modilaynen@axis.com>
[robh: silence stderr]
Signed-off-by: Rob Herring <robh@kernel.org>

show more ...

treewide: prefix header search paths with $(srctree)/

Currently, the Kbuild core manipulates header search paths in a crazy
way [1].

To fix this mess, I want all Makefiles to add explicit $(srctree

treewide: prefix header search paths with $(srctree)/

Currently, the Kbuild core manipulates header search paths in a crazy
way [1].

To fix this mess, I want all Makefiles to add explicit $(srctree)/ to
the search paths in the srctree. Some Makefiles are already written in
that way, but not all. The goal of this work is to make the notation
consistent, and finally get rid of the gross hacks.

Having whitespaces after -I does not matter since commit 48f6e3cf5bc6
("kbuild: do not drop -I without parameter").

[1]: https://patchwork.kernel.org/patch/9632347/

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>

show more ...

kbuild: Add support for DT binding schema checks

This adds the build infrastructure for checking DT binding schema
documents and validating dts files using the binding schema.

Check DT binding sche

kbuild: Add support for DT binding schema checks

This adds the build infrastructure for checking DT binding schema
documents and validating dts files using the binding schema.

Check DT binding schema documents:
make dt_binding_check

Build dts files and check using DT binding schema:
make dtbs_check

Optionally, DT_SCHEMA_FILES can be passed in with a schema file(s) to
use for validation. This makes it easier to find and fix errors
generated by a specific schema.

Currently, the validation targets are separate from a normal build to
avoid a hard dependency on the external DT schema project and because
there are lots of warnings generated.

Cc: Jonathan Corbet <corbet@lwn.net>
Cc: Mark Rutland <mark.rutland@arm.com>
Acked-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Cc: Michal Marek <michal.lkml@markovi.net>
Cc: linux-doc@vger.kernel.org
Cc: devicetree@vger.kernel.org
Cc: linux-kbuild@vger.kernel.org
Signed-off-by: Rob Herring <robh@kernel.org>

show more ...

kbuild: consolidate Devicetree dtb build rules

There is nothing arch specific about building dtb files other than their
location under /arch/*/boot/dts/. Keeping each arch aligned is a pain.
The dep

kbuild: consolidate Devicetree dtb build rules

There is nothing arch specific about building dtb files other than their
location under /arch/*/boot/dts/. Keeping each arch aligned is a pain.
The dependencies and supported targets are all slightly different.
Also, a cross-compiler for each arch is needed, but really the host
compiler preprocessor is perfectly fine for building dtbs. Move the
build rules to a common location and remove the arch specific ones. This
is done in a single step to avoid warnings about overriding rules.

The build dependencies had been a mixture of 'scripts' and/or 'prepare'.
These pull in several dependencies some of which need a target compiler
(specifically devicetable-offsets.h) and aren't needed to build dtbs.
All that is really needed is dtc, so adjust the dependencies to only be
dtc.

This change enables support 'dtbs_install' on some arches which were
missing the target.

Acked-by: Will Deacon <will.deacon@arm.com>
Acked-by: Paul Burton <paul.burton@mips.com>
Acked-by: Ley Foon Tan <ley.foon.tan@intel.com>
Acked-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Cc: Michal Marek <michal.lkml@markovi.net>
Cc: Vineet Gupta <vgupta@synopsys.com>
Cc: Russell King <linux@armlinux.org.uk>
Cc: Catalin Marinas <catalin.marinas@arm.com>
Cc: Yoshinori Sato <ysato@users.sourceforge.jp>
Cc: Michal Simek <monstr@monstr.eu>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: James Hogan <jhogan@kernel.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Michael Ellerman <mpe@ellerman.id.au>
Cc: Chris Zankel <chris@zankel.net>
Cc: Max Filippov <jcmvbkbc@gmail.com>
Cc: linux-kbuild@vger.kernel.org
Cc: linux-snps-arc@lists.infradead.org
Cc: linux-arm-kernel@lists.infradead.org
Cc: uclinux-h8-devel@lists.sourceforge.jp
Cc: linux-mips@linux-mips.org
Cc: nios2-dev@lists.rocketboards.org
Cc: linuxppc-dev@lists.ozlabs.org
Cc: linux-xtensa@linux-xtensa.org
Signed-off-by: Rob Herring <robh@kernel.org>

show more ...

scripts/dtc: Update to upstream version v1.4.7-14-gc86da84d30e4

Major changes are I2C and SPI bus checks, YAML output format (for
future validation), some new libfdt functions, and more libfdt
valid

scripts/dtc: Update to upstream version v1.4.7-14-gc86da84d30e4

Major changes are I2C and SPI bus checks, YAML output format (for
future validation), some new libfdt functions, and more libfdt
validation of dtbs.

The YAML addition adds an optional dependency on libyaml. pkg-config is
used to test for it and pkg-config became a kconfig dependency in 4.18.

This adds the following commits from upstream:

c86da84d30e4 Add support for YAML encoded output
361b5e7d8067 Make type_marker_length helper public
bfbfab047e45 pylibfdt: Add a means to add and delete notes
9005f4108e7c pylibfdt: Allow delprop() to return errors
b94c056b137e Make valgrind optional
fd06c54d4711 tests: Better testing of dtc -I fs mode
c3f50c9a86d9 tests: Allow dtbs_equal_unordered to ignore mem reserves
0ac9fdee37c7 dtc: trivial '-I fs -O dts' test
0fd1c8c783f3 pylibfdt: fdt_get_mem_rsv returns 2 uint64_t values
04853cad18f4 pylibfdt: Don't incorrectly / unnecessarily override uint64_t typemap
9619c8619c37 Kill bogus TYPE_BLOB marker type
ac68ff92ae20 parser: add TYPE_STRING marker to path references
90a190eb04d9 checks: add SPI bus checks
53a1bd546905 checks: add I2C bus checks
88f18909db73 dtc: Bump version to v1.4.7
85bce8b2f06d tests: Correction to vg_prepare_blob()
57f7f9e7bc7c tests: Don't call memcmp() with NULL arguments
c12b2b0c20eb libfdt: fdt_address_cells() and fdt_size_cells()
3fe0eeda0b7f livetree: Set phandle properties type to uint32
853649acceba pylibfdt: Support the sequential-write interface
9b0e4fe26093 tests: Improve fdt_resize() tests
1087504bb3e8 libfdt: Add necessary header padding in fdt_create()
c72fa777e613 libfdt: Copy the struct region in fdt_resize()
32b9c6130762 Preserve datatype markers when emitting dts format
6dcb8ba408ec libfdt: Add helpers for accessing unaligned words
42607f21d43e tests: Fix incorrect check name 'prop_name_chars'
9d78c33bf8a1 tests: fix grep for checks error messages
b770f3d1c13f pylibfdt: Support setting the name of a node
2f0d07e678e0 pylibfdt: Add functions to set and get properties as strings
354d3dc55939 pylibfdt: Update the bytearray size with pack()
3c374d46acce pylibfdt: Allow reading integer values from properties
49d32ce40bb4 pylibfdt: Use an unsigned type for fdt32_t
481246a0c13a pylibfdt: Avoid accessing the internal _fdt member in tests
9aafa33d99ed pylibfdt: Add functions to update properties
5a598671fdbf pylibfdt: Support device-tree creation/expansion
483e170625e1 pylibfdt: Add support for reading the memory reserve map
29bb05aa4200 pylibfdt: Add support for the rest of the header functions
582a7159a5d0 pylibfdt: Add support for fdt_next_node()
f0f8c9169819 pylibfdt: Reorder functions to match libfdt.h
64a69d123935 pylibfdt: Return string instead of bytearray from getprop()
4d09a83420df fdtput: Add documentation
e617cbe1bd67 fdtget: Add documentation
180a93924014 Use <inttypes.h> format specifiers in a bunch of places we should
b9af3b396576 scripts/dtc: Fixed format mismatch in fprintf
4b8fcc3d015c libfdt: Add fdt_check_full() function
c14223fb2292 tests: Use valgrind client requests for better checking
5b67d2b955a3 tests: Better handling of valgrind errors saving blobs
e2556aaeb506 tests: Remove unused #define
fb9c6abddaa8 Use size_t for blob lengths in utilfdt_read*
0112fda03bf6 libfdt: Add fdt_header_size()
6473a21d8bfe Consolidate utilfdt_read_len() variants
d5db5382c5e5 libfdt: Safer access to memory reservations
719d582e98ec libfdt: Propagate name errors in fdt_getprop_by_offset()
70166d62a27f libfdt: Safer access to strings section
eb890c0f77dc libfdt: Make fdt_check_header() more thorough
899d6fad93f3 libfdt: Improve sequential write state checking
04b5b4062ccd libfdt: Clean up header checking functions
44d3efedc816 Preserve datatype information when parsing dts
f0be81bd8de0 Make Property a subclass of bytearray
24b1f3f064d4 pylibfdt: Add a method to access the device tree directly

Signed-off-by: Rob Herring <robh@kernel.org>

show more ...

scripts/dtc: consolidate include path options in Makefile

It is tedious to specify extra compiler options for every file.
HOST_EXTRACFLAGS is useful to add options to all files in a
directory.

-I$(

scripts/dtc: consolidate include path options in Makefile

It is tedious to specify extra compiler options for every file.
HOST_EXTRACFLAGS is useful to add options to all files in a
directory.

-I$(src)/libfdt is needed for all the files in this directory
to include libfdt_env.h etc. from scripts/dtc/libfdt/.

On the other hand, -I$(src) is used to include check-in headers
from generated C files. Thus, I added it only to dtc-lexer.lex.o
and dtc-parser.tab.o .

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Reviewed-by: Rob Herring <robh@kernel.org>

show more ...

kbuild: add %.lex.c and %.tab.[ch] to 'targets' automatically

Files generated by if_changed* must be added to 'targets' to include
*.cmd files. Otherwise, they would be regenerated every time.

The

kbuild: add %.lex.c and %.tab.[ch] to 'targets' automatically

Files generated by if_changed* must be added to 'targets' to include
*.cmd files. Otherwise, they would be regenerated every time.

The build system automatically adds objects to 'targets' where
appropriate, such as obj-y, extra-y, etc. but does nothing for
intermediate files. So, each Makefile needs to add them by itself.

There are some common cases where objects are generated by chained
rules. Lexers and parsers are compiled like follows:

%.lex.o <- %.lex.c <- %.l
%.tab.o <- %.tab.c <- %.y

They are common patterns, so it is reasonable to take care of them
in the core Makefile instead of requiring each Makefile to do so.

At this moment, you cannot delete 'target += zconf.lex.c' in the
Kconfig Makefile because zconf.lex.c is included from zconf.tab.c
instead of being compiled separately. It should be deleted after
Kconfig is more refactored.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Frank Rowand <frowand.list@gmail.com>

show more ...

kbuild: clean up *.lex.c and *.tab.[ch] patterns from top-level Makefile

Files suffixed by .lex.c, .tab.[ch] are generated lexers, parsers,
respectively. Clean them up globally from the top Makefil

kbuild: clean up *.lex.c and *.tab.[ch] patterns from top-level Makefile

Files suffixed by .lex.c, .tab.[ch] are generated lexers, parsers,
respectively. Clean them up globally from the top Makefile.

Some of the final host programs those lexer/parser are linked into
are necessary for building external modules, but the intermediates
are unneeded. They can be cleaned away by 'make clean' instead of
'make mrproper'.

Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Acked-by: Frank Rowand <frowand.list@gmail.com>

show more ...

scripts/dtc: generate lexer and parser during build instead of shipping

Now that the kernel build supports flex and bison, remove the _shipped
files and generate them during the build instead.

Base

scripts/dtc: generate lexer and parser during build instead of shipping

Now that the kernel build supports flex and bison, remove the _shipped
files and generate them during the build instead.

Based on Masahiro's original patch.

Reviewed-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Signed-off-by: Rob Herring <robh@kernel.org>

show more ...

License cleanup: add SPDX GPL-2.0 license identifier to files with no license

Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine

License cleanup: add SPDX GPL-2.0 license identifier to files with no license

Many source files in the tree are missing licensing information, which
makes it harder for compliance tools to determine the correct license.

By default all files without license information are under the default
license of the kernel, which is GPL version 2.

Update the files which contain no license information with the 'GPL-2.0'
SPDX license identifier. The SPDX identifier is a legally binding
shorthand, which can be used instead of the full boiler plate text.

This patch is based on work done by Thomas Gleixner and Kate Stewart and
Philippe Ombredanne.

How this work was done:

Patches were generated and checked against linux-4.14-rc6 for a subset of
the use cases:
- file had no licensing information it it.
- file was a */uapi/* one with no licensing information in it,
- file was a */uapi/* one with existing licensing information,

Further patches will be generated in subsequent months to fix up cases
where non-standard license headers were used, and references to license
had to be inferred by heuristics based on keywords.

The analysis to determine which SPDX License Identifier to be applied to
a file was done in a spreadsheet of side by side results from of the
output of two independent scanners (ScanCode & Windriver) producing SPDX
tag:value files created by Philippe Ombredanne. Philippe prepared the
base worksheet, and did an initial spot review of a few 1000 files.

The 4.13 kernel was the starting point of the analysis with 60,537 files
assessed. Kate Stewart did a file by file comparison of the scanner
results in the spreadsheet to determine which SPDX license identifier(s)
to be applied to the file. She confirmed any determination that was not
immediately clear with lawyers working with the Linux Foundation.

Criteria used to select files for SPDX license identifier tagging was:
- Files considered eligible had to be source code files.
- Make and config files were included as candidates if they contained >5
lines of source
- File already had some variant of a license header in it (even if <5
lines).

All documentation files were explicitly excluded.

The following heuristics were used to determine which SPDX license
identifiers to apply.

- when both scanners couldn't find any license traces, file was
considered to have no license information in it, and the top level
COPYING file license applied.

For non */uapi/* files that summary was:

SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 11139

and resulted in the first patch in this series.

If that file was a */uapi/* path one, it was "GPL-2.0 WITH
Linux-syscall-note" otherwise it was "GPL-2.0". Results of that was:

SPDX license identifier # files
---------------------------------------------------|-------
GPL-2.0 WITH Linux-syscall-note 930

and resulted in the second patch in this series.

- if a file had some form of licensing information in it, and was one
of the */uapi/* ones, it was denoted with the Linux-syscall-note if
any GPL family license was found in the file or had no licensing in
it (per prior point). Results summary:

SPDX license identifier # files
---------------------------------------------------|------
GPL-2.0 WITH Linux-syscall-note 270
GPL-2.0+ WITH Linux-syscall-note 169
((GPL-2.0 WITH Linux-syscall-note) OR BSD-2-Clause) 21
((GPL-2.0 WITH Linux-syscall-note) OR BSD-3-Clause) 17
LGPL-2.1+ WITH Linux-syscall-note 15
GPL-1.0+ WITH Linux-syscall-note 14
((GPL-2.0+ WITH Linux-syscall-note) OR BSD-3-Clause) 5
LGPL-2.0+ WITH Linux-syscall-note 4
LGPL-2.1 WITH Linux-syscall-note 3
((GPL-2.0 WITH Linux-syscall-note) OR MIT) 3
((GPL-2.0 WITH Linux-syscall-note) AND MIT) 1

and that resulted in the third patch in this series.

- when the two scanners agreed on the detected license(s), that became
the concluded license(s).

- when there was disagreement between the two scanners (one detected a
license but the other didn't, or they both detected different
licenses) a manual inspection of the file occurred.

- In most cases a manual inspection of the information in the file
resulted in a clear resolution of the license that should apply (and
which scanner probably needed to revisit its heuristics).

- When it was not immediately clear, the license identifier was
confirmed with lawyers working with the Linux Foundation.

- If there was any question as to the appropriate license identifier,
the file was flagged for further research and to be revisited later
in time.

In total, over 70 hours of logged manual review was done on the
spreadsheet to determine the SPDX license identifiers to apply to the
source files by Kate, Philippe, Thomas and, in some cases, confirmation
by lawyers working with the Linux Foundation.

Kate also obtained a third independent scan of the 4.13 code base from
FOSSology, and compared selected files where the other two scanners
disagreed against that SPDX file, to see if there was new insights. The
Windriver scanner is based on an older version of FOSSology in part, so
they are related.

Thomas did random spot checks in about 500 files from the spreadsheets
for the uapi headers and agreed with SPDX license identifier in the
files he inspected. For the non-uapi files Thomas did random spot checks
in about 15000 files.

In initial set of patches against 4.14-rc6, 3 files were found to have
copy/paste license identifier errors, and have been fixed to reflect the
correct identifier.

Additionally Philippe spent 10 hours this week doing a detailed manual
inspection and review of the 12,461 patched files from the initial patch
version early this week with:
- a full scancode scan run, collecting the matched texts, detected
license ids and scores
- reviewing anything where there was a license detected (about 500+
files) to ensure that the applied SPDX license was correct
- reviewing anything where there was no detection but the patch license
was not GPL-2.0 WITH Linux-syscall-note to ensure that the applied
SPDX license was correct

This produced a worksheet with 20 files needing minor correction. This
worksheet was then exported into 3 different .csv files for the
different types of files to be modified.

These .csv files were then reviewed by Greg. Thomas wrote a script to
parse the csv files and add the proper SPDX tag to the file, in the
format that the file expected. This script was further refined by Greg
based on the output to detect more types of files automatically and to
distinguish between header and source .c files (which need different
comment types.) Finally Greg ran the script using the .csv files to
generate the patches.

Reviewed-by: Kate Stewart <kstewart@linuxfoundation.org>
Reviewed-by: Philippe Ombredanne <pombredanne@nexb.com>
Reviewed-by: Thomas Gleixner <tglx@linutronix.de>
Signed-off-by: Greg Kroah-Hartman <gregkh@linuxfoundation.org>

show more ...

script: dtc: clean generated files

Fix "make distclean" to clean up generated dtc files.

Without this patch the following files are left around:
- dtc-lexer.lex.c
- dtc-parser.tab.c
- dtc-parser

script: dtc: clean generated files

Fix "make distclean" to clean up generated dtc files.

Without this patch the following files are left around:
- dtc-lexer.lex.c
- dtc-parser.tab.c
- dtc-parser.tab.h

Signed-off-by: Magnus Damm <damm@opensource.se>
Reviewed-by: Simon Horman <horms@verge.net.au>
Signed-off-by: Grant Likely <grant.likely@secretlab.ca>

show more ...

dtc: migrate parser to implicit rules

Cc: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Arnaud Lacombe <lacombar@gmail.com>

of/flattree: Update dtc to current mainline.

Pull in recent changes from the main dtc repository. These changes
primarily allow multiple device trees to be declared which are merged
by dtc. This fea

of/flattree: Update dtc to current mainline.

Pull in recent changes from the main dtc repository. These changes
primarily allow multiple device trees to be declared which are merged
by dtc. This feature allows us to include a basic dts file and then
provide more information for the specific system through the merging
functionality.

Changes pulled from git://git.jdl.com/software/dtc.git
commit id: 37c0b6a0, "dtc: Add code to make diffing trees easier"

Signed-off-by: John Bonesio <bones@secretlab.ca>
Signed-off-by: Grant Likely <grant.likely@secretlab.ca>

show more ...

Move dtc and libfdt sources from arch/powerpc/boot to scripts/dtc

The powerpc kernel always requires an Open Firmware like device tree
to supply device information. On systems without OF, this come

Move dtc and libfdt sources from arch/powerpc/boot to scripts/dtc

The powerpc kernel always requires an Open Firmware like device tree
to supply device information. On systems without OF, this comes from
a flattened device tree blob. This blob is usually generated by dtc,
a tool which compiles a text description of the device tree into the
flattened format used by the kernel. Sometimes, the bootwrapper makes
small changes to the pre-compiled device tree blob (e.g. filling in
the size of RAM). To do this it uses the libfdt library.

Because these are only used on powerpc, the code for both these tools
is included under arch/powerpc/boot (these were imported and are
periodically updated from the upstream dtc tree).

However, the microblaze architecture, currently being prepared for
merging to mainline also uses dtc to produce device tree blobs. A few
other archs have also mentioned some interest in using dtc.
Therefore, this patch moves dtc and libfdt from arch/powerpc into
scripts, where it can be used by any architecture.

The vast bulk of this patch is a literal move, the rest is adjusting
the various Makefiles to use dtc and libfdt correctly from their new
locations.

Signed-off-by: David Gibson <david@gibson.dropbear.id.au>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>

show more ...