With the previous changes, $(extmod_prefix), $(MODORDER), and
$(MODULES_NSDEPS) are constant. (empty, modules.order, and
modules.nsdeps, respectively).
Remove these variables and hard-code their values.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>
Currently, Kbuild always operates in the output directory of the kernel,
even when building external modules. This increases the risk of external
module Makefiles attempting to write to the kernel directory.
This commit switches the working directory to the external module
directory, allowing the removal of the $(KBUILD_EXTMOD)/ prefix from
some build artifacts.
The command for building external modules maintains backward
compatibility, but Makefiles that rely on working in the kernel
directory may break. In such cases, $(objtree) and $(srctree) should
be used to refer to the output and source directories of the kernel.
The appearance of the build log will change as follows:
[Before]
$ make -C /path/to/my/linux M=/path/to/my/externel/module
make: Entering directory '/path/to/my/linux'
CC [M] /path/to/my/externel/module/helloworld.o
MODPOST /path/to/my/externel/module/Module.symvers
CC [M] /path/to/my/externel/module/helloworld.mod.o
CC [M] /path/to/my/externel/module/.module-common.o
LD [M] /path/to/my/externel/module/helloworld.ko
make: Leaving directory '/path/to/my/linux'
[After]
$ make -C /path/to/my/linux M=/path/to/my/externel/module
make: Entering directory '/path/to/my/linux'
make[1]: Entering directory '/path/to/my/externel/module'
CC [M] helloworld.o
MODPOST Module.symvers
CC [M] helloworld.mod.o
CC [M] .module-common.o
LD [M] helloworld.ko
make[1]: Leaving directory '/path/to/my/externel/module'
make: Leaving directory '/path/to/my/linux'
Printing "Entering directory" twice is cumbersome. This will be
addressed later.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nicolas Schier <n.schier@avm.de>
$(objtree) refers to the top of the output directory of kernel builds.
This commit adds the explicit $(objtree)/ prefix to build artifacts
needed for building external modules.
This change has no immediate impact, as the top-level Makefile
currently defines:
objtree := .
This commit prepares for supporting the building of external modules
in a different directory.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>
- Avoid 'constexpr', which is a keyword in C23
- Allow 'dtbs_check' and 'dt_compatible_check' run independently of
'dt_binding_check'
- Fix weak references to avoid GOT entries in position-independent
code generation
- Convert the last use of 'optional' property in arch/sh/Kconfig
- Remove support for the 'optional' property in Kconfig
- Remove support for Clang's ThinLTO caching, which does not work with
the .incbin directive
- Change the semantics of $(src) so it always points to the source
directory, which fixes Makefile inconsistencies between upstream and
downstream
- Fix 'make tar-pkg' for RISC-V to produce a consistent package
- Provide reasonable default coverage for objtool, sanitizers, and
profilers
- Remove redundant OBJECT_FILES_NON_STANDARD, KASAN_SANITIZE, etc.
- Remove the last use of tristate choice in drivers/rapidio/Kconfig
- Various cleanups and fixes in Kconfig
-----BEGIN PGP SIGNATURE-----
iQJJBAABCgAzFiEEbmPs18K1szRHjPqEPYsBB53g2wYFAmZFlGcVHG1hc2FoaXJv
eUBrZXJuZWwub3JnAAoJED2LAQed4NsG8voQALC8NtFpduWVfLRj2Qg6Ll/xf1vX
2igcTJEOFHkeqXLGoT8dTDKLEipUBUvKyguPq66CGwVTe2g6zy/nUSXeVtFrUsIa
msLTi8FqhqUo5lodNvGMRf8qqmuqcvnXoiQwIocF92jtsFy14bhiFY+n4HfcFNjj
GOKwqBZYQUwY/VVb090efc7RfS9c7uwABJSBelSoxg3AGZriwjGy7Pw5aSKGgVYi
inqL1eR6qwPP6z7CgQWM99soP+zwybFZmnQrsD9SniRBI4rtAat8Ih5jQFaSUFUQ
lk2w0NQBRFN88/uR2IJ2GWuIlQ74WeJ+QnCqVuQ59tV5zw90wqSmLzngfPD057Dv
JjNuhk0UyXVtpIg3lRtd4810ppNSTe33b9OM4O2H846W/crju5oDRNDHcflUXcwm
Rmn5ho1rb5QVzDVejJbgwidnUInSgJ9PZcvXQ/RJVZPhpgsBzAY9pQexG1G3hviw
y9UDrt6KP6bF9tHjmolmtdIes9Pj0c4dN6/Rdj4HS4hIQ/GDar0tnwvOvtfUctNL
orJlBsA6GeMmDVXKkR0ytOCWRYqWWbyt8g70RVKQJfuHX7/hGyAQPaQ2/u4mQhC2
aevYfbNJMj0VDfGz81HDBKFtkc5n+Ite8l157dHEl2LEabkOkRdNVcn7SNbOvZmd
ZCSnZ31h7woGfNho
=D5B/
-----END PGP SIGNATURE-----
Merge tag 'kbuild-v6.10' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild
Pull Kbuild updates from Masahiro Yamada:
- Avoid 'constexpr', which is a keyword in C23
- Allow 'dtbs_check' and 'dt_compatible_check' run independently of
'dt_binding_check'
- Fix weak references to avoid GOT entries in position-independent code
generation
- Convert the last use of 'optional' property in arch/sh/Kconfig
- Remove support for the 'optional' property in Kconfig
- Remove support for Clang's ThinLTO caching, which does not work with
the .incbin directive
- Change the semantics of $(src) so it always points to the source
directory, which fixes Makefile inconsistencies between upstream and
downstream
- Fix 'make tar-pkg' for RISC-V to produce a consistent package
- Provide reasonable default coverage for objtool, sanitizers, and
profilers
- Remove redundant OBJECT_FILES_NON_STANDARD, KASAN_SANITIZE, etc.
- Remove the last use of tristate choice in drivers/rapidio/Kconfig
- Various cleanups and fixes in Kconfig
* tag 'kbuild-v6.10' of git://git.kernel.org/pub/scm/linux/kernel/git/masahiroy/linux-kbuild: (46 commits)
kconfig: use sym_get_choice_menu() in sym_check_prop()
rapidio: remove choice for enumeration
kconfig: lxdialog: remove initialization with A_NORMAL
kconfig: m/nconf: merge two item_add_str() calls
kconfig: m/nconf: remove dead code to display value of bool choice
kconfig: m/nconf: remove dead code to display children of choice members
kconfig: gconf: show checkbox for choice correctly
kbuild: use GCOV_PROFILE and KCSAN_SANITIZE in scripts/Makefile.modfinal
Makefile: remove redundant tool coverage variables
kbuild: provide reasonable defaults for tool coverage
modules: Drop the .export_symbol section from the final modules
kconfig: use menu_list_for_each_sym() in sym_check_choice_deps()
kconfig: use sym_get_choice_menu() in conf_write_defconfig()
kconfig: add sym_get_choice_menu() helper
kconfig: turn defaults and additional prompt for choice members into error
kconfig: turn missing prompt for choice members into error
kconfig: turn conf_choice() into void function
kconfig: use linked list in sym_set_changed()
kconfig: gconf: use MENU_CHANGED instead of SYMBOL_CHANGED
kconfig: gconf: remove debug code
...
If UNUSED_KSYMS_WHITELIST is a file generated
before Kbuild runs, and the source tree is in
a read-only filesystem, the developer must put
the file somewhere and specify an absolute
path to UNUSED_KSYMS_WHITELIST. This worked,
but if IKCONFIG=y, an absolute path is embedded
into .config and eventually into vmlinux, causing
the build to be less reproducible when building
on a different machine.
This patch makes the handling of
UNUSED_KSYMS_WHITELIST to be similar to
MODULE_SIG_KEY.
First, check if UNUSED_KSYMS_WHITELIST is an
absolute path, just as before this patch. If so,
use the path as is.
If it is a relative path, use wildcard to check
the existence of the file below objtree first.
If it does not exist, fall back to the original
behavior of adding $(srctree)/ before the value.
After this patch, the developer can put the generated
file in objtree, then use a relative path against
objtree in .config, eradicating any absolute paths
that may be evaluated differently on different machines.
Signed-off-by: Yifan Hong <elsk@google.com>
Reviewed-by: Elliot Berman <quic_eberman@quicinc.com>
Signed-off-by: Luis Chamberlain <mcgrof@kernel.org>
Kbuild conventionally uses $(obj)/ for generated files, and $(src)/ for
checked-in source files. It is merely a convention without any functional
difference. In fact, $(obj) and $(src) are exactly the same, as defined
in scripts/Makefile.build:
src := $(obj)
When the kernel is built in a separate output directory, $(src) does
not accurately reflect the source directory location. While Kbuild
resolves this discrepancy by specifying VPATH=$(srctree) to search for
source files, it does not cover all cases. For example, when adding a
header search path for local headers, -I$(srctree)/$(src) is typically
passed to the compiler.
This introduces inconsistency between upstream and downstream Makefiles
because $(src) is used instead of $(srctree)/$(src) for the latter.
To address this inconsistency, this commit changes the semantics of
$(src) so that it always points to the directory in the source tree.
Going forward, the variables used in Makefiles will have the following
meanings:
$(obj) - directory in the object tree
$(src) - directory in the source tree (changed by this commit)
$(objtree) - the top of the kernel object tree
$(srctree) - the top of the kernel source tree
Consequently, $(srctree)/$(src) in upstream Makefiles need to be replaced
with $(src).
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>
Currently, all files with EXPORT_SYMBOL() are rebuilt when CONFIG_MODULES
is flipped due to <linux/export.h> depending on CONFIG_MODULES.
Now that modpost can make a final decision about export symbols,
<linux/export.h> does not need to make EXPORT_SYMBOL() no-op.
Instead, modpost can skip emitting KSYMTAB when CONFIG_MODULES is unset.
This commit will reduce the number of recompilation when CONFIG_MODULES
is toggled.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
When CONFIG_TRIM_UNUSED_KSYMS is enabled, Kbuild recursively traverses
the directory tree to determine which EXPORT_SYMBOL to trim. If an
EXPORT_SYMBOL turns out to be unused by anyone, Kbuild begins the
second traverse, where some source files are recompiled with their
EXPORT_SYMBOL() tuned into a no-op.
Linus stated negative opinions about this slowness in commits:
- 5cf0fd591f ("Kbuild: disable TRIM_UNUSED_KSYMS option")
- a555bdd0c5 ("Kbuild: enable TRIM_UNUSED_KSYMS again, with some guarding")
We can do this better now. The final data structures of EXPORT_SYMBOL
are generated by the modpost stage, so modpost can selectively emit
KSYMTAB entries that are really used by modules.
Commit f73edc8951 ("kbuild: unify two modpost invocations") is another
ground-work to do this in a one-pass algorithm. With the list of modules,
modpost sets sym->used if it is used by a module. modpost emits KSYMTAB
only for symbols with sym->used==true.
BTW, Nicolas explained why the trimming was implemented with recursion:
https://lore.kernel.org/all/2o2rpn97-79nq-p7s2-nq5-8p83391473r@syhkavp.arg/
Actually, we never achieved that level of optimization where the chain
reaction of trimming comes into play because:
- CONFIG_LTO_CLANG cannot remove any unused symbols
- CONFIG_LD_DEAD_CODE_DATA_ELIMINATION is enabled only for vmlinux,
but not modules
If deeper trimming is required, we need to revisit this, but I guess
that is unlikely to happen.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
"No build warning" is a strong requirement these days, so you must fix
all issues before enabling a new warning flag.
We often add a new warning to W=1 first so that the kbuild test robot
blocks new breakages.
This commit allows modpost to show extra warnings only when W=1
(or KBUILD_EXTRA_WARN=1) is given.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
When there is a missing input file (vmlinux.o or Module.symvers), you
are likely to get a ton of unresolved symbols.
Currently, Kbuild automatically adds the -w option to allow module builds
to continue with warnings instead of errors.
This may not be what the user expects because it is generally more useful
to catch all possible issues at build time instead of at run time.
Let's not do what the user did not ask.
If you still want to build modules anyway, you can proceed by explicitly
setting KBUILD_MODPOST_WARN=1. Since you may miss a real issue, you need
to be aware of what you are doing.
Suggested-by: William McVicker <willmcvicker@google.com>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Tested-by: Will McVicker <willmcvicker@google.com>
Commit 63ffe00d8c ("kbuild: Fix running modpost with musl libc")
accidentally turned the unresolved symbol warnings into errors when
vmlinux.o (for in-tree builds) or Module.symver (for external module
builds) is missing.
In those cases, unresolved symbols are expected, but the -w option
is not set because 'missing-input' is referenced before set.
Move $(missing-input) back to the original place. This should be fine
for musl libc because vmlinux.o and -w are not added at the same time.
With this change, -w may be passed twice, but it is not a big deal.
Link: https://lore.kernel.org/all/b56a03b8-2a2a-f833-a5d2-cdc50a7ca2bb@cschramm.eu/
Fixes: 63ffe00d8c ("kbuild: Fix running modpost with musl libc")
Reported-by: Christopher Schramm <debian@cschramm.eu>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Tested-by: Samuel Holland <samuel@sholland.org>
commit 3d57e1b7b1 ("kbuild: refactor the prerequisites of the modpost
rule") moved 'vmlinux.o' inside modpost-args, possibly before some of
the other options. However, getopt() in musl libc follows POSIX and
stops looking for options upon reaching the first non-option argument.
As a result, the '-T' option is misinterpreted as a positional argument,
and the build fails:
make -f ./scripts/Makefile.modpost
scripts/mod/modpost -E -o Module.symvers vmlinux.o -T modules.order
-T: No such file or directory
make[1]: *** [scripts/Makefile.modpost:137: Module.symvers] Error 1
make: *** [Makefile:1960: modpost] Error 2
The fix is to move all options before 'vmlinux.o' in modpost-args.
Fixes: 3d57e1b7b1 ("kbuild: refactor the prerequisites of the modpost rule")
Signed-off-by: Samuel Holland <samuel@sholland.org>
Reviewed-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
The prerequisites of modpost are cluttered. The variables *-if-present
and *-if-needed are unreadable.
It is cleaner to append them into modpost-deps.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
scripts/Makefile.build replaces the suffix .o with .ko, then
scripts/Makefile.modpost calls the sed command to change .ko back
to the original .o suffix.
Instead of converting the suffixes back-and-forth, store the .o paths
in modules.order, and replace it with .ko in 'make modules_install'.
This avoids the unneeded sed command.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Luis Chamberlain <mcgrof@kernel.org>
While building, installing, cleaning, Kbuild visits sub-directories
and includes 'Kbuild' or 'Makefile' that exists there.
Add 'kbuild-file' macro, and reuse it from scripts/Makefie.*
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>
Reviewed-by: Alexander Lobakin <alobakin@pm.me>
Tested-by: Alexander Lobakin <alobakin@pm.me>
Commit f73edc8951 ("kbuild: unify two modpost invocations") introduced
a typo (moudle.symvers-if-present) which results in the kernel's
Module.symvers to not be included as a prerequisite for
$(KBUILD_EXTMOD)/Module.symvers. Fix the typo to restore the intended
functionality.
Fixes: f73edc8951 ("kbuild: unify two modpost invocations")
Signed-off-by: Will McVicker <willmcvicker@google.com>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
The musl implementation of getopt stops looking for options after the
first non-option argument. Put the options before the non-option
argument so environments using musl can still build the kernel and
modules.
Fixes: f73edc8951 ("kbuild: unify two modpost invocations")
Link: https://git.musl-libc.org/cgit/musl/tree/src/misc/getopt.c?h=dc9285ad1dc19349c407072cc48ba70dab86de45#n44
Signed-off-by: Richard Acayan <mailingradian@gmail.com>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
The objects placed at the head of vmlinux need special treatments:
- arch/$(SRCARCH)/Makefile adds them to head-y in order to place
them before other archives in the linker command line.
- arch/$(SRCARCH)/kernel/Makefile adds them to extra-y instead of
obj-y to avoid them going into built-in.a.
This commit gets rid of the latter.
Create vmlinux.a to collect all the objects that are unconditionally
linked to vmlinux. The objects listed in head-y are moved to the head
of vmlinux.a by using 'ar m'.
With this, arch/$(SRCARCH)/kernel/Makefile can consistently use obj-y
for builtin objects.
There is no *.o that is directly linked to vmlinux. Drop unneeded code
in scripts/clang-tools/gen_compile_commands.py.
$(AR) mPi needs 'T' to workaround the llvm-ar bug. The fix was suggested
by Nathan Chancellor [1].
[1]: https://lore.kernel.org/llvm/YyjjT5gQ2hGMH0ni@dev-arch.thelio-3990X/
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Tested-by: Nick Desaulniers <ndesaulniers@google.com>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>
Modpost generates .vmlinux.export.c and *.mod.c, which are prerequisites
of vmlinux and modules, respectively.
The modpost stage should be re-run when the modpost code is updated.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Currently, modpost is executed twice; first for vmlinux, second
for modules.
This commit merges them.
Current build flow
==================
1) build obj-y and obj-m objects
2) link vmlinux.o
3) modpost for vmlinux
4) link vmlinux
5) modpost for modules
6) link modules (*.ko)
The build steps 1) through 6) are serialized, that is, modules are
built after vmlinux. You do not get benefits of parallel builds when
scripts/link-vmlinux.sh is being run.
New build flow
==============
1) build obj-y and obj-m objects
2) link vmlinux.o
3) modpost for vmlinux and modules
4a) link vmlinux
4b) link modules (*.ko)
In the new build flow, modpost is invoked just once.
vmlinux and modules are built in parallel. One exception is
CONFIG_DEBUG_INFO_BTF_MODULES=y, where modules depend on vmlinux.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Tested-by: Nick Desaulniers <ndesaulniers@google.com>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>
.vmlinux.objs is used by modpost, so scripts/Makefile.modpost is
a better place to generate it.
It is used only when CONFIG_MODVERSIONS=y. It should be guarded
by "ifdef CONFIG_MODVERSIONS".
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Tested-by: Nick Desaulniers <ndesaulniers@google.com>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>
When CONFIG_LTO_CLANG=y, additional intermediate *.prelink.o is created
for each module. Also, objtool is postponed until LLVM IR is converted
to ELF.
CONFIG_X86_KERNEL_IBT works in a similar way to postpone objtool until
objects are merged together.
This commit stops generating *.prelink.o, so the build flow will look
similar with/without LTO.
The following figures show how the LTO build currently works, and
how this commit is changing it.
Current build flow
==================
[1] single-object module
$(LD)
$(CC) +objtool $(LD)
foo.c --------------------> foo.o -----> foo.prelink.o -----> foo.ko
(LLVM IR) (ELF) | (ELF)
|
foo.mod.o --/
(LLVM IR)
[2] multi-object module
$(LD)
$(CC) $(AR) +objtool $(LD)
foo1.c -----> foo1.o -----> foo.o -----> foo.prelink.o -----> foo.ko
| (archive) (ELF) | (ELF)
foo2.c -----> foo2.o --/ |
(LLVM IR) foo.mod.o --/
(LLVM IR)
One confusion is that foo.o in multi-object module is an archive
despite of its suffix.
New build flow
==============
[1] single-object module
Since there is only one object, there is no need to keep the LLVM IR.
Use $(CC)+$(LD) to generate an ELF object in one build rule. When LTO
is disabled, $(LD) is unneeded because $(CC) produces an ELF object.
$(CC)+$(LD)+objtool $(LD)
foo.c ----------------------------> foo.o ---------> foo.ko
(ELF) | (ELF)
|
foo.mod.o --/
(LLVM IR)
[2] multi-object module
Previously, $(AR) was used to combine LLVM IR files into an archive,
but there was no technical reason to do so. Use $(LD) to merge them
into a single ELF object.
$(LD)
$(CC) +objtool $(LD)
foo1.c ---------> foo1.o ---------> foo.o ---------> foo.ko
| (ELF) | (ELF)
foo2.c ---------> foo2.o ----/ |
(LLVM IR) foo.mod.o --/
(LLVM IR)
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>
Tested-by: Nathan Chancellor <nathan@kernel.org>
Reviewed-by: Sami Tolvanen <samitolvanen@google.com>
Tested-by: Sedat Dilek <sedat.dilek@gmail.com> # LLVM-14 (x86-64)
Acked-by: Josh Poimboeuf <jpoimboe@kernel.org>
Similar cleanup to commit 5c8166419a ("kbuild: replace $(if A,A,B)
with $(or A,B)").
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Reviewed-by: Nick Desaulniers <ndesaulniers@google.com>
When building an external module, if users don't need to separate the
compilation output and source code, they run the following command:
"make -C $(LINUX_SRC_DIR) M=$(PWD)". At this point, "$(KBUILD_EXTMOD)"
and "$(src)" are the same.
If they need to separate them, they run "make -C $(KERNEL_SRC_DIR)
O=$(KERNEL_OUT_DIR) M=$(OUT_DIR) src=$(PWD)". Before running the
command, they need to copy "Kbuild" or "Makefile" to "$(OUT_DIR)" to
prevent compilation failure.
So the kernel should change the included path to avoid the copy operation.
Signed-off-by: Jing Leng <jleng@ambarella.com>
[masahiro: I do not think "M=$(OUT_DIR) src=$(PWD)" is the official way,
but this patch is a nice clean up anyway.]
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Change comment "create one <module>.mod.c file pr. module"
to "create one <module>.mod.c file per module"
Signed-off-by: Ramji Jiyani <ramjiyani@google.com>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
With CONFIG_LTO_CLANG, we currently link modules into native
code just before modpost, which means with TRIM_UNUSED_KSYMS
enabled, we still look at the LLVM bitcode in the .o files when
generating the list of used symbols. As the bitcode doesn't
yet have calls to compiler intrinsics and llvm-nm doesn't see
function references that only exist in function-level inline
assembly, we currently need a whitelist for TRIM_UNUSED_KSYMS to
work with LTO.
This change moves module LTO linking to happen earlier, and
thus avoids the issue with LLVM bitcode and TRIM_UNUSED_KSYMS
entirely, allowing us to also drop the whitelist from
gen_autoksyms.sh.
Link: https://github.com/ClangBuiltLinux/linux/issues/1369
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Reviewed-by: Alexander Lobakin <alobakin@pm.me>
Tested-by: Alexander Lobakin <alobakin@pm.me>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Nathan reports that the mips defconfig emits the following warning:
WARNING: modpost: Symbol info of vmlinux is missing. Unresolved symbol check will be entirely skipped.
This false-positive happens when CONFIG_TRIM_UNUSED_KSYMS is enabled,
but no CONFIG option is set to 'm'.
Commit a0590473c5 ("nfs: fix PNFS_FLEXFILE_LAYOUT Kconfig default")
turned the last 'm' into 'y' for the mips defconfig, and uncovered
this issue.
In this case, the module feature itself is enabled, but we have no
module to build. As a result, CONFIG_TRIM_UNUSED_KSYMS drops all the
instances of EXPORT_SYMBOL. Then, modpost wrongly assumes vmlinux is
missing because vmlinux.symvers is empty. (As another false-positive
case, you can create a module that does not use any symbol of vmlinux).
The current behavior is to entirely suppress the unresolved symbol
warnings when vmlinux is missing just because there are too many.
I found the origin of this code in the historical git tree. [1]
If this is a matter of noisiness, I think modpost can display the
first 10 warnings, and the number of suppressed warnings at the end.
You will get a bit noisier logs when you run 'make modules' without
vmlinux, but such warnings are better to show because you never know
the resulting modules are actually loadable or not.
This commit changes the following:
- If any of input *.symver files is missing, pass -w option to let
the module build keep going with warnings instead of errors.
- If there are too many (10+) unresolved symbol warnings, show only
the first 10, and also the number of suppressed warnings.
[1]: https://git.kernel.org/pub/scm/linux/kernel/git/history/history.git/commit/?id=1cc0e0529569bf6a94f6d49770aa6d4b599d2c46
Reported-by: Nathan Chancellor <nathan@kernel.org>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
The -w option is meaningless for the first pass of modpost (vmlinux.o).
We know there are unresolved symbols in vmlinux.o, hence we skip
check_exports() and other checks when mod->is_vmlinux is set.
See the following part in the for-loop.
if (mod->is_vmlinux || mod->from_dump)
continue;
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
The external module build shows the following warning if Module.symvers
is missing in the kernel tree.
WARNING: Symbol version dump "Module.symvers" is missing.
Modules may not have dependencies or modversions.
I think this is an important heads-up because the resulting modules may
not work as expected. This happens when you did not build the entire
kernel tree, for example, you might have prepared the minimal setups
for external modules by 'make defconfig && make modules_preapre'.
A problem is that 'make modules' creates Module.symvers even without
vmlinux. In this case, that warning is suppressed since Module.symvers
already exists in spite of its incomplete content.
The incomplete (i.e. invalid) Module.symvers should not be created.
This commit changes the second pass of modpost to dump symbols into
modules-only.symvers. The final Module.symvers is created by
concatenating vmlinux.symvers and modules-only.symvers if both exist.
Module.symvers is supposed to collect symbols from both vmlinux and
modules. It might be a bit confusing, and I am not quite sure if it
is an official interface, but presumably it is difficult to rename it
because some tools (e.g. kmod) parse it.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
VPATH is used in Kbuild to make pattern rules search for prerequisites
in both $(objtree) and $(srctree). Some of *.c, *.S files are not real
sources, but generated by tools such as flex, bison, perl.
In contrast, I doubt the benefit of --include-dir=$(abs_srctree) because
it is always clear which Makefiles are real sources, and which are not.
So, my hope is to add $(srctree)/ prefix to all check-in Makefiles,
then remove --include-dir=$(abs_srctree) flag in the future.
I am touching only some Kbuild core parts for now. Treewide fixes will
be needed to achieve this goal.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
With CONFIG_MODVERSIONS, version information is linked into each
compilation unit that exports symbols. With LTO, we cannot use this
method as all C code is compiled into LLVM bitcode instead. This
change collects symbol versions into .symversions files and merges
them in link-vmlinux.sh where they are all linked into vmlinux.o at
the same time.
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20201211184633.3213045-4-samitolvanen@google.com
This change adds build system support for Clang's Link Time
Optimization (LTO). With -flto, instead of ELF object files, Clang
produces LLVM bitcode, which is compiled into native code at link
time, allowing the final binary to be optimized globally. For more
details, see:
https://llvm.org/docs/LinkTimeOptimization.html
The Kconfig option CONFIG_LTO_CLANG is implemented as a choice,
which defaults to LTO being disabled. To use LTO, the architecture
must select ARCH_SUPPORTS_LTO_CLANG and support:
- compiling with Clang,
- compiling all assembly code with Clang's integrated assembler,
- and linking with LLD.
While using CONFIG_LTO_CLANG_FULL results in the best runtime
performance, the compilation is not scalable in time or
memory. CONFIG_LTO_CLANG_THIN enables ThinLTO, which allows
parallel optimization and faster incremental builds. ThinLTO is
used by default if the architecture also selects
ARCH_SUPPORTS_LTO_CLANG_THIN:
https://clang.llvm.org/docs/ThinLTO.html
To enable LTO, LLVM tools must be used to handle bitcode files, by
passing LLVM=1 and LLVM_IAS=1 options to make:
$ make LLVM=1 LLVM_IAS=1 defconfig
$ scripts/config -e LTO_CLANG_THIN
$ make LLVM=1 LLVM_IAS=1
To prepare for LTO support with other compilers, common parts are
gated behind the CONFIG_LTO option, and LTO can be disabled for
specific files by filtering out CC_FLAGS_LTO.
Signed-off-by: Sami Tolvanen <samitolvanen@google.com>
Reviewed-by: Kees Cook <keescook@chromium.org>
Signed-off-by: Kees Cook <keescook@chromium.org>
Link: https://lore.kernel.org/r/20201211184633.3213045-3-samitolvanen@google.com
The same code exists a few lines above.
Fixes: 436b2ac603 ("modpost: invoke modpost only when input files are updated")
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
The -s option was added by commit 8d8d8289df ("kbuild: do not do
section mismatch checks on vmlinux in 2nd pass").
Now that the second pass does not parse vmlinux, this option is
unneeded.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
If modpost fails to load a symbol dump file, it cannot check unresolved
symbols, hence module dependency will not be added. Nor CRCs can be added.
Currently, external module builds check only $(objtree)/Module.symvers,
but it should check files specified by KBUILD_EXTRA_SYMBOLS as well.
Move the warning message from the top Makefile to scripts/Makefile.modpost
and print the warning if any dump file is missing.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Currently, the second pass of modpost is always invoked when you run
'make' or 'make modules' even if none of modules is changed.
Use if_changed to invoke it only when it is necessary.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
The full build runs modpost twice, first for vmlinux.o and second for
modules.
The first pass dumps all the vmlinux symbols into Module.symvers, but
the second pass parses vmlinux again instead of reusing the dump file,
presumably because it needs to avoid accumulating stale symbols.
Loading symbol info from a dump file is faster than parsing an ELF object.
Besides, modpost deals with various issues to parse vmlinux in the second
pass.
A solution is to make the first pass dumps symbols into a separate file,
vmlinux.symvers. The second pass reads it, and parses module .o files.
The merged symbol information is dumped into Module.symvers in the same
way as before.
This makes further modpost cleanups possible.
Also, it fixes the problem of 'make vmlinux', which previously overwrote
Module.symvers, throwing away module symbols.
I slightly touched scripts/link-vmlinux.sh so that vmlinux is re-linked
when you cross this commit. Otherwise, vmlinux.symvers would not be
generated.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Previously, the -i option had two functions; load a symbol dump file,
and set the external_module flag.
I want to assign a dedicate option for each of them.
Going forward, the -i is used to load a symbol dump file, and the -e
to set the external_module flag.
With this, we will be able to use -i for loading in-kernel symbols.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Now that there is no difference between -i and -e, they can be unified.
Make modpost accept the -i option multiple times, then remove -e.
I will reuse -e for a different purpose.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
The built-in only code is not required to have MODULE_IMPORT_NS() to
use symbols. So, the namespace is not checked for vmlinux(.o).
Do not pass the meaningless -N option to the first pass of modpost.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
The '-T -' option reads the file list from stdin.
It is clearer to put it close to the piped command.
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
$(filter -i,$(MAKEFLAGS)) works only in limited use-cases.
The representation of $(MAKEFLAGS) depends on various factors:
- GNU Make version (version 3.8x or version 4.x)
- The presence of other flags like -j
In my experiments, $(MAKEFLAGS) is expanded as follows:
* GNU Make 3.8x:
* without -j option:
--no-print-directory -Rri
* with -j option:
--no-print-directory -Rr --jobserver-fds=3,4 -j -i
* GNU Make 4.x:
* without -j option:
irR --no-print-directory
* with -j option:
irR -j --jobserver-fds=3,4 --no-print-directory
For GNU Make 4.x, the flags are grouped as 'irR', which does not work.
For the single thread build with GNU Make 3.8x, the flags are grouped
as '-Rri', which does not work either.
To make it work for all cases, do likewise as commit 6f0fa58e45
("kbuild: simplify silent build (-s) detection").
BTW, since commit ff9b45c55b ("kbuild: modpost: read modules.order
instead of $(MODVERDIR)/*.mod"), you also need to pass -k option to
build final *.ko files. 'make -i -k' ignores compile errors in modules,
and build as many remaining *.ko as possible.
Please note this feature is kind of dangerous if other modules depend
on the broken module because the generated modules will lack the correct
module dependency or CRC. Honestly, I am not a big fan of it, but I am
keeping this feature.
Fixes: eed380f3f5 ("modpost: Optionally ignore secondary errors seen if a single module build fails")
Cc: Guenter Roeck <linux@roeck-us.net>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
$(firstword ...) in scripts/Makefile.modpost was added by commit
3f3fd3c055 ("[PATCH] kbuild: allow multi-word $M in Makefile.modpost")
to build multiple external module directories.
It was a solution to resolve symbol dependencies when an external
module depends on another external module.
Commit 0d96fb20b7 ("kbuild: Add new Kbuild variable
KBUILD_EXTRA_SYMBOLS") introduced another solution by passing symbol
info via KBUILD_EXTRA_SYMBOLS, then broke the multi-word M= support.
include $(if $(wildcard $(KBUILD_EXTMOD)/Kbuild), \
$(KBUILD_EXTMOD)/Kbuild, $(KBUILD_EXTMOD)/Makefile)
... does not work if KBUILD_EXTMOD contains multiple words.
This feature has been broken for more than a decade. Remove the
bitrotten code, and stop parsing if M or KBUILD_EXTMOD contains
multiple words.
As Documentation/kbuild/modules.rst explains, if your module depends
on another one, there are two solutions:
- add a common top-level Kbuild file
- use KBUILD_EXTRA_SYMBOLS
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
Currently when CONFIG_MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS=n, modpost
only warns when a module is missing namespace imports. Under this
configuration, such a module cannot be loaded into the kernel anyway, as
the module loader would reject it. We might as well return a build
error when a module is missing namespace imports under
CONFIG_MODULE_ALLOW_MISSING_NAMESPACE_IMPORTS=n, so that the build
warning does not go ignored/unnoticed.
Signed-off-by: Jessica Yu <jeyu@kernel.org>
Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
scripts/nsdeps is written to take care of only in-tree modules.
Perhaps, this is not a bug, but just a design. At least,
Documentation/core-api/symbol-namespaces.rst focuses on in-tree modules.
Having said that, some people already tried nsdeps for external modules.
So, it would be nice to support it.
Reported-by: Steve French <smfrench@gmail.com>
Reported-by: Jessica Yu <jeyu@kernel.org>
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Tested-by: Jessica Yu <jeyu@kernel.org>
Acked-by: Jessica Yu <jeyu@kernel.org>
Reviewed-by: Matthias Maennich <maennich@google.com>
Tested-by: Matthias Maennich <maennich@google.com>
The modpost, with the -d option given, generates per-module .ns_deps
files.
Kbuild generates per-module .mod files to carry module information.
This is convenient because Make handles multiple jobs in parallel
when the -j option is given.
On the other hand, the modpost always runs as a single thread.
I do not see a strong reason to produce separate .ns_deps files.
This commit changes the modpost to generate just one file,
modules.nsdeps, each line of which has the following format:
<module_name>: <list of missing namespaces>
Please note it contains *missing* namespaces instead of required ones.
So, modules.nsdeps is empty if the namespace dependency is all good.
This will work more efficiently because spatch will no longer process
already imported namespaces. I removed the '(if needed)' from the
nsdeps log since spatch is invoked only when needed.
This also solves the stale .ns_deps problem reported by Jessica Yu:
https://lkml.org/lkml/2019/10/28/467
Signed-off-by: Masahiro Yamada <yamada.masahiro@socionext.com>
Tested-by: Jessica Yu <jeyu@kernel.org>
Acked-by: Jessica Yu <jeyu@kernel.org>
Reviewed-by: Matthias Maennich <maennich@google.com>
Tested-by: Matthias Maennich <maennich@google.com>
