Linux kernel stable tree
Go to file
Masahiro Yamada 5c3d1d0abb kbuild: add a tool to list files ignored by git
In short, the motivation of this commit is to build a source package
without cleaning the source tree.

The deb-pkg and (src)rpm-pkg targets first run 'make clean' before
creating a source tarball. Otherwise build artifacts such as *.o,
*.a, etc. would be included in the tarball. Yet, the tarball ends up
containing several garbage files since 'make clean' does not clean
everything.

Cleaning the tree every time is annoying since it makes the incremental
build impossible. It is desirable to create a source tarball without
cleaning the tree.

In fact, there are some ways to achieve this.

The easiest solution is 'git archive'. 'make perf-tar*-src-pkg' uses
it, but I do not like it because it works only when the source tree is
managed by git, and all files you want in the tarball must be committed
in advance.

I want to make it work without relying on git. We can do this.

Files that are ignored by git are generated files, so should be excluded
from the source tarball. We can list them out by parsing the .gitignore
files. Of course, .gitignore does not cover all the cases, but it works
well enough.

tar(1) claims to support it:

  --exclude-vcs-ignores

    Exclude files that match patterns read from VCS-specific ignore files.
    Supported files are: .cvsignore, .gitignore, .bzrignore, and .hgignore.

The best scenario would be to use 'tar --exclude-vcs-ignores', but this
option does not work. --exclude-vcs-ignore does not understand any of
the negation (!), preceding slash, following slash, etc.. So, this option
is just useless.

Hence, I wrote this gitignore parser. The previous version [1], written
in Python, was so slow. This version is implemented in C, so it works
much faster.

I imported the code from git (commit: 23c56f7bd5f1), so we get the same
result.

This tool traverses the source tree, parsing all .gitignore files, and
prints file paths that are ignored by git.

The output is similar to 'git ls-files --ignored --directory --others
--exclude-per-directory=.gitignore', except

  [1] Not sorted
  [2] No trailing slash for directories

[2] is intentional because tar's --exclude-from option cannot handle
trailing slashes.

[How to test this tool]

  $ git clean -dfx
  $ make -s -j$(nproc) defconfig all                       # or allmodconifg or whatever
  $ git archive -o ../linux1.tar --prefix=./ HEAD
  $ tar tf ../linux1.tar | LANG=C sort > ../file-list1     # files emitted by 'git archive'
  $ make scripts_package
    HOSTCC  scripts/list-gitignored
  $ scripts/list-gitignored  --prefix=./ -o ../exclude-list
  $ tar cf ../linux2.tar --exclude-from=../exclude-list .
  $ tar tf ../linux2.tar | LANG=C sort > ../file-list2     # files emitted by 'tar'
  $ diff  ../file-list1 ../file-list2 | grep -E '^(<|>)'
  < ./Documentation/devicetree/bindings/.yamllint
  < ./drivers/clk/.kunitconfig
  < ./drivers/gpu/drm/tests/.kunitconfig
  < ./drivers/hid/.kunitconfig
  < ./fs/ext4/.kunitconfig
  < ./fs/fat/.kunitconfig
  < ./kernel/kcsan/.kunitconfig
  < ./lib/kunit/.kunitconfig
  < ./mm/kfence/.kunitconfig
  < ./tools/testing/selftests/arm64/tags/
  < ./tools/testing/selftests/arm64/tags/.gitignore
  < ./tools/testing/selftests/arm64/tags/Makefile
  < ./tools/testing/selftests/arm64/tags/run_tags_test.sh
  < ./tools/testing/selftests/arm64/tags/tags_test.c
  < ./tools/testing/selftests/kvm/.gitignore
  < ./tools/testing/selftests/kvm/Makefile
  < ./tools/testing/selftests/kvm/config
  < ./tools/testing/selftests/kvm/settings

The source tarball contains most of files that are tracked by git. You
see some diffs, but it is just because some .gitignore files are wrong.

  $ git ls-files -i -c --exclude-per-directory=.gitignore
  Documentation/devicetree/bindings/.yamllint
  drivers/clk/.kunitconfig
  drivers/gpu/drm/tests/.kunitconfig
  drivers/hid/.kunitconfig
  fs/ext4/.kunitconfig
  fs/fat/.kunitconfig
  kernel/kcsan/.kunitconfig
  lib/kunit/.kunitconfig
  mm/kfence/.kunitconfig
  tools/testing/selftests/arm64/tags/.gitignore
  tools/testing/selftests/arm64/tags/Makefile
  tools/testing/selftests/arm64/tags/run_tags_test.sh
  tools/testing/selftests/arm64/tags/tags_test.c
  tools/testing/selftests/kvm/.gitignore
  tools/testing/selftests/kvm/Makefile
  tools/testing/selftests/kvm/config
  tools/testing/selftests/kvm/settings

[1]: https://lore.kernel.org/all/20230128173843.765212-1-masahiroy@kernel.org/

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
2023-02-16 17:00:41 +09:00
arch kbuild: remove --include-dir MAKEFLAG from top Makefile 2023-02-05 18:51:22 +09:00
block block-6.2-2023-01-20 2023-01-20 12:44:41 -08:00
certs kbuild: do not print extra logs for V=2 2023-01-22 23:43:32 +09:00
crypto This update includes the following changes: 2022-12-14 12:31:09 -08:00
Documentation Documentation/llvm: add Chimera Linux, Google and Meta datacenters 2023-02-15 02:59:12 +09:00
drivers kbuild: remove --include-dir MAKEFLAG from top Makefile 2023-02-05 18:51:22 +09:00
fs kbuild: remove --include-dir MAKEFLAG from top Makefile 2023-02-05 18:51:22 +09:00
include USB / Thunderbolt fixes for 6.2-rc5 2023-01-21 11:10:03 -08:00
init scripts: remove bin2c 2023-01-26 12:43:33 +09:00
io_uring io_uring/poll: don't reissue in case of poll race on multishot request 2023-01-20 15:11:54 -07:00
ipc Non-MM patches for 6.2-rc1. 2022-12-12 17:28:58 -08:00
kernel kheaders: use standard naming for the temporary directory 2023-01-22 23:43:34 +09:00
lib v6.2 second rc pull request 2023-01-20 14:15:51 -08:00
LICENSES LICENSES: Add the copyleft-next-0.3.1 license 2022-11-08 15:44:01 +01:00
mm slab fixes for 6.2-rc5 2023-01-19 12:24:39 -08:00
net tcp: fix rate_app_limited to default to 1 2023-01-20 13:23:35 +00:00
rust kbuild: rust: move rust/target.json to scripts/ 2023-01-22 23:43:33 +09:00
samples Char/Misc driver changes for 6.2-rc1 2022-12-16 03:49:24 -08:00
scripts kbuild: add a tool to list files ignored by git 2023-02-16 17:00:41 +09:00
security tomoyo: Update website link 2023-01-13 23:11:38 +09:00
sound sound fixes for 6.2-rc4 2023-01-13 08:20:29 -06:00
tools linux-kselftest-fixes-6.2-rc5 2023-01-20 11:35:21 -08:00
usr usr/gen_init_cpio.c: remove unnecessary -1 values from int file 2022-10-03 14:21:44 -07:00
virt KVM: Ensure lockdep knows about kvm->lock vs. vcpu->mutex ordering rule 2023-01-11 13:32:21 -05:00
.clang-format iommufd for 6.2 2022-12-14 09:15:43 -08:00
.cocciconfig scripts: add Linux .cocciconfig for coccinelle 2016-07-22 12:13:39 +02:00
.get_maintainer.ignore get_maintainer: add Alan to .get_maintainer.ignore 2022-08-20 15:17:44 -07:00
.gitattributes .gitattributes: use 'dts' diff driver for dts files 2019-12-04 19:44:11 -08:00
.gitignore .gitignore: ignore *.cover and *.mbx 2023-02-05 18:51:22 +09:00
.mailmap 21 hotfixes. Thirteen of these address pre-6.1 issues and hence have 2023-01-16 16:36:39 -08:00
.rustfmt.toml rust: add .rustfmt.toml 2022-09-28 09:02:20 +02:00
COPYING COPYING: state that all contributions really are covered by this file 2020-02-10 13:32:20 -08:00
CREDITS MAINTAINERS: zram: zsmalloc: Add an additional co-maintainer 2022-12-15 16:37:49 -08:00
Kbuild Kbuild updates for v6.1 2022-10-10 12:00:45 -07:00
Kconfig kbuild: ensure full rebuild when the compiler is updated 2020-05-12 13:28:33 +09:00
MAINTAINERS MAINTAINERS: fix kbuild repo branch 2023-01-22 23:43:33 +09:00
Makefile kbuild: add a tool to list files ignored by git 2023-02-16 17:00:41 +09:00
README Drop all 00-INDEX files from Documentation/ 2018-09-09 15:08:58 -06:00

Linux kernel
============

There are several guides for kernel developers and users. These guides can
be rendered in a number of formats, like HTML and PDF. Please read
Documentation/admin-guide/README.rst first.

In order to build the documentation, use ``make htmldocs`` or
``make pdfdocs``.  The formatted documentation can also be read online at:

    https://www.kernel.org/doc/html/latest/

There are various text files in the Documentation/ subdirectory,
several of them using the Restructured Text markup notation.

Please read the Documentation/process/changes.rst file, as it contains the
requirements for building and running the kernel, and information about
the problems which may result by upgrading your kernel.