LLVM
In order to achieve the goals of testability concerning the LLVM projects, I had to prepare a new pkgsrc-wip package called llvm-all-in-one that contains 12 active LLVM projects within one tree. The set of these projects is composed of: llvm, clang, compiler-rt, libcxx, libcxxabi, libunwind, test-suite, openmp, llgo, lld, lldb, clang-tools-extra. These were required to build and execute test-suites in the LLVM's projects. Ideally the tests should work in standalone packages - built out-of-LLVM-sources - and with GCC/Clang, however the real life is less bright and this forced me to use Clang as the system compiler an all-in-one package in order to develop the work environment with the ability to build and execute unit tests.
There were four threads within LLVM:
- Broken std::call_once with libstdc++. This is an old and well-known bug, which was usually worked around with a homegrown implementation llvm::call_once. I've discovered that the llvm::call_once workaround isn't sufficient for the whole LLVM functionality, as std::call_once can be called internally inside the libstdc++ libraries - like within the C++11 futures interface. This bug has been solved by Joerg Sonnenberger in the ELF dynamic linker.
- Unportable shell construct hardcoded in tests ">&". This has been fixed upstream.
- LLVM JIT. The LLVM Memory generic allocator (or page mapper) was designed to freely map pages with any combination of the protection bits: R,W,X. This approach breaks on NetBSD with PaX MPROTECT and requires redesign of the interfaces. This is the continuation of the past month AllocateRWX and ReleaseRWX compatibility with NetBSD improvements. I've prepared few variations of local patches addressing these issues and it's still open for discussion with upstream. My personal preference is to remove the current API entirely and introduce a newer one with narrowed down functionality to swap between readable (R--), writable (RW-) and executable (R-X) memory pages. This would effectively enforce W^X.
- Sanitizers support. Right now, I keep the patches locally in order to upstream the common sanitizer code in compiler-rt.
The LLVM JIT API is the last cause of unexpected failures in check-llvm. This breaks MCJIT, ORCJIT and ExecutionEngine libraries and causes around 200 unexpected failures within tests.
Clang
I've upstreamed a patch that enables ubsan and asan on Clang's frontend for NetBSD/amd64. This support isn't complete, and requires sanitizers' support code upstreamed to compiler-rt.
compiler-rt
The current compiler-rt tasks can be divided into:
- upstream sanitizer common code shared with POSIX platforms
- upstream sanitizer common code shared with Linux and FreeBSD
- upstream sanitizer common code shared with FreeBSD
- upstream sanitizer common code specific to NetBSD
- build, execute and pass tests for sanitizer common code in check-santizer
This means that ubsan, asan and the rest of the specific sanitizers wait in queue.
All the mentioned tasks are being worked on simultaneously, with a soft goal to finish them one after another from the first to the last one.
The last point with check-sanitizer unveiled so far two generic bugs on NetBSD:
- Return errno EFAULT instead of EACCES on memory fault with read(2)/write(2)-like syscalls.
- Honor PTHREAD_DESTRUCTOR_ITERATIONS in libpthread.
Plan for the next milestone
I have decided not to open new issues in with the coming month and focus on upstreaming the remaining LLVM code. The roadmap for the next month is to continue working on the goals of the previous months. std::call_once is an example that every delayed bug keeps biting again and again in future.
LLVM 5.0.0 is planned to be released this month (August) and there is a joint motivation with the upstream maintainer to push compatibility fixes for LLVM JIT. There is an option to submit a workaround now and introduce refactoring for the trunk and next version (6.0.0).
This work was sponsored by The NetBSD Foundation.
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
LLVM
In order to achieve the goals of testability concerning the LLVM projects, I had to prepare a new pkgsrc-wip package called llvm-all-in-one that contains 12 active LLVM projects within one tree. The set of these projects is composed of: llvm, clang, compiler-rt, libcxx, libcxxabi, libunwind, test-suite, openmp, llgo, lld, lldb, clang-tools-extra. These were required to build and execute test-suites in the LLVM's projects. Ideally the tests should work in standalone packages - built out-of-LLVM-sources - and with GCC/Clang, however the real life is less bright and this forced me to use Clang as the system compiler an all-in-one package in order to develop the work environment with the ability to build and execute unit tests.
There were four threads within LLVM:
- Broken std::call_once with libstdc++. This is an old and well-known bug, which was usually worked around with a homegrown implementation llvm::call_once. I've discovered that the llvm::call_once workaround isn't sufficient for the whole LLVM functionality, as std::call_once can be called internally inside the libstdc++ libraries - like within the C++11 futures interface. This bug has been solved by Joerg Sonnenberger in the ELF dynamic linker.
- Unportable shell construct hardcoded in tests ">&". This has been fixed upstream.
- LLVM JIT. The LLVM Memory generic allocator (or page mapper) was designed to freely map pages with any combination of the protection bits: R,W,X. This approach breaks on NetBSD with PaX MPROTECT and requires redesign of the interfaces. This is the continuation of the past month AllocateRWX and ReleaseRWX compatibility with NetBSD improvements. I've prepared few variations of local patches addressing these issues and it's still open for discussion with upstream. My personal preference is to remove the current API entirely and introduce a newer one with narrowed down functionality to swap between readable (R--), writable (RW-) and executable (R-X) memory pages. This would effectively enforce W^X.
- Sanitizers support. Right now, I keep the patches locally in order to upstream the common sanitizer code in compiler-rt.
The LLVM JIT API is the last cause of unexpected failures in check-llvm. This breaks MCJIT, ORCJIT and ExecutionEngine libraries and causes around 200 unexpected failures within tests.
Clang
I've upstreamed a patch that enables ubsan and asan on Clang's frontend for NetBSD/amd64. This support isn't complete, and requires sanitizers' support code upstreamed to compiler-rt.
compiler-rt
The current compiler-rt tasks can be divided into:
- upstream sanitizer common code shared with POSIX platforms
- upstream sanitizer common code shared with Linux and FreeBSD
- upstream sanitizer common code shared with FreeBSD
- upstream sanitizer common code specific to NetBSD
- build, execute and pass tests for sanitizer common code in check-santizer
This means that ubsan, asan and the rest of the specific sanitizers wait in queue.
All the mentioned tasks are being worked on simultaneously, with a soft goal to finish them one after another from the first to the last one.
The last point with check-sanitizer unveiled so far two generic bugs on NetBSD:
- Return errno EFAULT instead of EACCES on memory fault with read(2)/write(2)-like syscalls.
- Honor PTHREAD_DESTRUCTOR_ITERATIONS in libpthread.
Plan for the next milestone
I have decided not to open new issues in with the coming month and focus on upstreaming the remaining LLVM code. The roadmap for the next month is to continue working on the goals of the previous months. std::call_once is an example that every delayed bug keeps biting again and again in future.
LLVM 5.0.0 is planned to be released this month (August) and there is a joint motivation with the upstream maintainer to push compatibility fixes for LLVM JIT. There is an option to submit a workaround now and introduce refactoring for the trunk and next version (6.0.0).
This work was sponsored by The NetBSD Foundation.
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
In this blog post series I will discuss about SUBPACKAGES work
done during
Google Summer of Code 2017.
In this first part I'll briefly introduce what are SUBPACKAGES,
why and when can be useful and finally we'll give a quick look to a trivial
pkgsrc package that uses them. At the end
we'll also dive a bit on parts of the pkgsrc infrastructure that needed to be
adjusted for implementing that.
Introduction
SUBPACKAGES (on some package systems they are known as
multi-packages, but this term for pkgsrc is already used by packages
that can be built against several versions (e.g. Python, PHP, Ruby packages))
consist in generating multiple binary packages from a single pkgsrc package.
For example, from a pkgsrc package - local/frobnitzem - we will
see how to generate three separate binary packages:
frobnitzem-foo, frobnitzem-bar and
frobnitzem-baz.
This can be useful to separate several components of binary
packages (and avoid to run the extract and
configure phase two times!),
for debugpkgs
(so that all *.debug files containing debug symbols are contained
in a separate -debugpkg package that can be installed only when it
is needed), etc..
A simple SUBPACKAGES package: frobnitzem!
To understand how SUBPACKAGES works and can be useful let's start
to see an example of it in practice: frobnitzem.
frobnitzem is a trivial package that just install three scripts in
${PREFIX}/bin, let's see it:
% cd pkgsrc/local/frobnitzem
% tree
.
|-- DESCR
|-- Makefile
|-- PLIST
`-- files
`-- frobnitzem
|-- frobnitzem-bar
|-- frobnitzem-baz
`-- frobnitzem-foo
2 directories, 6 files
% find . -type f | xargs tail -n +1
==> ./Makefile <==
# $NetBSD$
DISTNAME= frobnitzem-0
CATEGORIES= local
MASTER_SITES= # empty
DISTFILES= # empty
MAINTAINER= leot%NetBSD.org@localhost
HOMEPAGE= http://netbsd.org/~leot/gsoc2017-diary/
COMMENT= Simple subpackages example
LICENSE= public-domain
FILESDIR= ${.CURDIR}/../../local/frobnitzem/files
WRKSRC= ${WRKDIR}/frobnitzem
NO_BUILD= yes
do-extract:
${CP} -r ${FILESDIR}/frobnitzem ${WRKDIR}
do-install:
${INSTALL_SCRIPT_DIR} ${DESTDIR}${PREFIX}/bin
${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-foo ${DESTDIR}${PREFIX}/bin
${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-bar ${DESTDIR}${PREFIX}/bin
${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-baz ${DESTDIR}${PREFIX}/bin
.include "../../mk/bsd.pkg.mk"
==> ./files/frobnitzem/frobnitzem-bar <==
#!/bin/sh
echo "bar"
==> ./files/frobnitzem/frobnitzem-baz <==
#!/bin/sh
echo "baz"
==> ./files/frobnitzem/frobnitzem-foo <==
#!/bin/sh
echo "foo"
==> ./PLIST <==
@comment $NetBSD$
bin/frobnitzem-bar
bin/frobnitzem-baz
bin/frobnitzem-foo
==> ./DESCR <==
frobnitzem, collection of foo, bar, baz scripts.
(Or, a bit more seriously this is just a very simple package to
test subpackages support!)
Nothing fancy, just three simple scripts, frobnitzem-{bar,baz,foo}
that will respectively print to the standard output bar,
baz and foo.
Let's build and install the frobnitzem package:
% make install ===> Installing dependencies for frobnitzem-0 ===> Overriding tools for frobnitzem-0 ===> Extracting for frobnitzem-0 [...] ===> Installing for frobnitzem-0 [...] ===> Installing binary package of frobnitzem-0 [...]
And now let's try scripts installed as part of the frobnitzem
package:
% foreach w (bar baz foo) ... frobnitzem-$w ... end bar baz foo
Okay, as we expected. Despite frobnitzem-{foo,bar,baz} don't do
anything particularly useful we can split the frobnitzem-0
package in three separate subpackages: frobnitzem-foo-0,
frobnitzem-bar-0 and
frobnitzem-baz-0 (they provides different functionalities and can
also coexist if they're in separated binary packages).
To do that we need to slighty adjust Makefile, split the
PLIST in PLIST.{foo,bar,baz} (one for each separate
subpackage), split the DESCR in DESCR.{foo,bar,baz}.
So, at the end in local/frobnitzem we'll have:
% tree
.
|-- DESCR.bar
|-- DESCR.baz
|-- DESCR.foo
|-- Makefile
|-- PLIST.bar
|-- PLIST.baz
|-- PLIST.foo
`-- files
`-- frobnitzem
|-- frobnitzem-bar
|-- frobnitzem-baz
`-- frobnitzem-foo
2 directories, 10 files
Splitting DESCR and PLIST
DESCR and PLIST splits are straightforward. We just
provide a separate DESCR.<spkg> for each subpackage,
e.g. for the foo subpackage:
% cat DESCR.foo frobnitzem, collection of foo, bar, baz scripts. (Or, a bit more seriously this is just a very simple package to test subpackages support!) This package provide the foo functionalities.
Similarly, regarding PLISTs, we just provide a separate
PLIST.<spkg> for each subpackage, e.g. for the
foo subpackage:
% cat PLIST.foo @comment $NetBSD$ bin/frobnitzem-foo
Makefile changes
In Makefile we'll need to list all SUBPACKAGES hence we'll add the
following line as first paragraph:
SUBPACKAGES= foo bar baz
We'll then need to define a PKGNAME.<spkg> for each
subpackages:
PKGNAME.foo= frobnitzem-foo-0 PKGNAME.bar= frobnitzem-bar-0 PKGNAME.baz= frobnitzem-baz-0
...and similarly COMMENT variable should be defined for each
subpackage via COMMENT.<spkg>:
COMMENT.foo= Simple subpackages example (foo) COMMENT.bar= Simple subpackages example (bar) COMMENT.baz= Simple subpackages example (baz)
To recap here how we have adjusted Makefile, all the other lines rest unchanged:
% sed '/LICENSE/q' < Makefile # $NetBSD$ SUBPACKAGES= foo bar baz DISTNAME= frobnitzem-0 PKGNAME.foo= frobnitzem-foo-0 PKGNAME.bar= frobnitzem-bar-0 PKGNAME.baz= frobnitzem-baz-0 MASTER_SITES= # empty DISTFILES= # empty CATEGORIES= local MAINTAINER= leot%NetBSD.org@localhost HOMEPAGE= http://netbsd.org/~leot/gsoc2017-diary/ COMMENT.foo= Simple subpackages example (foo) COMMENT.bar= Simple subpackages example (bar) COMMENT.baz= Simple subpackages example (baz) LICENSE= public-domain
Finally we can install it^Wthem! The usual make
install will generate three binary packages
(frobnitzem-foo-0.tgz, frobnitzem-bar-0.tgz,
frobnitzem-baz-0.tgz) and install all of them:
% make install ===> Installing dependencies for frobnitzem-0 [...] ===> Overriding tools for frobnitzem-0 ===> Extracting for frobnitzem-0 [...] ===> Installing for frobnitzem-0 [...] => Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-foo-0.tgz => Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-bar-0.tgz => Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-baz-0.tgz [...] ===> Installing binary package of frobnitzem-foo-0 ===> Installing binary package of frobnitzem-bar-0 ===> Installing binary package of frobnitzem-baz-0 [...]
Now we can try them and use pkg_info(1) to get some information about them:
% frobnitzem-foo foo % pkg_info -Fe /usr/pkg/bin/frobnitzem-foo frobnitzem-foo-0 % pkg_info frobnitzem-baz Information for frobnitzem-baz-0: Comment: Simple subpackages example (baz) Description: frobnitzem, collection of foo, bar, baz scripts. (Or, a bit more seriously this is just a very simple package to test subpackages support!) This package provide the baz functionalities. Homepage: http://netbsd.org/~leot/gsoc2017-diary/ % pkg_info -L frobnitzem-bar Information for frobnitzem-bar-0: Files: /usr/pkg/bin/frobnitzem-bar
So we can see that make install actually installed three different
binary packages.
To deinstall all SUBPACKAGES we can run make deinstall
in the local/frobnitzem directory (that will remove all
subpackages) or we can just manually invoke
pkg_delete(1).
An high-level look at how SUBPACKAGES support is implemented
Most of the changes needed are in mk/pkgformat/pkg/ hierarchy
(previously known as mk/flavour and then renamed and generalized to
other package formats during
Anton Panev's Google Summer of Code 2011).
The code in mk/pkgformat/${PKG_FORMAT}/ handle the interaction of
pkgsrc with the particular ${PKG_FORMAT}, e.g. for pkg
populate meta-data files used by
pkg_create(1),
install/delete packages via
pkg_add(1),
and
pkg_delete(1),
etc.
For more information
mk/pkgformat/README
is a good introduction to pkgformat hierarchy.
Most of the changes done respect the following template:
.if !empty(SUBPACKAGES)
. for _spkg_ in ${SUBPACKAGES}
[... code that handles SUBPACKAGES case ...]
. endfor
.else # !SUBPACKAGES
[... existing (and usually completely unmodified) code ...]
.endif # SUBPACKAGES
In particular, in mk/pkgformat/pkg/ targets were adjusted to
create/install/deinstall/etc. all subpackages.
Apart mk/pkgformat other changes were needed in
mk/install/install.mk in order to adjust the install
phase for SUBPACKAGES.
Regarding PLIST.<spkg> handling mk/plist/plist.mk
needed some adjustments to honor each PLIST per-subpackage.
mk/bsd.pkg.mk needed to be adjusted too in order to honor several
per-subpackage variables (the *.<spkg> ones) and
per-subpackage DESCR.<spkg>.
Conclusion
In this first part of this blog post series we have seen what are
SUBPACKAGES, when and why they can be useful.
We have then seen a practical example of them taking a very trivial package and learned how to "subpackage-ify" it.
Then we have described - from an high-level perspective - the changes needed to
the pkgsrc infrastructure for the SUBPACKAGES features that we have
used. If you are more interested in them please give a look to the
pkgsrc debugpkg
branch that contains all work done described in this blog post.
In the next part we will see how to handle *DEPENDS and
buildlink3 inclusion for subpackages.
I would like to thanks Google for organizing Google Summer of Code, the entire The NetBSD Foundation and in particular my mentors Taylor R. Campbell, William J. Coldwell and Thomas Klausner for providing precious guidance during these three months. A special thank you also to Jörg Sonnenberger who provided very useful suggestions. Thank you!
In this blog post series I will discuss about SUBPACKAGES work
done during
Google Summer of Code 2017.
In this first part I'll briefly introduce what are SUBPACKAGES,
why and when can be useful and finally we'll give a quick look to a trivial
pkgsrc package that uses them. At the end
we'll also dive a bit on parts of the pkgsrc infrastructure that needed to be
adjusted for implementing that.
Introduction
SUBPACKAGES (on some package systems they are known as
multi-packages, but this term for pkgsrc is already used by packages
that can be built against several versions (e.g. Python, PHP, Ruby packages))
consist in generating multiple binary packages from a single pkgsrc package.
For example, from a pkgsrc package - local/frobnitzem - we will
see how to generate three separate binary packages:
frobnitzem-foo, frobnitzem-bar and
frobnitzem-baz.
This can be useful to separate several components of binary
packages (and avoid to run the extract and
configure phase two times!),
for debugpkgs
(so that all *.debug files containing debug symbols are contained
in a separate -debugpkg package that can be installed only when it
is needed), etc..
A simple SUBPACKAGES package: frobnitzem!
To understand how SUBPACKAGES works and can be useful let's start
to see an example of it in practice: frobnitzem.
frobnitzem is a trivial package that just install three scripts in
${PREFIX}/bin, let's see it:
% cd pkgsrc/local/frobnitzem
% tree
.
|-- DESCR
|-- Makefile
|-- PLIST
`-- files
`-- frobnitzem
|-- frobnitzem-bar
|-- frobnitzem-baz
`-- frobnitzem-foo
2 directories, 6 files
% find . -type f | xargs tail -n +1
==> ./Makefile <==
# $NetBSD$
DISTNAME= frobnitzem-0
CATEGORIES= local
MASTER_SITES= # empty
DISTFILES= # empty
MAINTAINER= leot%NetBSD.org@localhost
HOMEPAGE= http://netbsd.org/~leot/gsoc2017-diary/
COMMENT= Simple subpackages example
LICENSE= public-domain
FILESDIR= ${.CURDIR}/../../local/frobnitzem/files
WRKSRC= ${WRKDIR}/frobnitzem
NO_BUILD= yes
do-extract:
${CP} -r ${FILESDIR}/frobnitzem ${WRKDIR}
do-install:
${INSTALL_SCRIPT_DIR} ${DESTDIR}${PREFIX}/bin
${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-foo ${DESTDIR}${PREFIX}/bin
${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-bar ${DESTDIR}${PREFIX}/bin
${INSTALL_SCRIPT} ${WRKSRC}/frobnitzem-baz ${DESTDIR}${PREFIX}/bin
.include "../../mk/bsd.pkg.mk"
==> ./files/frobnitzem/frobnitzem-bar <==
#!/bin/sh
echo "bar"
==> ./files/frobnitzem/frobnitzem-baz <==
#!/bin/sh
echo "baz"
==> ./files/frobnitzem/frobnitzem-foo <==
#!/bin/sh
echo "foo"
==> ./PLIST <==
@comment $NetBSD$
bin/frobnitzem-bar
bin/frobnitzem-baz
bin/frobnitzem-foo
==> ./DESCR <==
frobnitzem, collection of foo, bar, baz scripts.
(Or, a bit more seriously this is just a very simple package to
test subpackages support!)
Nothing fancy, just three simple scripts, frobnitzem-{bar,baz,foo}
that will respectively print to the standard output bar,
baz and foo.
Let's build and install the frobnitzem package:
% make install ===> Installing dependencies for frobnitzem-0 ===> Overriding tools for frobnitzem-0 ===> Extracting for frobnitzem-0 [...] ===> Installing for frobnitzem-0 [...] ===> Installing binary package of frobnitzem-0 [...]
And now let's try scripts installed as part of the frobnitzem
package:
% foreach w (bar baz foo) ... frobnitzem-$w ... end bar baz foo
Okay, as we expected. Despite frobnitzem-{foo,bar,baz} don't do
anything particularly useful we can split the frobnitzem-0
package in three separate subpackages: frobnitzem-foo-0,
frobnitzem-bar-0 and
frobnitzem-baz-0 (they provides different functionalities and can
also coexist if they're in separated binary packages).
To do that we need to slighty adjust Makefile, split the
PLIST in PLIST.{foo,bar,baz} (one for each separate
subpackage), split the DESCR in DESCR.{foo,bar,baz}.
So, at the end in local/frobnitzem we'll have:
% tree
.
|-- DESCR.bar
|-- DESCR.baz
|-- DESCR.foo
|-- Makefile
|-- PLIST.bar
|-- PLIST.baz
|-- PLIST.foo
`-- files
`-- frobnitzem
|-- frobnitzem-bar
|-- frobnitzem-baz
`-- frobnitzem-foo
2 directories, 10 files
Splitting DESCR and PLIST
DESCR and PLIST splits are straightforward. We just
provide a separate DESCR.<spkg> for each subpackage,
e.g. for the foo subpackage:
% cat DESCR.foo frobnitzem, collection of foo, bar, baz scripts. (Or, a bit more seriously this is just a very simple package to test subpackages support!) This package provide the foo functionalities.
Similarly, regarding PLISTs, we just provide a separate
PLIST.<spkg> for each subpackage, e.g. for the
foo subpackage:
% cat PLIST.foo @comment $NetBSD$ bin/frobnitzem-foo
Makefile changes
In Makefile we'll need to list all SUBPACKAGES hence we'll add the
following line as first paragraph:
SUBPACKAGES= foo bar baz
We'll then need to define a PKGNAME.<spkg> for each
subpackages:
PKGNAME.foo= frobnitzem-foo-0 PKGNAME.bar= frobnitzem-bar-0 PKGNAME.baz= frobnitzem-baz-0
...and similarly COMMENT variable should be defined for each
subpackage via COMMENT.<spkg>:
COMMENT.foo= Simple subpackages example (foo) COMMENT.bar= Simple subpackages example (bar) COMMENT.baz= Simple subpackages example (baz)
To recap here how we have adjusted Makefile, all the other lines rest unchanged:
% sed '/LICENSE/q' < Makefile # $NetBSD$ SUBPACKAGES= foo bar baz DISTNAME= frobnitzem-0 PKGNAME.foo= frobnitzem-foo-0 PKGNAME.bar= frobnitzem-bar-0 PKGNAME.baz= frobnitzem-baz-0 MASTER_SITES= # empty DISTFILES= # empty CATEGORIES= local MAINTAINER= leot%NetBSD.org@localhost HOMEPAGE= http://netbsd.org/~leot/gsoc2017-diary/ COMMENT.foo= Simple subpackages example (foo) COMMENT.bar= Simple subpackages example (bar) COMMENT.baz= Simple subpackages example (baz) LICENSE= public-domain
Finally we can install it^Wthem! The usual make
install will generate three binary packages
(frobnitzem-foo-0.tgz, frobnitzem-bar-0.tgz,
frobnitzem-baz-0.tgz) and install all of them:
% make install ===> Installing dependencies for frobnitzem-0 [...] ===> Overriding tools for frobnitzem-0 ===> Extracting for frobnitzem-0 [...] ===> Installing for frobnitzem-0 [...] => Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-foo-0.tgz => Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-bar-0.tgz => Creating binary package /home/leot/repos/netbsd-github/pkgsrc/packages/All/frobnitzem-baz-0.tgz [...] ===> Installing binary package of frobnitzem-foo-0 ===> Installing binary package of frobnitzem-bar-0 ===> Installing binary package of frobnitzem-baz-0 [...]
Now we can try them and use pkg_info(1) to get some information about them:
% frobnitzem-foo foo % pkg_info -Fe /usr/pkg/bin/frobnitzem-foo frobnitzem-foo-0 % pkg_info frobnitzem-baz Information for frobnitzem-baz-0: Comment: Simple subpackages example (baz) Description: frobnitzem, collection of foo, bar, baz scripts. (Or, a bit more seriously this is just a very simple package to test subpackages support!) This package provide the baz functionalities. Homepage: http://netbsd.org/~leot/gsoc2017-diary/ % pkg_info -L frobnitzem-bar Information for frobnitzem-bar-0: Files: /usr/pkg/bin/frobnitzem-bar
So we can see that make install actually installed three different
binary packages.
To deinstall all SUBPACKAGES we can run make deinstall
in the local/frobnitzem directory (that will remove all
subpackages) or we can just manually invoke
pkg_delete(1).
An high-level look at how SUBPACKAGES support is implemented
Most of the changes needed are in mk/pkgformat/pkg/ hierarchy
(previously known as mk/flavour and then renamed and generalized to
other package formats during
Anton Panev's Google Summer of Code 2011).
The code in mk/pkgformat/${PKG_FORMAT}/ handle the interaction of
pkgsrc with the particular ${PKG_FORMAT}, e.g. for pkg
populate meta-data files used by
pkg_create(1),
install/delete packages via
pkg_add(1),
and
pkg_delete(1),
etc.
For more information
mk/pkgformat/README
is a good introduction to pkgformat hierarchy.
Most of the changes done respect the following template:
.if !empty(SUBPACKAGES)
. for _spkg_ in ${SUBPACKAGES}
[... code that handles SUBPACKAGES case ...]
. endfor
.else # !SUBPACKAGES
[... existing (and usually completely unmodified) code ...]
.endif # SUBPACKAGES
In particular, in mk/pkgformat/pkg/ targets were adjusted to
create/install/deinstall/etc. all subpackages.
Apart mk/pkgformat other changes were needed in
mk/install/install.mk in order to adjust the install
phase for SUBPACKAGES.
Regarding PLIST.<spkg> handling mk/plist/plist.mk
needed some adjustments to honor each PLIST per-subpackage.
mk/bsd.pkg.mk needed to be adjusted too in order to honor several
per-subpackage variables (the *.<spkg> ones) and
per-subpackage DESCR.<spkg>.
Conclusion
In this first part of this blog post series we have seen what are
SUBPACKAGES, when and why they can be useful.
We have then seen a practical example of them taking a very trivial package and learned how to "subpackage-ify" it.
Then we have described - from an high-level perspective - the changes needed to
the pkgsrc infrastructure for the SUBPACKAGES features that we have
used. If you are more interested in them please give a look to the
pkgsrc debugpkg
branch that contains all work done described in this blog post.
In the next part we will see how to handle *DEPENDS and
buildlink3 inclusion for subpackages.
I would like to thanks Google for organizing Google Summer of Code, the entire The NetBSD Foundation and in particular my mentors Taylor R. Campbell, William J. Coldwell and Thomas Klausner for providing precious guidance during these three months. A special thank you also to Jörg Sonnenberger who provided very useful suggestions. Thank you!