If you've been reading source-changes@, you likely noticed the recent creation of the netbsd-8 branch. If you haven't been reading source-changes@, here's some news: the netbsd-8 branch has been created, signaling the beginning of the release process for NetBSD 8.0.
We don't have a strict timeline for the 8.0 release, but things are looking pretty good at the moment, and we expect this release to happen in a shorter amount of time than the last couple major releases did.
At this point, we would love for folks to test out netbsd-8 and let us know how it goes. A couple of major improvements since 7.0 are the addition of USB 3 support and an overhaul of the audio subsystem, including an in-kernel mixer. Feedback about these areas is particularly desired.
To download the latest binaries built from the netbsd-8 branch, head to http://daily-builds.NetBSD.org/pub/NetBSD-daily/netbsd-8/
Thanks in advance for helping make NetBSD 8.0 a stellar release!
If you've been reading source-changes@, you likely noticed the recent creation of the netbsd-8 branch. If you haven't been reading source-changes@, here's some news: the netbsd-8 branch has been created, signaling the beginning of the release process for NetBSD 8.0.
We don't have a strict timeline for the 8.0 release, but things are looking pretty good at the moment, and we expect this release to happen in a shorter amount of time than the last couple major releases did.
At this point, we would love for folks to test out netbsd-8 and let us know how it goes. A couple of major improvements since 7.0 are the addition of USB 3 support and an overhaul of the audio subsystem, including an in-kernel mixer. Feedback about these areas is particularly desired.
To download the latest binaries built from the netbsd-8 branch, head to http://daily-builds.NetBSD.org/pub/NetBSD-daily/netbsd-8/
Thanks in advance for helping make NetBSD 8.0 a stellar release!
It turned out that more bugs were unveiled and this is not the final report on LLDB.
The good
Besides the greater enhancements this month I performed a cleanup in the ATF ptrace(2) tests again. Additionally I have managed to unbreak the LLDB Debug build and to eliminate compiler warnings in the NetBSD Native Process Plugin.
It is worth noting that LLVM can run tests on NetBSD again, the patch in gtest/LLVM has been installed by Joerg Sonnenberg and a more generic one has been submitted to the upstream googletest repository. There was also an improvement in ftruncate(2) on the LLVM side (authored by Joerg).
Since LLD (the LLVM linker) is advancing rapidly, it improved support for NetBSD and it can link a functional executable on NetBSD. I submitted a patch to stop crashing it on startup anymore. It was nearly used for linking LLDB/NetBSD and it spotted a real linking error... however there are further issues that need to be addressed in the future. Currently LLD is not part of the mainline LLDB tasks - it's part of improving the work environment. This linker should reduce the linking time - compared to GNU linkers - of LLDB by a factor of 3x-10x and save precious developer time. As of now LLDB linking can take minutes on a modern amd64 machine designed for performance.
Kernel correctness
I have researched (in pkgsrc-wip) initial support for multiple threads in the NetBSD Native Process Plugin. This code revealed - when running the LLDB regression test-suite - new kernel bugs. This unfortunately affects the usability of a debugger in a multithread environment in general and explains why GDB was never doing its job properly in such circumstances.
One of the first errors was asserting kernel panic with PT_*STEP, when a debuggee has more than a single thread. I have narrowed it down to lock primitives misuse in the do_ptrace() kernel code. The fix has been committed.
LLDB and userland correctness
LLDB introduced support for kevent(2) and it contains the following function:
Status MainLoop::RunImpl::Poll() {
in_events.resize(loop.m_read_fds.size());
unsigned i = 0;
for (auto &fd : loop.m_read_fds)
EV_SET(&in_events[i++], fd.first, EVFILT_READ, EV_ADD, 0, 0, 0);
num_events = kevent(loop.m_kqueue, in_events.data(), in_events.size(),
out_events, llvm::array_lengthof(out_events), nullptr);
if (num_events < 0)
return Status("kevent() failed with error %d\n", num_events);
return Status();
}
It works on FreeBSD and MacOSX, however it broke on NetBSD.
Culprit line:
EV_SET(&in_events[i++], fd.first, EVFILT_READ, EV_ADD, 0, 0, 0);
FreeBSD defined EV_SET() as a macro this way:
#define EV_SET(kevp_, a, b, c, d, e, f) do { \
struct kevent *kevp = (kevp_); \
(kevp)->ident = (a); \
(kevp)->filter = (b); \
(kevp)->flags = (c); \
(kevp)->fflags = (d); \
(kevp)->data = (e); \
(kevp)->udata = (f); \
} while(0)
NetBSD version was different:
#define EV_SET(kevp, a, b, c, d, e, f) \
do { \
(kevp)->ident = (a); \
(kevp)->filter = (b); \
(kevp)->flags = (c); \
(kevp)->fflags = (d); \
(kevp)->data = (e); \
(kevp)->udata = (f); \
} while (/* CONSTCOND */ 0)
This resulted in heap damage, as keyp was incremented every time a value was
assigned to (keyp)->.
Without GCC asan and ubsan tools, finding this bug would be much more time consuming, as the random memory corruption was affecting unrelated lambda function in a different part of the code.
To use the GCC sanitizers with packages from pkgsrc, on NetBSD-current, one has to use one or both of these lines:
_WRAP_EXTRA_ARGS.CXX+= -fno-omit-frame-pointer -O0 -g -ggdb -U_FORTIFY_SOURCE -fsanitize=address -fsanitize=undefined -lasan -lubsan CWRAPPERS_APPEND.cxx+= -fno-omit-frame-pointer -O0 -g -ggdb -U_FORTIFY_SOURCE -fsanitize=address -fsanitize=undefined -lasan -lubsan
While there, I have fixed another - generic - bug in the LLVM headers. The class Triple constructor hadn't initialized the SubArch field, which upsetting the GCC address sanitizer. It was triggered in LLDB in the following code:
void ArchSpec::Clear() {
m_triple = llvm::Triple();
m_core = kCore_invalid;
m_byte_order = eByteOrderInvalid;
m_distribution_id.Clear();
m_flags = 0;
}
I have filed a patch for review to address this.
The bad
Unfortunately this is not the full story and there is further mandatory work.
LLDB acceleration
The EV_SET() bug broke upstream LLDB over a month ago, and during this period the debugger was significantly accelerated and parallelized. It is difficult to declare it definitely, but it might be the reason why the tracer's runtime broke due to threading desynchronization. LLDB behaves differently when run standalone, under ktruss(1) and under gdb(1) - the shared bug is that it always fails in one way or another, which isn't trivial to debug.
The ugly
There are also unpleasant issues at the core of the Operating System.
Kernel troubles
Another bug with single-step functions that affects another aspect of correctness - this time with reliable execution of a program - is that processes die in non-deterministic ways when single-stepped. My current impression is that there is no appropriate translation between process and thread (LWP) states under a debugger.
These issues are sibling problems to unreliable PT_RESUME and PT_SUSPEND.
In order to be able to appropriately address this, I have diligently studied this month the Solaris Internals book to get a better image of the design of the NetBSD kernel multiprocessing, which was modeled after this commercial UNIX.
Plan for the next milestone
The current troubles can be summarized as data races in the kernel and at the same time in LLDB. I have decided to port the LLVM sanitizers, as I require the Thread Sanitizer (tsan). Temporarily I have removed the code for tracing processes with multiple threads to hide the known kernel bugs and focus on the LLDB races.
Unfortunately LLDB is not easily bisectable (build time of the LLVM+Clang+LLDB stack, number of revisions), therefore the debugging has to be performed on the most recent code from upstream trunk.
This work was sponsored by The NetBSD Foundation.
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
It turned out that more bugs were unveiled and this is not the final report on LLDB.
The good
Besides the greater enhancements this month I performed a cleanup in the ATF ptrace(2) tests again. Additionally I have managed to unbreak the LLDB Debug build and to eliminate compiler warnings in the NetBSD Native Process Plugin.
It is worth noting that LLVM can run tests on NetBSD again, the patch in gtest/LLVM has been installed by Joerg Sonnenberg and a more generic one has been submitted to the upstream googletest repository. There was also an improvement in ftruncate(2) on the LLVM side (authored by Joerg).
Since LLD (the LLVM linker) is advancing rapidly, it improved support for NetBSD and it can link a functional executable on NetBSD. I submitted a patch to stop crashing it on startup anymore. It was nearly used for linking LLDB/NetBSD and it spotted a real linking error... however there are further issues that need to be addressed in the future. Currently LLD is not part of the mainline LLDB tasks - it's part of improving the work environment. This linker should reduce the linking time - compared to GNU linkers - of LLDB by a factor of 3x-10x and save precious developer time. As of now LLDB linking can take minutes on a modern amd64 machine designed for performance.
Kernel correctness
I have researched (in pkgsrc-wip) initial support for multiple threads in the NetBSD Native Process Plugin. This code revealed - when running the LLDB regression test-suite - new kernel bugs. This unfortunately affects the usability of a debugger in a multithread environment in general and explains why GDB was never doing its job properly in such circumstances.
One of the first errors was asserting kernel panic with PT_*STEP, when a debuggee has more than a single thread. I have narrowed it down to lock primitives misuse in the do_ptrace() kernel code. The fix has been committed.
LLDB and userland correctness
LLDB introduced support for kevent(2) and it contains the following function:
Status MainLoop::RunImpl::Poll() {
in_events.resize(loop.m_read_fds.size());
unsigned i = 0;
for (auto &fd : loop.m_read_fds)
EV_SET(&in_events[i++], fd.first, EVFILT_READ, EV_ADD, 0, 0, 0);
num_events = kevent(loop.m_kqueue, in_events.data(), in_events.size(),
out_events, llvm::array_lengthof(out_events), nullptr);
if (num_events < 0)
return Status("kevent() failed with error %d\n", num_events);
return Status();
}
It works on FreeBSD and MacOSX, however it broke on NetBSD.
Culprit line:
EV_SET(&in_events[i++], fd.first, EVFILT_READ, EV_ADD, 0, 0, 0);
FreeBSD defined EV_SET() as a macro this way:
#define EV_SET(kevp_, a, b, c, d, e, f) do { \
struct kevent *kevp = (kevp_); \
(kevp)->ident = (a); \
(kevp)->filter = (b); \
(kevp)->flags = (c); \
(kevp)->fflags = (d); \
(kevp)->data = (e); \
(kevp)->udata = (f); \
} while(0)
NetBSD version was different:
#define EV_SET(kevp, a, b, c, d, e, f) \
do { \
(kevp)->ident = (a); \
(kevp)->filter = (b); \
(kevp)->flags = (c); \
(kevp)->fflags = (d); \
(kevp)->data = (e); \
(kevp)->udata = (f); \
} while (/* CONSTCOND */ 0)
This resulted in heap damage, as keyp was incremented every time a value was
assigned to (keyp)->.
Without GCC asan and ubsan tools, finding this bug would be much more time consuming, as the random memory corruption was affecting unrelated lambda function in a different part of the code.
To use the GCC sanitizers with packages from pkgsrc, on NetBSD-current, one has to use one or both of these lines:
_WRAP_EXTRA_ARGS.CXX+= -fno-omit-frame-pointer -O0 -g -ggdb -U_FORTIFY_SOURCE -fsanitize=address -fsanitize=undefined -lasan -lubsan CWRAPPERS_APPEND.cxx+= -fno-omit-frame-pointer -O0 -g -ggdb -U_FORTIFY_SOURCE -fsanitize=address -fsanitize=undefined -lasan -lubsan
While there, I have fixed another - generic - bug in the LLVM headers. The class Triple constructor hadn't initialized the SubArch field, which upsetting the GCC address sanitizer. It was triggered in LLDB in the following code:
void ArchSpec::Clear() {
m_triple = llvm::Triple();
m_core = kCore_invalid;
m_byte_order = eByteOrderInvalid;
m_distribution_id.Clear();
m_flags = 0;
}
I have filed a patch for review to address this.
The bad
Unfortunately this is not the full story and there is further mandatory work.
LLDB acceleration
The EV_SET() bug broke upstream LLDB over a month ago, and during this period the debugger was significantly accelerated and parallelized. It is difficult to declare it definitely, but it might be the reason why the tracer's runtime broke due to threading desynchronization. LLDB behaves differently when run standalone, under ktruss(1) and under gdb(1) - the shared bug is that it always fails in one way or another, which isn't trivial to debug.
The ugly
There are also unpleasant issues at the core of the Operating System.
Kernel troubles
Another bug with single-step functions that affects another aspect of correctness - this time with reliable execution of a program - is that processes die in non-deterministic ways when single-stepped. My current impression is that there is no appropriate translation between process and thread (LWP) states under a debugger.
These issues are sibling problems to unreliable PT_RESUME and PT_SUSPEND.
In order to be able to appropriately address this, I have diligently studied this month the Solaris Internals book to get a better image of the design of the NetBSD kernel multiprocessing, which was modeled after this commercial UNIX.
Plan for the next milestone
The current troubles can be summarized as data races in the kernel and at the same time in LLDB. I have decided to port the LLVM sanitizers, as I require the Thread Sanitizer (tsan). Temporarily I have removed the code for tracing processes with multiple threads to hide the known kernel bugs and focus on the LLDB races.
Unfortunately LLDB is not easily bisectable (build time of the LLVM+Clang+LLDB stack, number of revisions), therefore the debugging has to be performed on the most recent code from upstream trunk.
This work was sponsored by The NetBSD Foundation.
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
the repository conversion setup for NetBSD CVS -> Fossil -> Git has found a new home. Ironically, on former cvs.NetBSD.org hardware. This provides a somewhat faster conversion cycle as well as removing anoncvs.NetBSD.org from the process. This should avoid occasional problems with incomplete syncs. Two other changes have been applied at the same time:
- The Fossil repositories have moved to using the SHA512 checksums internally. To avoid accidents, a new project code is used. This requires Fossil 2.x.
- The Git repositories map user names to @NetBSD.org addresses (src, xsrc) or @pkgsrc.org addresses (pkgsrc). This allows consolidation on github with user accounts, assuming you have the corresponding addresses as primary or secondary address.
The new locations for the repositories are:
The old conversions will be provided for the near future, but likely stop toward the end of the month.
A special thanks goes to Petra and Christos from the admin team for the assistance with the machine setup and low-level CVS clean-ups.
Joerg
-- source https://mail-index.netbsd.org/tech-repository/2017/06/10/msg000637.html
the repository conversion setup for NetBSD CVS -> Fossil -> Git has found a new home. Ironically, on former cvs.NetBSD.org hardware. This provides a somewhat faster conversion cycle as well as removing anoncvs.NetBSD.org from the process. This should avoid occasional problems with incomplete syncs. Two other changes have been applied at the same time:
- The Fossil repositories have moved to using the SHA512 checksums internally. To avoid accidents, a new project code is used. This requires Fossil 2.x.
- The Git repositories map user names to @NetBSD.org addresses (src, xsrc) or @pkgsrc.org addresses (pkgsrc). This allows consolidation on github with user accounts, assuming you have the corresponding addresses as primary or secondary address.
The new locations for the repositories are:
The old conversions will be provided for the near future, but likely stop toward the end of the month.
A special thanks goes to Petra and Christos from the admin team for the assistance with the machine setup and low-level CVS clean-ups.
Joerg
-- source https://mail-index.netbsd.org/tech-repository/2017/06/10/msg000637.html
Born in 1969, he developed a passion for computer science in his youth but chose to study biology. He then attended a bioinformatics training at the Pasteur Institute in 1997 at the end of which he got hired in the scientific IT department. He has continuously worked there since. Not only was he skilled in biology but also in system administration, databases, and programming.
He was introduced to NetBSD in 1999 and quickly became a dedicated user and hacker. He became a NetBSD developer in 2007 for working on Linux/i386 (32 bits) emulation on NetBSD/amd64 (64 bits).
Nicolas leaves a wife and three children.
Memorial note submitted by Marc Baudoin.
Born in 1969, he developed a passion for computer science in his youth but chose to study biology. He then attended a bioinformatics training at the Pasteur Institute in 1997 at the end of which he got hired in the scientific IT department. He has continuously worked there since. Not only was he skilled in biology but also in system administration, databases, and programming.
He was introduced to NetBSD in 1999 and quickly became a dedicated user and hacker. He became a NetBSD developer in 2007 for working on Linux/i386 (32 bits) emulation on NetBSD/amd64 (64 bits).
Nicolas leaves a wife and three children.
Memorial note submitted by Marc Baudoin.