/proc/#/ctl).
LLVM
The majority of the work has been done in the LLVM projects.
The developed features are not production ready and they will need productization in the future. There are still issues with paths mismatch ("netbsd" vs "netbsd8.99" vs "netbsd8.99.1") when looking for NetBSD specific support for the compiler-rt features. There is also a need for integration with pkgsrc, as not everything behaves sanely (conflicts with wrappers). The tools are also restricted to be built with the Clang compiler, as GCC support is currently broken. I also noted that the sanitizers behave wrongly in the standalone build (out of the LLVM sources).
I expect to sort out the mentioned problems after finishing LLDB.
LLVM JIT
There is ongoing discussion with the LLVM community about new JIT API that will be compatible with NetBSD PaX MPROTECT. There have been developed and introduced cleanups in the code (like better error handling templates) in order to prepare a draft of new API.
ASAN
All local code for ASan has been merged upstream. This includes NetBSD patches in LLVM, Clang and compiler-rt.
All but two (one on i386 version and the other on amd64) tests (check-asan) pass.
UBSAN
Similarly to ASan, UBsan has been fully upstreamed. All tests (check-ubsan) pass.
SafeStack
SafeStack is a software security hardening technique that creates two stacks: one for data that needs to be kept safe, such as return addresses and function pointers; and an unsafe stack for everything else.
With PaX ASLR (Address Space Layout Randomization) and PaX MPROTECT (mprotect(2) restrictions) SafeStack is an excellent candidate for inclusion in pkgsrc.
Core programs could be hardened as well, but the shortcoming of SafeStack for basesystem utilities is pulling in additional dependencies like libpthread on every executable.
Using SafeStack adds marginal overhead.
libFuzzer
Citing the project page, LibFuzzer is an in-process, coverage-guided, evolutionary fuzzing engine.
LibFuzzer is linked with the library under test, and feeds fuzzed inputs to the library via a specific fuzzing entry point (aka 'target function'); the fuzzer then tracks which areas of the code are reached, and generates mutations on the corpus of input data in order to maximize the code coverage. The code coverage information for libFuzzer is provided by LLVM's SanitizerCoverage instrumentation.
This functionality still requires more sanitizers to get aboard and is now of restricted functionality.
TSAN
Part of the TSan code has been upstreamed. However, a functional port isn't finished yet.
The current issues are: proper process memory map handling and NetBSD specific setjmp(3)-like functions support.
LSAN
I was also working on LSan. This sanitizer already builds and appears to be quite completed, however there is work needed for the implementation of StopTheWorld() function to self-introspect the process and threads. I'm researching a new kernel API for this purpose, but it might wait till the end of LLDB porting.
MSAN
So far I have not been working on the MSan specific bits. The majority of the code has been upstreamed for this sanitizer in the common sanitizer framework, the proper handling of the NetBSD specific process map is still to be done.
PROFILE
The profile library is used to collect coverage information. It already passes most of the tests, however it's not turned on, as upstream requested additional time to be spent on the issues and it's not a priority right now.
NetBSD kernel
I've removed the filesystem tracing feature.
This is a legacy interface from 4.4BSD, and it was
introduced to overcome shortcomings of ptrace(2) at that time, which are
no longer relevant (performance). Today /proc/#/ctl offers a narrow
subset of ptrace(2) commands and is not applicable for modern
applications use beyond simplistic tracing scenarios.
This removal simplified kernel internals. Users are still able to
use all the other /proc files.
This change doesn't affect other procfs files and Linux compat
features within mount_procfs(8). /proc/#/ctl isn't available on Linux.
Plan for the next milestone
This month I will not work on a new development and I will focus on relax and taking part in EuroBSDCon in Paris. I will speak about the LLDB porting to NetBSD.
Long-term goals are finishing the basis sanitizers (msan, tsan) and switching back to LLDB porting. The sanitizers will be used to develop and debug the LLVM debugger. There is also integration with sanitizers in LLDB.
This work was sponsored by The NetBSD Foundation.
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
/proc/#/ctl).
LLVM
The majority of the work has been done in the LLVM projects.
The developed features are not production ready and they will need productization in the future. There are still issues with paths mismatch ("netbsd" vs "netbsd8.99" vs "netbsd8.99.1") when looking for NetBSD specific support for the compiler-rt features. There is also a need for integration with pkgsrc, as not everything behaves sanely (conflicts with wrappers). The tools are also restricted to be built with the Clang compiler, as GCC support is currently broken. I also noted that the sanitizers behave wrongly in the standalone build (out of the LLVM sources).
I expect to sort out the mentioned problems after finishing LLDB.
LLVM JIT
There is ongoing discussion with the LLVM community about new JIT API that will be compatible with NetBSD PaX MPROTECT. There have been developed and introduced cleanups in the code (like better error handling templates) in order to prepare a draft of new API.
ASAN
All local code for ASan has been merged upstream. This includes NetBSD patches in LLVM, Clang and compiler-rt.
All but two (one on i386 version and the other on amd64) tests (check-asan) pass.
UBSAN
Similarly to ASan, UBsan has been fully upstreamed. All tests (check-ubsan) pass.
SafeStack
SafeStack is a software security hardening technique that creates two stacks: one for data that needs to be kept safe, such as return addresses and function pointers; and an unsafe stack for everything else.
With PaX ASLR (Address Space Layout Randomization) and PaX MPROTECT (mprotect(2) restrictions) SafeStack is an excellent candidate for inclusion in pkgsrc.
Core programs could be hardened as well, but the shortcoming of SafeStack for basesystem utilities is pulling in additional dependencies like libpthread on every executable.
Using SafeStack adds marginal overhead.
libFuzzer
Citing the project page, LibFuzzer is an in-process, coverage-guided, evolutionary fuzzing engine.
LibFuzzer is linked with the library under test, and feeds fuzzed inputs to the library via a specific fuzzing entry point (aka 'target function'); the fuzzer then tracks which areas of the code are reached, and generates mutations on the corpus of input data in order to maximize the code coverage. The code coverage information for libFuzzer is provided by LLVM's SanitizerCoverage instrumentation.
This functionality still requires more sanitizers to get aboard and is now of restricted functionality.
TSAN
Part of the TSan code has been upstreamed. However, a functional port isn't finished yet.
The current issues are: proper process memory map handling and NetBSD specific setjmp(3)-like functions support.
LSAN
I was also working on LSan. This sanitizer already builds and appears to be quite completed, however there is work needed for the implementation of StopTheWorld() function to self-introspect the process and threads. I'm researching a new kernel API for this purpose, but it might wait till the end of LLDB porting.
MSAN
So far I have not been working on the MSan specific bits. The majority of the code has been upstreamed for this sanitizer in the common sanitizer framework, the proper handling of the NetBSD specific process map is still to be done.
PROFILE
The profile library is used to collect coverage information. It already passes most of the tests, however it's not turned on, as upstream requested additional time to be spent on the issues and it's not a priority right now.
NetBSD kernel
I've removed the filesystem tracing feature.
This is a legacy interface from 4.4BSD, and it was
introduced to overcome shortcomings of ptrace(2) at that time, which are
no longer relevant (performance). Today /proc/#/ctl offers a narrow
subset of ptrace(2) commands and is not applicable for modern
applications use beyond simplistic tracing scenarios.
This removal simplified kernel internals. Users are still able to
use all the other /proc files.
This change doesn't affect other procfs files and Linux compat
features within mount_procfs(8). /proc/#/ctl isn't available on Linux.
Plan for the next milestone
This month I will not work on a new development and I will focus on relax and taking part in EuroBSDCon in Paris. I will speak about the LLDB porting to NetBSD.
Long-term goals are finishing the basis sanitizers (msan, tsan) and switching back to LLDB porting. The sanitizers will be used to develop and debug the LLVM debugger. There is also integration with sanitizers in LLDB.
This work was sponsored by The NetBSD Foundation.
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
You can find Mercurial versions of src, pkgsrc and xsrc under
and
The same rules as for the fossil and github repositories apply, i.e. there may be occasional glitches and if it becomes too bad, they might be recreated from scratch.
See more information in the posted thread to tech-repository.
You can find Mercurial versions of src, pkgsrc and xsrc under
and
The same rules as for the fossil and github repositories apply, i.e. there may be occasional glitches and if it becomes too bad, they might be recreated from scratch.
See more information in the posted thread to tech-repository.
Traditionally, developers in distributions like NetBSD merge 3rd party sources upstream once in a while with major release bumps, like switching from GCC 4.8.x to GCC 5.x. The time frame between the releases can be several months or a few years. This appears as a one-time effort from time to time, however with each software revision the code starts to rot on undermaintained targets. This results in local compatibility patches, which are rarely ready or applicable for upstream and thus detached from the development progress. Upstream developers tend to assume that minimal activity from such projects is a result of having no users and not verifying new code on them.
A good way to improve the situation and ensure quality of software that would shorten developers' time and cost, to deepen relations between NetBSD developers and upstream 3rd party software is to attach a build cluster node within the testing infrastructure.
The shortcoming of this approach is that it requires hardware, bandwidth and admin maintenance. The advantage is closer and better support for the NetBSD platform directly from the 3rd party software developers and the immediate detection of regressions, further reducing development time.
After the process of restoration the NetBSD support within the GDB and binutils projects, there is a new member in the GDB's cluster farm that verifies correct build status on NetBSD/amd64. This bot is hosted within The NetBSD Foundation's internal infrastructure.
The immediate follow up is to turn on --enable-targets=all, which will build all the available backends. Only a few more patches are needed to achieve this milestone.
Next steps involve extending this bot to verify other projects within the shared binutils-gdb repository. This includes GNU binutils itself, ld, gold (after adding appropriate platform support), gas, sim and gprof.
The ultimate goal is to enable execution of all tests for each new binutils-gdb commit in the upstream repository. This must be preceded by accomplishing the ongoing contracted task sponsored by The NetBSD Foundation - to port the LLDB debugger to NetBSD, as the LLVM debugger opens a door for new software from the same field.
Traditionally, developers in distributions like NetBSD merge 3rd party sources upstream once in a while with major release bumps, like switching from GCC 4.8.x to GCC 5.x. The time frame between the releases can be several months or a few years. This appears as a one-time effort from time to time, however with each software revision the code starts to rot on undermaintained targets. This results in local compatibility patches, which are rarely ready or applicable for upstream and thus detached from the development progress. Upstream developers tend to assume that minimal activity from such projects is a result of having no users and not verifying new code on them.
A good way to improve the situation and ensure quality of software that would shorten developers' time and cost, to deepen relations between NetBSD developers and upstream 3rd party software is to attach a build cluster node within the testing infrastructure.
The shortcoming of this approach is that it requires hardware, bandwidth and admin maintenance. The advantage is closer and better support for the NetBSD platform directly from the 3rd party software developers and the immediate detection of regressions, further reducing development time.
After the process of restoration the NetBSD support within the GDB and binutils projects, there is a new member in the GDB's cluster farm that verifies correct build status on NetBSD/amd64. This bot is hosted within The NetBSD Foundation's internal infrastructure.
The immediate follow up is to turn on --enable-targets=all, which will build all the available backends. Only a few more patches are needed to achieve this milestone.
Next steps involve extending this bot to verify other projects within the shared binutils-gdb repository. This includes GNU binutils itself, ld, gold (after adding appropriate platform support), gas, sim and gprof.
The ultimate goal is to enable execution of all tests for each new binutils-gdb commit in the upstream repository. This must be preceded by accomplishing the ongoing contracted task sponsored by The NetBSD Foundation - to port the LLDB debugger to NetBSD, as the LLVM debugger opens a door for new software from the same field.
We welcome Pierre Pronchery and Makoto Fujiwara to the 2017 Board of Directors. We look forward to working with you!
We appreciate all of the wonderful work that S.P.Zeidler and Erik Berls have done on the board during their time as directors, and are grateful for their excellent service to the foundation.
Thank you to all members participating by nominating candidates and voting on the slate.
Respectfully submitted for The NetBSD Foundation,
William J. Coldwell
President/Chairperson
We welcome Pierre Pronchery and Makoto Fujiwara to the 2017 Board of Directors. We look forward to working with you!
We appreciate all of the wonderful work that S.P.Zeidler and Erik Berls have done on the board during their time as directors, and are grateful for their excellent service to the foundation.
Thank you to all members participating by nominating candidates and voting on the slate.
Respectfully submitted for The NetBSD Foundation,
William J. Coldwell
President/Chairperson