The NetBSD support is no longer visibly lacking behind Linux in sanitizers, although there are still failing tests on NetBSD that are not observed on Linux. On the other hand there are features working on NetBSD that are not functional on Linux, like sanitizing programs during early initialization process of OS (this is caused by /proc dependency on Linux that is mounted by startup programs, while NetBSD relies on sysctl(3) interfaces that is always available).
Changes in compiler-rt
A number of patches have been merged upstream this month. Part of the upstreamed code has been originally written by Yang Zheng during GSoC-2018. My work was about cleaning the patches, applying comments from upstream review and writing new regression tests. Some of the changes were newly written over the past month, like background thread support in ASan/NetBSD or NetBSD compatible per-thread cleanup destructors in ASan and MSan.
Additionally, I've also ported the LLVM profile (--coverage) feature to NetBSD and investigated the remaining failing tests. Part of the failures were caused by already a copy of older runtime inside the NetBSD libc (ABIv2 libc vs ABIv4 current). The remaining two tests are affected by incompatible behavior of atexit(3) in Dynamic Shared Objects. Replacing the functionality with destructors didn't work and I've marked these tests as expected failures and moved on.
Changes in compiler-rt:
- 3d5a3668a Reenable hard_rss_limit_mb_test.cc for android-26
- decb231c3 Add support for background thread on NetBSD in ASan
- 3ebc523bb Fix a mistake in previous
- df1f46250 Update NetBSD ioctl(2) entries with 8.99.28
- 2de4ff725 Enable asan_and_llvm_coverage_test.cc for NetBSD
- 4d9ac421b Reimplement Thread Static Data MSan routines with TLS
- f4a536af4 Adjust NetBSD/sha2.cc to be portable to more environments
- 21bd4bd9f Adjust NetBSD/md2.cc to be portable to more environments
- 1f2d0324e Adjust NetBSD/md[45].cc to be portable to more environments
- 2835fe7cf Add support for LLVM profile for NetBSD
- 52af2fe7c Reimplement Thread Static Data ASan routines with TLS
- 79d385b5c Improve the comment in previous
- 384486fa4 Expand TSan sysroot workaround to NetBSD
- 81e370964 Enable test/msan/pthread_getname_np.cc for NetBSD
- 19e2af50e Enable SANITIZER_INTERCEPT_PTHREAD_GETNAME_NP for NetBSD
- 5088473c7 Fix internal_sleep() for NetBSD
- cd24f2f94 Mark interception_failure_test.cc as passing for NetBSD and asan-dynamic-runtime
- ececda6ca Set shared_libasan_path in lit tests for NetBSD
- 429bc2d51 Add a new interceptors for cdbr(3) and cdbw(3) API from NetBSD
- 71553eb50 Add new interceptors for vis(3) API in NetBSD
- a3e78a793 Add data types needed for md2(3)/NetBSD interceptors
- 0ddb9d099 Add interceptors for the sha2(3) from NetBSD
- 9e2ff43a6 Add interceptors for md2(3) from NetBSD
- 42ac31ee6 Add new interceptors for FILE repositioning stream
- 8627b4b30 Fix a typo in the strtoi test
- 660f7441b Revert a chunk of previous change in sanitizer_platform_limits_netbsd.h
- 9a087462c Add interceptors for md5(3) from NetBSD
- 086caf6a2 Add interceptors for the rmd160(3) from NetBSD
- 8f77a2e89 Add interceptors for the md4(3) from NetBSD
- 27af3db52 Add interceptors for the sha1(3) from NetBSD
- 6b9f7889b Add interceptors for the strtoi(3)/strtou(3) from NetBSD
- 195044df9 Add a new interceptors for statvfs1(2) and fstatvfs1(2) from NetBSD
- f0835eb01 Add a new interceptor for fparseln(3) from NetBSD
- 11ecbe602 Add new interceptor for strtonum(3)
- 19b47fcc0 Remove XFAIL in get_module_and_offset_for_pc.cc for NetBSD-MSan
- b3a7f1d78 Add a new interceptor for modctl(2) from NetBSD
- 39c2acc81 Add a new interceptor for nl_langinfo(3) from NetBSD
- 2eb9a4c53 Update GET_LINK_MAP_BY_DLOPEN_HANDLE() for NetBSD x86
- e8dd644be Improve the regerror(3) interceptor
- dd939986a Add interceptors for the sysctl(3) API family from NetBSD
- 67639f9cc Add interceptors for the fts(3) API family from NetBSD
- c8fae517a Add new interceptor for regex(3) in NetBSD
Part of the new code has been quickly ported from NetBSD to other Operating Systems, mostly FreeBSD, and when applicable to Darwin and Linux.
Changes in other LLVM projects
In order to eliminate local diffs in other LLVM projects, I've upstreamed two patches to LLVM and two to OpenMP. I've also helped other BSDs to get their support in OpenMP (DragonFlyBSD and OpenBSD).
LLVM changes:
- 50df229c26a Add NetBSD support in needsRuntimeRegistrationOfSectionRange.
- 267dfed3ade Register kASan shadow offset for NetBSD/amd64
OpenMP changes:
- 67d037d Implement __kmp_is_address_mapped() for NetBSD
- 9761977 Implement __kmp_gettid() for NetBSD
- a72c79b Add OpenBSD support to OpenMP
- b3d05ab Add DragonFlyBSD support to OpenMP
NetBSD changes
I've introduced 5 changes to the NetBSD source tree over the past month, not counting updates to TODO lists.
- Raise the fill_vmentries() E2BIG limit from 1MB to 10MB
- Correct libproc_p.a in distribution sets
- compiler_rt: Update prepare-import.sh according to future updates
- Correct handling of minval > maxval in strtonum(3)
- Stop mangling __func__ for C++11 and newer
The first change is needed to handle large address space with sysctl(3) operation to retrieve the map. This feature is required in sanitizers and part of the tests were failing because within 1MB it wasn't possible to pass all the information about the process virtual map (mostly due to a large number of small allocations).
The second change was introduced to unbreak MKPROFILE=no build, I needed this during my work of porting the modern LLVM profile feature.
The third change is a preparation for import of compiler-rt sanitizers into the NetBSD distribution.
The forth change was a bug fix for strtonum(3) implementation in libc.
The fifth change was intended to reuse native compiler support for the __func__ compiler symbol.
Integration of LLVM sanitizers with the NetBSD basesystem
We are ready to push support for LLVM sanitizers into the NetBSD basesystem as all the needed patches have been merged. I've divided the remaining tests of integration of LLVM sanitizers into three milestones:
- Import compiler-rt sources into src/. A complete diff is pending for final acceptance in internal review.
- Integrate building of compiler-rt stanitizer under the MKLLVM=yes option. This has been made functional, but it needs polishing and submitting to internal review.
- Make MKSANITIZER available out of the box with the toolchain available in "./build.sh tools". This will be continuation of the previous point. All the MKSANITIZER patches independent from compiler type are already committed into the NetBSD distribution, however there will be likely some extra minor adaptation work here too.
Plan for the next milestone
Finish the integration of LLVM sanitizers with the NetBSD distribution.
This work was sponsored by The NetBSD Foundation.
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:
LLD is the link editor (linker) component of Clang toolchain. Its main advantage over GNU ld is much lower memory footprint, and linking speed. It is of specific interest to me since currently 8 GiB of memory are insufficient to link LLVM statically (which is the upstream default).
The first goal of LLD porting is to ensure that LLD can produce working NetBSD executables, and be used to build LLVM itself. Then, it is desirable to look into trying to build additional NetBSD components, and eventually into replacing /usr/bin/ld entirely with lld.
In this report, I would like to shortly summarize the issues I have found so far trying to use LLD on NetBSD.
DT_RPATH vs DT_RUNPATH
RPATH is used to embed a library search path in the executable. Since it takes precedence over default system library paths, it can be used both to specify the location of additional program libraries and to override system libraries.
Currently, RPATH can be embedded in executables using two tags: the “old” DT_RPATH tag and the “new” DT_RUNPATH tag. The existence of two tags comes from behavior exhibited by some operating systems (e.g. glibc systems): DT_RPATH used to take precedence over LD_LIBRARY_PATH, making it impossible to override the paths specified there. Therefore, a new DT_RUNPATH tag was added that comes after LD_LIBRARY_PATH in precedence. When both DT_RPATH and DT_RUNPATH are specified, the former is ignored.
On NetBSD, DT_RPATH does not take precedence over LD_LIBRARY_PATH. Therefore, there wasn't ever a need for DT_RUNPATH and the support for it (as alias to DT_RPATH) was added only very recently: on 2018-12-30.
Unlike GNU ld, LLD defaults to using “new” tag by default and therefore produces executables whose RPATHs do not work on older NetBSD versions. Given that using DT_RUNPATH on NetBSD has no real advantage, using the --disable-new-dtags option to suppress them is preferable.
More than two PT_LOAD segments
PT_LOAD segments are used to map executable image into the memory. Traditionally, GNU ld produces exactly two PT_LOAD segments: a RX text (code) segment, and a RW data segment. NetBSD dynamic loader (ld.elf_so) hardcodes the assumption of that design. However, lld sometimes produces an additional read-only data segment, causing the assertions to fail in the dynamic loader.
I have attempted to rewrite the memory mapping routine to allow for arbitrary number of segments. However, apparently my patch is just “a step in the wrong direction” and Joerg Sonnenberger is working on a proper fix.
Alternatively, LLD has a --no-rosegment option that can be used to suppress the additional segment and work around the problem.
Clang/LLD driver design issues
Both GCC and Clang use a design based on a front-end driver component. That is, the executable called directly by the user is a driver whose purpose is to perform initial command-line option and input processing, and run appropriate tools performing the actual work. Those tools may include the C preprocessor (cpp), C/C++ compiler (cc1), assembler, link editor (ld).
This follows the original UNIX principle of simple tools that perform a single task well, and a wrapper that combines those tools into complete workflows. Interesting enough, it makes it possible to keep all system-specific defaults and logic in a single place, without having to make every single tool aware of them. Instead, they are passed to those tools as command-line options.
This also accounts for simpler and more portable build system design. The gcc/clang driver provides a single high-level interface for performing a multitude of tasks, including compiling assembly files or linking executables. Therefore, the build system and the user do not need to be explicitly aware of low-level tooling and its usage. Not to mention it makes much easier to swap that tooling transparently.
For example, if you are linking an executable via the driver, it takes care of finding the appropriate link editor (and makes it easy to change it via -fuse-ld), preparing appropriate command-line options (e.g. if you do a -m32 multilib build, it sets the emulation for you) and passes necessary libraries to link (e.g. an appropriate standard C++ library when building a C++ program).
The clang toolchain considers LLD an explicit part of this workflow, and — unlike GNU ld — ld.lld is not really suitable for using stand-alone. For example, it does not include any standard search paths for libraries, expecting the driver to provide them in form of appropriate -L options. This way, all the logic responsible for figuring out the operating system used (including possible cross-compilation scenarios) and using appropriate paths is located in one component.
However, Joerg Sonnenberger disagrees with this and believes LLD should contain all the defaults necessary for it to be used stand-alone on NetBSD. Effectively, we have two conflicting designs: one where all logic is in clang driver, and the other where some of the logic is moved into LLD. At this moment, LLD is following the former assumption, and clang driver for NetBSD — the latter. As a result, neither using LLD directly nor via clang works out of the box on NetBSD; to use either, the user would have to pass all appropriate -L and -z options explicitly.
Fixing LLD to work with the current clang driver would require adding target awareness to LLD and changing a number of defaults for NetBSD based on the target used. However, LLD maintainer Rui Ueyama is opposed to introducing this extra logic specifically for NetBSD, and believes it should be added to the clang driver as for other platforms. On the other side, the NetBSD toolchain driver maintainer Joerg Sonnenberger blocks adding this to the driver. Therefore, we have reached a impasse that prevents LLD from working out of the box on NetBSD without local patches.
A work-in-progress implementation of the local target logic approach requested by Joerg can be seen in D56650. Afterwards, additional behavior can be enabled on NetBSD by using target triple properties such as in D56215 (which copies the libdir logic from clang).
For comparison, the same problem solved in a way consistent with other distributions (and rejected by Joerg) can be seen in D56932 (which updates D33726). However, in some cases it will require adding additional options to LLD (e.g. -z nognustack, D56554), and corresponding dummy switches in GNU ld.
Summary
At this point, it seems that using LLD for the majority of regular packages is a goal that can be achieved soon. The main blocker right now is the disagreement between developers on how to proceed. When we can resolve that and agree on a single way forward, most of the patches become trivial.
Sadly, at this point I really do not see any way to convince either of the sides. The problem was reported in May 2017, and in the same month a fix consistent with all other platforms was provided. However, it is being blocked and LLD can not work out of the box.
Hopefully, we will able to finally find a way forward that does not involve keeping upstream clang driver for NetBSD incompatible with upstream LLD, and carrying a number of patches to LLD locally to make it work.
The further goals include attempting to build the kernel and complete userland using LLD, as well as further verifying compatibility of executables produced with various combinations of linker options (e.g. static PIE executables).
Today, I am here to report: Mission Accomplished!
It's been a long road, but we now have hardware-accelerated virtualization in the kernel! And while I had only initially planned to get Oracle VirtualBox working, I have with the help of the Intel HAXM engine (the same backend used for virtualization in Android Studio) and a qemu frontend, successfully managed to boot a range of mainstream operating systems.
With the advent of Intel's open-sourcing of their HAXM engine, we now have access to an important set of features:
- A BSD-style license.
- Support for multiple platforms: Windows, Darwin, Linux, and now NetBSD .
- HAXM is an Intel hardware assisted virtualization for their CPUs (VTx and EPT needed).
- Support for an arbitrary number of concurrent VMs. For simplicity's sake, NetBSD only supports 8, whereas Windows/Darwin/Linux support 64.
- An arbitary number of supported VCPUS per VM. All OSes support up to 64 VCPUs.
- ioctl(2) based API (/dev/HAX, /dev/haxmvm/vmXX, /dev/haxmvm/haxmvmXXvcpuYY).
- Implement non-intrusively as part of the kernel, rather than as an out-of-tree, standalone executable kernel module.
- Default compatibility with qemu as a frontend.
- Active upstream support from Intel, which is driven by commercial needs.
- Optimized for desktop scenarios.
- Probably the only open-source cross-OS virtualization engine.
- An active and passionate community that's dedicated to keep improving it.
As well as a few of HAXM's downsides:
- No AMD (SVM) support (althought there are community plans to implement it).
- No support for non-x86 architectures.
- Need for a relatively recent Intel CPU (EPT required).
- Not as flexible as KVM-like solutions for embedded use-cases or servers.
- Not as quick as KVM (probably 80% as fast as KVM).
If you'd like more details on HAXM, check out the following sites:
- https://www.qemu.org/2017/11/22/haxm-usage-windows/
- https://software.intel.com/en-us/articles/intel-hardware-accelerated-execution-manager-intel-haxm
- https://github.com/intel/haxm
Showcase
I've managed to boot and run several operating systems as guests running on HAXM/NetBSD, all of which are multi-core (2-4 VCPUs):
- NetBSD.
And as our motto goes, of course it runs NetBSD.
And, of course, my main priority as a NetBSD developer is achieving excellent support for NetBSD guest operating systems.
With the massive performance gains of hardware-accelerated virtualization,
it'll finally be possible to run kernel fuzzers and many other practical, real-world workloads on virtualized NetBSD systems.
[NetBSD at the bootloader]
[NetBSD kernel booting]
[NetBSD at a shell prompt]
[NetBSD with X Window session]
[NetBSD with X Window session]
[NetBSD guest and qemu's ACPI integration - emitted poweroff button press]
[NetBSD in the qemu's curses mode display, which I find convenient to use, especially on a headless remote computer] - Linux. When I pledged myself to not boot any other OS before accomplishing my goal, I was mostly thinking about
resigning from benefits of the Linux ecosystem (driver and software base).
There is still a selection of programs that I miss such as valgrind.. but with each week we are getting closer to fill the missing gaps.
Linux guests seem to work, however there is need to tune or disable IOAPIC in order to get it running
(I had to pass "noapic" as a Linux kernel option).
[ArchLinux at a bootloader]
[ArchLinux at a shell (ZSH) prompt]
[Ubuntu's installer]
[Ubuntu at a bootloader] - Windows. While I have no personal need nor use-case to run Windows, it's a must-have prestigious target for virtualization.
I've obtained a Windows 7 86 trial image from the official Microsoft webpage for testing purposes.
Windows 8.1 or newer and 64-bit version is still in development in HAXM.
[Windows 7 booting]
[Windows 7 welcome message]
[Windows 7 running]
[Windows 7 Control Panel]
[Windows 7 multitasking]
[Windows 7 MS Paint] - DragonflyBSD. I was prompted to test this FreeBSD derivation by a developer of this OS and it just worked flawlessly.
[DragonflyBSD at a bootloader]
[DragonflyBSD at a shell prompt] - FreeDOS. It seems to just work, but I have no idea what I can use it for.
[FREEDOS at an installer]
[FREEDOS at a command line]
- Android. The kernel booting seems fine (with "noapic"), but later during distribution load it freezes the host computer.
[Android bootloader]
[Android just before the host crash] - FreeBSD. Hangs during the booting process.
[FreeBSD hanging at boot]
Summary and future plans
One thing I must clarify, since I'm frequently asked about it, is that HAXM/NetBSD does not attempt to compete with the NVMM (NetBSD Virtual Machine Monitor) work being done by Maxime Villard. I'm primarily doing this for my own educational purposes, and because I find reaching feature-parity with other open-source projects is important work. Additionally, NVMM only has AMD CPU support, whereas I'm primarily a user of the Intel x86 platform, and thus, so is HAXM/NetBSD. The Intel port of NVMM and NVMM in general is still in development, and this means that HAXM is probably the first solution that has ever successfuly managed to run Windows on NetBSD (has anyone done it with Xen before?)
I will keep working on this project in my spare time and try to correct IOAPIC issues on Linux, hangs during FreeBSD's boot process, and Android crashes.
Most of the NetBSD-specific patches for qemu and Intel HAXM have already been merged upstream. And after this process has been completed, there are plans to make it available in pkgsrc. There's also at least one kernel-level workaround for HAXM behavior related to FPU state, which triggers an assert due to an abnormal condition. For this to be amended, fixes would have to land upstream into HAXM's code.
Subnote
Althought I confess that I've been playing with OpenVMS/VAX in SIMH as I have got a hobbyist license, but on the other hand it's hardly to be treated as a competition to NetBSD. Another exception was during a DTrace tutorial during EuroBSDCon 2017 in Paris, where I had to boot and use a FreeBSD image shared by the lecturer.