NetBSD Wiki/blog/archives/2017/ 02
martin (NetBSD Blog) Firefox 51 on sparc64 - we did not hit the wall yet

Background

I am using a sparc64 Sun Blade 2500 (silver) as a desktop machine - for my pretty light "desktop" needs. Besides the usual developer tools (editors, compilers, subversion, hg, git) and admin stuff (all text based) I need mpg123 and mserv for music queues, Gimp for image manipulation and of course Firefox.

Recently I updated all my installed pkgs to pkgsrc-current and as usual the new Firefox version failed to build.

Fortunately the issues were minor, as they all had been handled upstream for Firefox 52 already, all I needed to do was back-porting a few fixes. This made the pkg build, but after a few minutes of test browsing, it crashed. Not surprisingly this was reproducible, any web site trying to play audio triggered it. A bit surprising though: the same happened on an amd64 machine I tried next. After a bit digging the bug was easy to fix, and upstream already took the fix and committed it to the libcubeb repository.

Success!

So I am now happily editing this post using Firefox 51 on the Blade 2500.

I saw one crash in two days of browsing, but unfortunately could not (yet) reproduce it (I have gdb attached now). There will be future pkg updates certainly.

Future Obstacles

You may have read elsewhere that Firefox will start to require a working Rust compiler to build.

This is a bit unfortunate, as Rust (while academically interesting) is right now not a very good implementation language if you care about portability. The only available compiler requires a working LLVM back end, which we are still debugging. Our auto-builds produce sparc sets with LLVM, but the result is not fully working (due to what we believe being code gen bugs in LLVM). It seems we need to fix this soon (which would be good anyway, independent of the Rust issue). Besides the back end, only very recently traces of sparc64 support popped up in Rust. However, we still have a few firefox versions time to get it all going. I am optimistic.

Another upcoming change is that Cairo (currently used as 2D graphics back end, at least on sparc64) will be phased out and Skia will be the only supported software rendering target. Unfortunately Skia does (as of now) not support any big endian machine at all. I am looking for help getting Skia to work on big endian hardware in general, and sparc64 in particular.

Alternatives

Just in case, I tested a few other browsers and (so far) they all failed:
  • NetSurf
    Nice, small, has a few "tweaks" and does not yet support JavaScript good enough for many sites
  • MidoriThey call it "lightweight" but it is based on WebKit, which alone is a few times more heavy than all of Firefox. It crashes immediately at startup on sparc64 (I am investigating, but with low priority - actually I had to replace the hard disk in my machine to make enough room for the debug object files for WebKit - it takes ~20GB)

So, while it is a bit of a struggle to keep a modern browser working on my favorite odd-ball architecture, it seems we will get at least to the Firefox 52 ESR release, and that should give us enough time to get Rust working and hopefully continue with Firefox.

Posted Wednesday afternoon, February 8th, 2017 Tags:
martin (NetBSD Blog) Firefox 51 on sparc64 - we did not hit the wall yet

Background

I am using a sparc64 Sun Blade 2500 (silver) as a desktop machine - for my pretty light "desktop" needs. Besides the usual developer tools (editors, compilers, subversion, hg, git) and admin stuff (all text based) I need mpg123 and mserv for music queues, Gimp for image manipulation and of course Firefox.

Recently I updated all my installed pkgs to pkgsrc-current and as usual the new Firefox version failed to build.

Fortunately the issues were minor, as they all had been handled upstream for Firefox 52 already, all I needed to do was back-porting a few fixes. This made the pkg build, but after a few minutes of test browsing, it crashed. Not surprisingly this was reproducible, any web site trying to play audio triggered it. A bit surprising though: the same happened on an amd64 machine I tried next. After a bit digging the bug was easy to fix, and upstream already took the fix and committed it to the libcubeb repository.

Success!

So I am now happily editing this post using Firefox 51 on the Blade 2500.

I saw one crash in two days of browsing, but unfortunately could not (yet) reproduce it (I have gdb attached now). There will be future pkg updates certainly.

Future Obstacles

You may have read elsewhere that Firefox will start to require a working Rust compiler to build.

This is a bit unfortunate, as Rust (while academically interesting) is right now not a very good implementation language if you care about portability. The only available compiler requires a working LLVM back end, which we are still debugging. Our auto-builds produce sparc sets with LLVM, but the result is not fully working (due to what we believe being code gen bugs in LLVM). It seems we need to fix this soon (which would be good anyway, independent of the Rust issue). Besides the back end, only very recently traces of sparc64 support popped up in Rust. However, we still have a few firefox versions time to get it all going. I am optimistic.

Another upcoming change is that Cairo (currently used as 2D graphics back end, at least on sparc64) will be phased out and Skia will be the only supported software rendering target. Unfortunately Skia does (as of now) not support any big endian machine at all. I am looking for help getting Skia to work on big endian hardware in general, and sparc64 in particular.

Alternatives

Just in case, I tested a few other browsers and (so far) they all failed:
  • NetSurf
    Nice, small, has a few "tweaks" and does not yet support JavaScript good enough for many sites
  • MidoriThey call it "lightweight" but it is based on WebKit, which alone is a few times more heavy than all of Firefox. It crashes immediately at startup on sparc64 (I am investigating, but with low priority - actually I had to replace the hard disk in my machine to make enough room for the debug object files for WebKit - it takes ~20GB)

So, while it is a bit of a struggle to keep a modern browser working on my favorite odd-ball architecture, it seems we will get at least to the Firefox 52 ESR release, and that should give us enough time to get Rust working and hopefully continue with Firefox.

Posted Wednesday afternoon, February 8th, 2017 Tags:
Kamil Rytarowski (NetBSD Blog) The first patch-bulk upstreamed to LLDB
The LLVM project is a quickly moving target, this also applies to the LLVM debugger -- LLDB. It's actively used in several first-class operating systems, while - thanks to my spare time dedication - NetBSD joined the LLDB club in 2014, only lately the native support has been substantially improved and the feature set is quickly approaching the support level of Linux and FreeBSD. During this work 12 patches were committed to upstream, 12 patches were submitted to review, 11 new ATF were tests added, 2 NetBSD bugs filed and several dozens of commits were introduced in pkgsrc-wip, reducing the local patch set to mostly Native Process Plugin for NetBSD.

What has been done in NetBSD

1. Triagged issues of ptrace(2) in the DTrace/NetBSD support

Chuck Silvers works on improving DTrace in NetBSD and he has detected an issue when tracer signals are being ignored in libproc. The libproc library is a compatibility layer for DTrace simulating /proc capabilities on the SunOS family of systems.

I've verified that the current behavior of signal routing is incorrect. The NetBSD kernel correctly masks signals emitted by a tracee, not routing them to its tracer. On the other hand the masking rules in the inferior process blacklists signals generated by the kernel, which is incorrect and turns a debugger into a deaf listener. This is the case for libproc as signals were masked and software breakpoints triggering INT3 on i386/amd64 CPUs and SIGTRAP with TRAP_BRKP si_code wasn't passed to the tracer.

This isn't limited to turning a debugger into a deaf listener, but also a regular execution of software breakpoints requires: rewinding the program counter register by a single instruction, removing trap instruction and restoring the original instruction. When an instruction isn't restored and further code execution is pretty randomly affected, it resulted in execution anomalies and breaking of tracee.

A workaround for this is to disable signal masking in tracee.

Another drawback inspired by the DTrace code is to enhance PT_SYSCALL handling by introducing a way to distinguish syscall entry and syscall exit events. I'm planning to add dedicated si_codes for these scenarios. While there, there are users requesting PT_STEP and PT_SYSCALL tracing at the same time in an efficient way without involving heuristcs.

I've filed the mentioned bug:

I've added new ATF tests:

  • Verify that masking single unrelated signal does not stop tracer from catching other signals
  • Verify that masking SIGTRAP in tracee stops tracer from catching this raised signal
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching software breakpoints
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching single step trap
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching exec() breakpoint
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching PTRACE_FORK breakpoint
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching PTRACE_VFORK breakpoint
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching PTRACE_VFORK_DONE breakpoint
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching PTRACE_LWP_CREATE breakpoint
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching PTRACE_LWP_EXIT breakpoint

2. ELF Auxiliary Vectors

The ELF file format permits to transfer additional information for a process with a dedicated container of properties, it's named ELF Auxilary Vector. Every system has its dedicated way to read this information in a debugger from a tracee. The NetBSD approach is to transfer this vector with a ptrace(2) API PIOD_READ_AUXV. Our interface shares the API with OpenBSD. I filed a bug that our interface returns vector size of 8496 bytes, while OpenBSD has constant 64 bytes. It was diagnosed and fixed by Christos Zoluas that we were incorrectly counting bits and bytes and this enlarged the data streamlined. The bug was harmless and had no known side-effects besides large chunk of zeroed data.

There is also a prepared local patch extending NetBSD platform support to read information for this vector, it's primarily required for correct handling of PIE binaries. At the moment there is no interface similar to "info auxv" to the one from GDB. Unfortunately at the current stage, this code is still unused by NetBSD. I will return to it once the Native Process Plugin is enhanced.

I've filed the mentioned bug:

I've added new ATF test:

  • Verify PT_READ_AUXV called for tracee.

What has been done in LLDB

1. Resolving executable's name with sysctl(7)

In the past the way to retrieve a specified process' executable path name was using Linux-compatibile feature in procfs (/proc). The canonical solution on Linux is to resolve path of /proc/$PID/exe. Christos Zoulas added in DTrace port enhancements a solution similar to FreeBSD to retrieve this property with sysctl(7).

This new approach removes dependency on /proc mounted and Linux compatibility functionality.

Support for this has been submitted to LLDB and merged upstream:

2. Real-Time Signals

The key feature of the POSIX standard with Asynchronous I/O is to support Real-Time Signals. One of their use-cases is in .NET debugging facilities. Support for this set of signals was developed during Google Summer of Code 2016 by Charles Cui and reviewed and committed by Christos Zoulas.

I've extended the LLDB capabilities for NetBSD to recognize these signals in the NetBSDSignals class.

Support for this has been submitted to LLDB and merged upstream:

3. Conflict removal with system-wide six.py

The transition from Python 2.x to 3.x is still ongoing and will take a while. The current deadline support for the 2.x generation has been extended to 2020. One of the ways to keep both generations supported in the same source-code is to use the six.py library (py2 x py3 = 6.py). It abstracts commonly used constructs to support both language families.

The issue for packaging LLDB in NetBSD was to install this tiny library unconditionally to a system-wide location. There were several solutions to this approach:

  • drop Python 2.x support,
  • install six.py into subdirectory,
  • make an installation of six.py conditional.

The first solution would turn discussion into flamewar, the second one happened to be too difficult to be properly implemented as the changes were invasive and Python is used in several places of the code-base (tests, bindings...). The final solution was to introduce a new CMake option LLDB_USE_SYSTEM_SIX - disabled by default to retain the current behavior.

To properly implement LLDB_USE_SYSTEM_SIX, I had to dig into installation scripts combined in CMake and Python files. It wasn't helping that Python scripts were reinventing getopt(3) functionality.. and I had to alter it in order to introduce a new option --useSystemSix.

Support for this has been submitted to LLDB and merged upstream:

4. Do not pass non-POD type variables through variadic function

There was a long standing local patch in pkgsrc, added by Tobias Nygren and detected with Clang.

According to the C++11 standard 5.2.2/7:

Passing a potentially-evaluated argument of class type having a non-trivial copy constructor, a non-trivial move constructor, or a non-trivial destructor, with no corresponding parameter, is conditionally-supported with implementation-defined semantics.

A short example to trigger similar warning was presented by Joerg Sonnenberg: 

#include <string>
#include <cstdarg>

void f(std::string msg, ...) {
  va_list ap;
  va_start(ap, msg);
}

This code compiled against libc++ gives: 

test.cc:6:3: error: cannot pass object of non-POD type 'std::string' (aka 'basic_string<char, char_traits<char>,

allocator<char> >') through variadic function; call will abort at runtime [-Wnon-pod-varargs]

Support for this has been submitted to LLDB and merged upstream:

5. Add NetBSD support in Host::GetCurrentThreadID

Linux has a very specific thread model, where process is mostly equivalent to native thread and POSIX thread - it's completely different on other mainstream general-purpose systems. That said fallback support to translate pthread_t on NetBSD to retrieve the native integer identifier was incorrect. The proper NetBSD function to retrieve light-weigth process identification is to call _lwp_self(2).

Support for this has been submitted to LLDB and merged upstream:

6. Synchronize PlatformNetBSD with Linux

The old PlatformNetBSD code was based on the FreeBSD version. While the FreeBSD current one is still similar to the one from a year ago, it's inappropriate to handle a remote process plugin approach. This forced me to base refreshed code on Linux.

After realizing that PlatformPlugin on POSIX platforms suffers from code duplication, Pavel Labath helped out to eliminate common functions shared by other systems. This resulted in a shorter patch synchronizing PlatformNetBSD with Linux, this step opened room for FreeBSD to catch up.

Support for this has been submitted to LLDB and merged upstream:

7. Transform ProcessLauncherLinux to ProcessLauncherPosixFork

It is UNIX specific that signal handlers are global per application. This introduces issues with wait(2)-like functions called in tracers, as these functions tend to conflict with real-life libraries, notably GUI toolkits (where SIGCHLD events are handled).

The current best approach to this limitation is to spawn a forkee and establish a remote connection over the GDB protocol with a debugger frontend. ProcessLauncherLinux was prepared with this design in mind and I have added support for NetBSD. Once FreeBSD will catch up, they might reuse the same code.

Support for this has been submitted to LLDB and merged upstream:

8. Document that LaunchProcessPosixSpawn is used on NetBSD

Host::GetPosixspawnFlags was built for most POSIX platforms - however only Apple, Linux, FreeBSD and other-GLIBC ones (I assume Debian/kFreeBSD to be GLIBC-like) were documented. I've included NetBSD to this list..

Support for this has been submitted to LLDB and merged upstream:

  • Document that LaunchProcessPosixSpawn is used on NetBSD committed r293770

9. Switch std::call_once to llvm::call_once

There is a long-standing bug in libstdc++ on several platforms that std::call_once is broken for cryptic reasons. This motivated me to follow the approach from LLVM and replace it with homegrown fallback implementation llvm::call_once.

This change wasn't that simple at first sight as the original LLVM version used different semantics that disallowed straight definition of non-static once_flag. Thanks to cooperation with upstream the proper solution was coined and LLDB now works without known regressions on libstdc++ out-of-the-box.

Support for this has been submitted to LLVM, LLDB and merged upstream:

10. Other enhancements

I a had plan to push more code in this milestone besides the mentioned above tasks. Unfortunately not everything was testable at this stage.

Among the rescheduled projects:

  • In the NetBSD platform code conflict removal in GetThreadName / SetThreadName between pthread_t and lwpid_t. It looks like another bite from the Linux thread model. Proper solution to this requires pushing forward the Process Plugin for NetBSD.
  • Host::LaunchProcessPosixSpawn proper setting ::posix_spawnattr_setsigdefault on NetBSD - currently untestable.
  • Fix false positives - premature before adding more functions in NetBSD Native Process Plugin.

On the other hand I've fixed a build issue of one test on NetBSD:

Plan for the next milestone

I've listed the following goals for the next milestone.

  • mark exect(3) obsolete in libc
  • remove libpthread_dbg(3) from the base distribution
  • add new API in ptrace(2) PT_SET_SIGMASK and PT_GET_SIGMASK
  • add new API in ptrace(2) to resume and suspend a specific thread
  • finish switch of the PT_WATCHPOINT API in ptrace(2) to PT_GETDBREGS & PT_SETDBREGS
  • validate i386, amd64 and Xen proper support of new interfaces
  • upstream to LLDB accessors for debug registers on NetBSD/amd64
  • validate PT_SYSCALL and add a functionality to detect and distinguish syscall-entry syscall-exit events
  • validate accessors for general purpose and floating point registers

Post mortem

FreeBSD is catching up after NetBSD changes, e.g. with the following commit:

This move allows to introduce further reduction of code-duplication. There still is a lot of room for improvement. Another benefit for other software distributions, is that they can now appropriately resolve the six.py conflict without local patches.

These examples clearly show that streamlining NetBSD code results in improved support for other systems and creates a cleaner environment for introducing new platforms.

A pure NetBSD-oriented gain is improvement of system interfaces in terms of quality and functionality, especially since DTrace/NetBSD is a quick adopter of new interfaces.. and indirectly a sandbox to sort out bugs in ptrace(2).

The tasks in the next milestone will turn NetBSD's ptrace(2) to be on par with Linux and FreeBSD, this time with marginal differences.

To render it more clearly NetBSD will have more interfaces in read/write mode than FreeBSD has (and be closer to Linux here), on the other hand not so many properites will be available in a thread specific field under the PT_LWPINFO operation that caused suspension of the process.

Another difference is that FreeBSD allows to trace only one type of syscall events: on entry or on exit. At the moment this is not needed in existing software, although it's on the longterm wishlist in the GDB project for Linux.

It turned out that, I was overly optimistic about the feature set in ptrace(2), while the basic ones from the first milestone were enough to implement basic support in LLDB.. it would require me adding major work in heuristics as modern tracers no longer want to perform guessing what might happened in the code and what was the source of signal interruption.

This was the final motivation to streamline the interfaces for monitoring capabilities and now I'm adding remaining interfaces as they are also needed, if not readily in LLDB, there is DTrace and other software that is waiting for them now. Somehow I suspect that I will need them in LLDB sooner than expected.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue to fund projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate

Posted in the wee hours of Monday night, February 14th, 2017 Tags:
Kamil Rytarowski (NetBSD Blog) The first patch-bulk upstreamed to LLDB
The LLVM project is a quickly moving target, this also applies to the LLVM debugger -- LLDB. It's actively used in several first-class operating systems, while - thanks to my spare time dedication - NetBSD joined the LLDB club in 2014, only lately the native support has been substantially improved and the feature set is quickly approaching the support level of Linux and FreeBSD. During this work 12 patches were committed to upstream, 12 patches were submitted to review, 11 new ATF were tests added, 2 NetBSD bugs filed and several dozens of commits were introduced in pkgsrc-wip, reducing the local patch set to mostly Native Process Plugin for NetBSD.

What has been done in NetBSD

1. Triagged issues of ptrace(2) in the DTrace/NetBSD support

Chuck Silvers works on improving DTrace in NetBSD and he has detected an issue when tracer signals are being ignored in libproc. The libproc library is a compatibility layer for DTrace simulating /proc capabilities on the SunOS family of systems.

I've verified that the current behavior of signal routing is incorrect. The NetBSD kernel correctly masks signals emitted by a tracee, not routing them to its tracer. On the other hand the masking rules in the inferior process blacklists signals generated by the kernel, which is incorrect and turns a debugger into a deaf listener. This is the case for libproc as signals were masked and software breakpoints triggering INT3 on i386/amd64 CPUs and SIGTRAP with TRAP_BRKP si_code wasn't passed to the tracer.

This isn't limited to turning a debugger into a deaf listener, but also a regular execution of software breakpoints requires: rewinding the program counter register by a single instruction, removing trap instruction and restoring the original instruction. When an instruction isn't restored and further code execution is pretty randomly affected, it resulted in execution anomalies and breaking of tracee.

A workaround for this is to disable signal masking in tracee.

Another drawback inspired by the DTrace code is to enhance PT_SYSCALL handling by introducing a way to distinguish syscall entry and syscall exit events. I'm planning to add dedicated si_codes for these scenarios. While there, there are users requesting PT_STEP and PT_SYSCALL tracing at the same time in an efficient way without involving heuristcs.

I've filed the mentioned bug:

I've added new ATF tests:

  • Verify that masking single unrelated signal does not stop tracer from catching other signals
  • Verify that masking SIGTRAP in tracee stops tracer from catching this raised signal
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching software breakpoints
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching single step trap
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching exec() breakpoint
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching PTRACE_FORK breakpoint
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching PTRACE_VFORK breakpoint
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching PTRACE_VFORK_DONE breakpoint
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching PTRACE_LWP_CREATE breakpoint
  • Verify that masking SIGTRAP in tracee does not stop tracer from catching PTRACE_LWP_EXIT breakpoint

2. ELF Auxiliary Vectors

The ELF file format permits to transfer additional information for a process with a dedicated container of properties, it's named ELF Auxilary Vector. Every system has its dedicated way to read this information in a debugger from a tracee. The NetBSD approach is to transfer this vector with a ptrace(2) API PIOD_READ_AUXV. Our interface shares the API with OpenBSD. I filed a bug that our interface returns vector size of 8496 bytes, while OpenBSD has constant 64 bytes. It was diagnosed and fixed by Christos Zoluas that we were incorrectly counting bits and bytes and this enlarged the data streamlined. The bug was harmless and had no known side-effects besides large chunk of zeroed data.

There is also a prepared local patch extending NetBSD platform support to read information for this vector, it's primarily required for correct handling of PIE binaries. At the moment there is no interface similar to "info auxv" to the one from GDB. Unfortunately at the current stage, this code is still unused by NetBSD. I will return to it once the Native Process Plugin is enhanced.

I've filed the mentioned bug:

I've added new ATF test:

  • Verify PT_READ_AUXV called for tracee.

What has been done in LLDB

1. Resolving executable's name with sysctl(7)

In the past the way to retrieve a specified process' executable path name was using Linux-compatibile feature in procfs (/proc). The canonical solution on Linux is to resolve path of /proc/$PID/exe. Christos Zoulas added in DTrace port enhancements a solution similar to FreeBSD to retrieve this property with sysctl(7).

This new approach removes dependency on /proc mounted and Linux compatibility functionality.

Support for this has been submitted to LLDB and merged upstream:

2. Real-Time Signals

The key feature of the POSIX standard with Asynchronous I/O is to support Real-Time Signals. One of their use-cases is in .NET debugging facilities. Support for this set of signals was developed during Google Summer of Code 2016 by Charles Cui and reviewed and committed by Christos Zoulas.

I've extended the LLDB capabilities for NetBSD to recognize these signals in the NetBSDSignals class.

Support for this has been submitted to LLDB and merged upstream:

3. Conflict removal with system-wide six.py

The transition from Python 2.x to 3.x is still ongoing and will take a while. The current deadline support for the 2.x generation has been extended to 2020. One of the ways to keep both generations supported in the same source-code is to use the six.py library (py2 x py3 = 6.py). It abstracts commonly used constructs to support both language families.

The issue for packaging LLDB in NetBSD was to install this tiny library unconditionally to a system-wide location. There were several solutions to this approach:

  • drop Python 2.x support,
  • install six.py into subdirectory,
  • make an installation of six.py conditional.

The first solution would turn discussion into flamewar, the second one happened to be too difficult to be properly implemented as the changes were invasive and Python is used in several places of the code-base (tests, bindings...). The final solution was to introduce a new CMake option LLDB_USE_SYSTEM_SIX - disabled by default to retain the current behavior.

To properly implement LLDB_USE_SYSTEM_SIX, I had to dig into installation scripts combined in CMake and Python files. It wasn't helping that Python scripts were reinventing getopt(3) functionality.. and I had to alter it in order to introduce a new option --useSystemSix.

Support for this has been submitted to LLDB and merged upstream:

4. Do not pass non-POD type variables through variadic function

There was a long standing local patch in pkgsrc, added by Tobias Nygren and detected with Clang.

According to the C++11 standard 5.2.2/7:

Passing a potentially-evaluated argument of class type having a non-trivial copy constructor, a non-trivial move constructor, or a non-trivial destructor, with no corresponding parameter, is conditionally-supported with implementation-defined semantics.

A short example to trigger similar warning was presented by Joerg Sonnenberg: 

#include <string>
#include <cstdarg>

void f(std::string msg, ...) {
  va_list ap;
  va_start(ap, msg);
}

This code compiled against libc++ gives: 

test.cc:6:3: error: cannot pass object of non-POD type 'std::string' (aka 'basic_string<char, char_traits<char>,

allocator<char> >') through variadic function; call will abort at runtime [-Wnon-pod-varargs]

Support for this has been submitted to LLDB and merged upstream:

5. Add NetBSD support in Host::GetCurrentThreadID

Linux has a very specific thread model, where process is mostly equivalent to native thread and POSIX thread - it's completely different on other mainstream general-purpose systems. That said fallback support to translate pthread_t on NetBSD to retrieve the native integer identifier was incorrect. The proper NetBSD function to retrieve light-weigth process identification is to call _lwp_self(2).

Support for this has been submitted to LLDB and merged upstream:

6. Synchronize PlatformNetBSD with Linux

The old PlatformNetBSD code was based on the FreeBSD version. While the FreeBSD current one is still similar to the one from a year ago, it's inappropriate to handle a remote process plugin approach. This forced me to base refreshed code on Linux.

After realizing that PlatformPlugin on POSIX platforms suffers from code duplication, Pavel Labath helped out to eliminate common functions shared by other systems. This resulted in a shorter patch synchronizing PlatformNetBSD with Linux, this step opened room for FreeBSD to catch up.

Support for this has been submitted to LLDB and merged upstream:

7. Transform ProcessLauncherLinux to ProcessLauncherPosixFork

It is UNIX specific that signal handlers are global per application. This introduces issues with wait(2)-like functions called in tracers, as these functions tend to conflict with real-life libraries, notably GUI toolkits (where SIGCHLD events are handled).

The current best approach to this limitation is to spawn a forkee and establish a remote connection over the GDB protocol with a debugger frontend. ProcessLauncherLinux was prepared with this design in mind and I have added support for NetBSD. Once FreeBSD will catch up, they might reuse the same code.

Support for this has been submitted to LLDB and merged upstream:

8. Document that LaunchProcessPosixSpawn is used on NetBSD

Host::GetPosixspawnFlags was built for most POSIX platforms - however only Apple, Linux, FreeBSD and other-GLIBC ones (I assume Debian/kFreeBSD to be GLIBC-like) were documented. I've included NetBSD to this list..

Support for this has been submitted to LLDB and merged upstream:

  • Document that LaunchProcessPosixSpawn is used on NetBSD committed r293770

9. Switch std::call_once to llvm::call_once

There is a long-standing bug in libstdc++ on several platforms that std::call_once is broken for cryptic reasons. This motivated me to follow the approach from LLVM and replace it with homegrown fallback implementation llvm::call_once.

This change wasn't that simple at first sight as the original LLVM version used different semantics that disallowed straight definition of non-static once_flag. Thanks to cooperation with upstream the proper solution was coined and LLDB now works without known regressions on libstdc++ out-of-the-box.

Support for this has been submitted to LLVM, LLDB and merged upstream:

10. Other enhancements

I a had plan to push more code in this milestone besides the mentioned above tasks. Unfortunately not everything was testable at this stage.

Among the rescheduled projects:

  • In the NetBSD platform code conflict removal in GetThreadName / SetThreadName between pthread_t and lwpid_t. It looks like another bite from the Linux thread model. Proper solution to this requires pushing forward the Process Plugin for NetBSD.
  • Host::LaunchProcessPosixSpawn proper setting ::posix_spawnattr_setsigdefault on NetBSD - currently untestable.
  • Fix false positives - premature before adding more functions in NetBSD Native Process Plugin.

On the other hand I've fixed a build issue of one test on NetBSD:

Plan for the next milestone

I've listed the following goals for the next milestone.

  • mark exect(3) obsolete in libc
  • remove libpthread_dbg(3) from the base distribution
  • add new API in ptrace(2) PT_SET_SIGMASK and PT_GET_SIGMASK
  • add new API in ptrace(2) to resume and suspend a specific thread
  • finish switch of the PT_WATCHPOINT API in ptrace(2) to PT_GETDBREGS & PT_SETDBREGS
  • validate i386, amd64 and Xen proper support of new interfaces
  • upstream to LLDB accessors for debug registers on NetBSD/amd64
  • validate PT_SYSCALL and add a functionality to detect and distinguish syscall-entry syscall-exit events
  • validate accessors for general purpose and floating point registers

Post mortem

FreeBSD is catching up after NetBSD changes, e.g. with the following commit:

This move allows to introduce further reduction of code-duplication. There still is a lot of room for improvement. Another benefit for other software distributions, is that they can now appropriately resolve the six.py conflict without local patches.

These examples clearly show that streamlining NetBSD code results in improved support for other systems and creates a cleaner environment for introducing new platforms.

A pure NetBSD-oriented gain is improvement of system interfaces in terms of quality and functionality, especially since DTrace/NetBSD is a quick adopter of new interfaces.. and indirectly a sandbox to sort out bugs in ptrace(2).

The tasks in the next milestone will turn NetBSD's ptrace(2) to be on par with Linux and FreeBSD, this time with marginal differences.

To render it more clearly NetBSD will have more interfaces in read/write mode than FreeBSD has (and be closer to Linux here), on the other hand not so many properites will be available in a thread specific field under the PT_LWPINFO operation that caused suspension of the process.

Another difference is that FreeBSD allows to trace only one type of syscall events: on entry or on exit. At the moment this is not needed in existing software, although it's on the longterm wishlist in the GDB project for Linux.

It turned out that, I was overly optimistic about the feature set in ptrace(2), while the basic ones from the first milestone were enough to implement basic support in LLDB.. it would require me adding major work in heuristics as modern tracers no longer want to perform guessing what might happened in the code and what was the source of signal interruption.

This was the final motivation to streamline the interfaces for monitoring capabilities and now I'm adding remaining interfaces as they are also needed, if not readily in LLDB, there is DTrace and other software that is waiting for them now. Somehow I suspect that I will need them in LLDB sooner than expected.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue to fund projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate

Posted in the wee hours of Monday night, February 14th, 2017 Tags:
Christos Zoulas (NetBSD Blog) NetBSD fully reproducible builds

Introduction

I have been working on and off for almost a year trying to get reproducible builds (the same source tree always builds an identical cdrom) on NetBSD. I did not think at the time it would take as long or be so difficult, so I did not keep a log of all the changes I needed to make. I was also not the only one working on this. Other NetBSD developers have been making improvements for the past 6 years.

I would like to acknowledge the NetBSD build system (aka build.sh) which is a fully portable cross-build system. This build system has given us a head-start in the reproducible builds work.

I would also like to acknowledge the work done by the Debian folks who have provided a platform to run, test and analyze reproducible builds. Special mention to the diffoscope tool that gives an excellent overview of what's different between binary files, by finding out what they are (and if they are containers what they contain) and then running the appropriate formatter and diff program to show what's different for each file.

Finally other developers who have started, motivated and did a lot of work getting us here like Joerg Sonnenberger and Thomas Klausner for their work on reproducible builds, and Todd Vierling and Luke Mewburn for their work on build.sh.

Sources of difference

Here's is what we found that we needed to fix, how we chose to fix it and why, and where are we now.

There are many reasons why two separate builds from the same sources can be different. Here's an (incomplete) list:

  1. timestamps

    Many things like to keep track of timestamps, specially archive formats (tar(1), ar(1)), filesystems etc. The way to handle each is different, but the approach is to make them either produce files with a 0 timestamp (where it does not matter like ar), or with a specific timestamp when using 0 does not make sense (it is not useful to the user).

  2. dates/times/authors etc. embedded in source files

    Some programs like to report the date/time they were built, the author, the system they were built on etc. This can be done either by programmatically finding and creating source files containing that information during build time, or by using standard macros such as __DATE__, __TIME__ etc. Usually putting a constant time or eliding the information (such as we do with kernels and bootblocks) solves the problem.

  3. timezone sensitive code

    Certain filesystem formats (iso 9660 etc.) don't store raw timestamps but formatted times; to achieve this they convert from a timestamp to localtime, so they are affected by the timezone.

  4. directory order/build order

    The build order is not constant especially in the presence of parallel builds; neither is directory scan order. If those are used to create output files, the output files will need to be sorted so they become consistent.

  5. non-sanitized data stored into files

    Writing data structures into raw files can lead to problems. Running the same program in different operating systems or using ASLR makes those issues more obvious.

  6. symbolic links/paths

    Having paths embedded into binaries (specially for debugging information) can lead to binary differences. Propagation of the logical path can prove problematic.

  7. general tool inconsistencies

    gcc(1) profiling uses a PROFILE_HOOK macro on RISC targets that utilizes the "current function" number to produce labels. Processing order of functions is not guaranteed. gpt(8) creation involves uuid generation; these are generally random. block allocation on msdos filesystems had a random component. makefs(8) uses timezones with timestamps (iso9660), randomness for block selection (msdos), stores stray pointers in superblock (ffs).

  8. toolchain

    Every program that is used to generate other output needs to have consistent results. In NetBSD this is done with build.sh, which builds a set of tools from known sources before it can use those tools to build the rest of the system). There is a large number of tools. There are also internal issues with the tools that make their output non reproducible, such as nondeterministic symbol creation or capturing parts of the environment in debugging information.

  9. build information / tunables / environment

    There are many environment settings, or build variable settings that can affect the build. This needs to be kept constant across builds so we've changed the list of variables that are reported in Makefile.params: 
    .if ${MKREPRO:Uno} != "yes"
RELEASEVARS+=   BSDOBJDIR BSDSRCDIR BUILDID BUILDINFO BUILDSEED \
                DESTDIR KERNARCHDIR KERNCONFDIR KERNOBJDIR KERNSRCDIR MAKE \
                MAKEFLAGS NBUILDJOBS NETBSDSRCDIR OBJMACHINE OBJMACHINE_ARCH \
                RELEASEDIR RELEASEMACHINEDIR TOOLDIR USR_OBJMACHINE X11SRCDIR
.endif

  10. making sure that the source tree has no local changes

Variables controlling reproducible builds

Reproducible builds are controlled on NetBSD with two variables: MKREPRO (which can be set to yes or no) and MKREPRO_TIMESTAMP which is used to set the timestamp of the builds artifacts. This is usually set to the number of seconds from the epoch. The build.sh -P flag handles reproducible builds automatically: sets the MKREPRO variable to yes, and then finds the latest source file timestamp in the tree and sets MKREPRO_TIMESTAMP to that.

Handling timestamps

The first thing that we needed to understand was how to deal with timestamps. Some of the timestamps are not very useful (for example inside random ar archives) so we choose to 0 them out. Others though become annoying if they are all 0. What does it mean when you mount install media and all the dates on the files are Jan 1, 1970?

We decided that a better timestamp would be the timestamp of the most recently modified file in the source tree. Unfortunately this was not easy to find on NetBSD, because we are still using CVS as the source control system, and CVS does not have a good way to provide that. For that we wrote a tool called cvslatest, that scans the CVS metadata files (CVS/Entries) and finds the latest commit. This works well for freshly checked out trees (since CVS uses the source timestamp when checking out), but not with updated trees (because CVS uses the current time when updating files, so that make(1) thinks they've been modified). To fix that, we've added a new flag to the cvs(1) "update" command -t, that uses the source checkout time.

The build system needs now to evaluate the tree for the latest file running cvslatest(1) and find the latest timestamp in seconds from the Epoch which is set in the MKREPRO_TIMESTAMP variable. This is the same as SOURCE_DATE_EPOCH. Various Makefiles are using this variable and MKRERPO to determine how to produce consistent build artifacts.

For example many commands (tar(1), makefs(8), gpt(8), ...) have been modified to take a --timestamp or -T command line switch to generate output files that use the given timestamp, instead of the current time.

Other software (am-utils, acpica, bootblocks, kernel) used __DATE__ or __TIME__, or captured the user, machine, etc. from the environment and had to be changed to a constant time, user, machine, etc.

roff(7) documents used the td macro to generate the date of formatting in the document have been changed to conditionally use the macro based on register R, for example as in intro.me and then the Makefile was changed to set that register for MKREPRO.

Handling Order

We don't control the build order of things and we also don't control the directory order which can be filesystem dependent. The collation order also is environment specific, and sorting needs to be stable (we have not encountered that problem yet). Two different programs caused us problems here:

  • file(1) with the generation of the compiled magic file using directory order (fixed by changing file(1)).
  • install-info(1), texinfo(5) files that have no specific order. For that we developed another tool called sortinfo(1) that sorts those files as a post-process step.

Fortunately the filesystem builders and tar programs usually work with input directories that appear to have a consistent order so far, so we did not have to fix things there.

Permissions

NetBSD already keeps permissions for most things consistent in different ways:

  • the build system uses install(8) and specifies ownership and mode.
  • the mtree(8) program creates build artifacts using consistent ownership and permissions.

Nevertheless, the various architecture-specific distribution media installers used cp(1)/mkdir(1) and needed to be corrected.

Toolchain

Most of the issues found had to do with capturing the environment in debugging information. The two biggest issues were: DW_AT_Producer and DW_AT_comp_dir: 

DW_AT_producer    : (indirect string, offset: 0x80): GNU C99 5.4.0 \
    -fno-canonical-system-headers -mtune=nocona \
    -march=x86-64 -g -O2 -std=gnu99 -fPIE -fstack-protector \
    -fdebug-prefix-map=$NETBSDSRCDIR=/usr/src \
    -fdebug-prefix-map=$X11SRCDIR=/usr/xsrc \
    -fdebug-regex-map=/usr/src/(.*)/obj.*=/usr/obj/\1 \
    -fdebug-regex-map=/usr/src/(.*)/obj.*/(.*)=/usr/obj/\1/\2 \
    --param ssp-buffer-size=1

Here you see two changes we made for reproducible builds:

  • We chose to allow variable names (and have gcc(1) expand them) for the source of the prefix map because the source tree location can vary. Others have chosen to skip -fdebug-prefix-map from the variables to be listed.
  • We added -fdebug-regex-map so that we could handle the NetBSD specific objdir build functionality. Object directories can have many flavors in NetBSD so it was difficult to use -fdebug-prefix-map to capture that.

DW_AT_comp_dir presented a different challenge. We got non-reproducibility when building on paths where either the source or the object directories contained symbolic links. Although gcc(1) does the right thing handling logical paths (respects $PWD), we found that there were problems both in the NetBSD sh(1) (fixed here) and in the NetBSD make(1) (fixed here). Unfortunately we can't depend on the shell to obey the logical path so we decided to go with: 

    ${MAKE} -C other/dir
instead of: 
    cd other/dir && ${MAKE}

This works because make(1) is a tool (part of the toolchain we provide) whereas sh(1) is not.

Another weird issue popped up on sparc64 where a single file in the whole source tree does not build reproducibly. This file is asn1_krb5_asn1.c which is generated in here. The problem is that when profiling on RISC machines gcc uses the PROFILE_HOOK macro which in turn uses the "function number" to generate labels. This number is assigned to each function in a source file as it is being compiled. Unfortunately this number is not deterministic because of optimization (a bug?), but fortunately turning optimization off fixes the problem.

Status and future work

As of 2017-02-20 we have fully reproducible builds on amd64 and sparc64. We are planning to work on the following areas:

  • Vary more parameters on the system build (filesystem types, build OS's)
  • Verify that cross building is reproducible
  • Verify that unprivileged builds work
  • Test on all the platforms
Posted late Monday afternoon, February 20th, 2017 Tags:
Christos Zoulas (NetBSD Blog) NetBSD fully reproducible builds

Introduction

I have been working on and off for almost a year trying to get reproducible builds (the same source tree always builds an identical cdrom) on NetBSD. I did not think at the time it would take as long or be so difficult, so I did not keep a log of all the changes I needed to make. I was also not the only one working on this. Other NetBSD developers have been making improvements for the past 6 years.

I would like to acknowledge the NetBSD build system (aka build.sh) which is a fully portable cross-build system. This build system has given us a head-start in the reproducible builds work.

I would also like to acknowledge the work done by the Debian folks who have provided a platform to run, test and analyze reproducible builds. Special mention to the diffoscope tool that gives an excellent overview of what's different between binary files, by finding out what they are (and if they are containers what they contain) and then running the appropriate formatter and diff program to show what's different for each file.

Finally other developers who have started, motivated and did a lot of work getting us here like Joerg Sonnenberger and Thomas Klausner for their work on reproducible builds, and Todd Vierling and Luke Mewburn for their work on build.sh.

Sources of difference

Here's is what we found that we needed to fix, how we chose to fix it and why, and where are we now.

There are many reasons why two separate builds from the same sources can be different. Here's an (incomplete) list:

  1. timestamps

    Many things like to keep track of timestamps, specially archive formats (tar(1), ar(1)), filesystems etc. The way to handle each is different, but the approach is to make them either produce files with a 0 timestamp (where it does not matter like ar), or with a specific timestamp when using 0 does not make sense (it is not useful to the user).

  2. dates/times/authors etc. embedded in source files

    Some programs like to report the date/time they were built, the author, the system they were built on etc. This can be done either by programmatically finding and creating source files containing that information during build time, or by using standard macros such as __DATE__, __TIME__ etc. Usually putting a constant time or eliding the information (such as we do with kernels and bootblocks) solves the problem.

  3. timezone sensitive code

    Certain filesystem formats (iso 9660 etc.) don't store raw timestamps but formatted times; to achieve this they convert from a timestamp to localtime, so they are affected by the timezone.

  4. directory order/build order

    The build order is not constant especially in the presence of parallel builds; neither is directory scan order. If those are used to create output files, the output files will need to be sorted so they become consistent.

  5. non-sanitized data stored into files

    Writing data structures into raw files can lead to problems. Running the same program in different operating systems or using ASLR makes those issues more obvious.

  6. symbolic links/paths

    Having paths embedded into binaries (specially for debugging information) can lead to binary differences. Propagation of the logical path can prove problematic.

  7. general tool inconsistencies

    gcc(1) profiling uses a PROFILE_HOOK macro on RISC targets that utilizes the "current function" number to produce labels. Processing order of functions is not guaranteed. gpt(8) creation involves uuid generation; these are generally random. block allocation on msdos filesystems had a random component. makefs(8) uses timezones with timestamps (iso9660), randomness for block selection (msdos), stores stray pointers in superblock (ffs).

  8. toolchain

    Every program that is used to generate other output needs to have consistent results. In NetBSD this is done with build.sh, which builds a set of tools from known sources before it can use those tools to build the rest of the system). There is a large number of tools. There are also internal issues with the tools that make their output non reproducible, such as nondeterministic symbol creation or capturing parts of the environment in debugging information.

  9. build information / tunables / environment

    There are many environment settings, or build variable settings that can affect the build. This needs to be kept constant across builds so we've changed the list of variables that are reported in Makefile.params: 
    .if ${MKREPRO:Uno} != "yes"
RELEASEVARS+=   BSDOBJDIR BSDSRCDIR BUILDID BUILDINFO BUILDSEED \
                DESTDIR KERNARCHDIR KERNCONFDIR KERNOBJDIR KERNSRCDIR MAKE \
                MAKEFLAGS NBUILDJOBS NETBSDSRCDIR OBJMACHINE OBJMACHINE_ARCH \
                RELEASEDIR RELEASEMACHINEDIR TOOLDIR USR_OBJMACHINE X11SRCDIR
.endif

  10. making sure that the source tree has no local changes

Variables controlling reproducible builds

Reproducible builds are controlled on NetBSD with two variables: MKREPRO (which can be set to yes or no) and MKREPRO_TIMESTAMP which is used to set the timestamp of the builds artifacts. This is usually set to the number of seconds from the epoch. The build.sh -P flag handles reproducible builds automatically: sets the MKREPRO variable to yes, and then finds the latest source file timestamp in the tree and sets MKREPRO_TIMESTAMP to that.

Handling timestamps

The first thing that we needed to understand was how to deal with timestamps. Some of the timestamps are not very useful (for example inside random ar archives) so we choose to 0 them out. Others though become annoying if they are all 0. What does it mean when you mount install media and all the dates on the files are Jan 1, 1970?

We decided that a better timestamp would be the timestamp of the most recently modified file in the source tree. Unfortunately this was not easy to find on NetBSD, because we are still using CVS as the source control system, and CVS does not have a good way to provide that. For that we wrote a tool called cvslatest, that scans the CVS metadata files (CVS/Entries) and finds the latest commit. This works well for freshly checked out trees (since CVS uses the source timestamp when checking out), but not with updated trees (because CVS uses the current time when updating files, so that make(1) thinks they've been modified). To fix that, we've added a new flag to the cvs(1) "update" command -t, that uses the source checkout time.

The build system needs now to evaluate the tree for the latest file running cvslatest(1) and find the latest timestamp in seconds from the Epoch which is set in the MKREPRO_TIMESTAMP variable. This is the same as SOURCE_DATE_EPOCH. Various Makefiles are using this variable and MKRERPO to determine how to produce consistent build artifacts.

For example many commands (tar(1), makefs(8), gpt(8), ...) have been modified to take a --timestamp or -T command line switch to generate output files that use the given timestamp, instead of the current time.

Other software (am-utils, acpica, bootblocks, kernel) used __DATE__ or __TIME__, or captured the user, machine, etc. from the environment and had to be changed to a constant time, user, machine, etc.

roff(7) documents used the td macro to generate the date of formatting in the document have been changed to conditionally use the macro based on register R, for example as in intro.me and then the Makefile was changed to set that register for MKREPRO.

Handling Order

We don't control the build order of things and we also don't control the directory order which can be filesystem dependent. The collation order also is environment specific, and sorting needs to be stable (we have not encountered that problem yet). Two different programs caused us problems here:

  • file(1) with the generation of the compiled magic file using directory order (fixed by changing file(1)).
  • install-info(1), texinfo(5) files that have no specific order. For that we developed another tool called sortinfo(1) that sorts those files as a post-process step.

Fortunately the filesystem builders and tar programs usually work with input directories that appear to have a consistent order so far, so we did not have to fix things there.

Permissions

NetBSD already keeps permissions for most things consistent in different ways:

  • the build system uses install(8) and specifies ownership and mode.
  • the mtree(8) program creates build artifacts using consistent ownership and permissions.

Nevertheless, the various architecture-specific distribution media installers used cp(1)/mkdir(1) and needed to be corrected.

Toolchain

Most of the issues found had to do with capturing the environment in debugging information. The two biggest issues were: DW_AT_Producer and DW_AT_comp_dir: 

DW_AT_producer    : (indirect string, offset: 0x80): GNU C99 5.4.0 \
    -fno-canonical-system-headers -mtune=nocona \
    -march=x86-64 -g -O2 -std=gnu99 -fPIE -fstack-protector \
    -fdebug-prefix-map=$NETBSDSRCDIR=/usr/src \
    -fdebug-prefix-map=$X11SRCDIR=/usr/xsrc \
    -fdebug-regex-map=/usr/src/(.*)/obj.*=/usr/obj/\1 \
    -fdebug-regex-map=/usr/src/(.*)/obj.*/(.*)=/usr/obj/\1/\2 \
    --param ssp-buffer-size=1

Here you see two changes we made for reproducible builds:

  • We chose to allow variable names (and have gcc(1) expand them) for the source of the prefix map because the source tree location can vary. Others have chosen to skip -fdebug-prefix-map from the variables to be listed.
  • We added -fdebug-regex-map so that we could handle the NetBSD specific objdir build functionality. Object directories can have many flavors in NetBSD so it was difficult to use -fdebug-prefix-map to capture that.

DW_AT_comp_dir presented a different challenge. We got non-reproducibility when building on paths where either the source or the object directories contained symbolic links. Although gcc(1) does the right thing handling logical paths (respects $PWD), we found that there were problems both in the NetBSD sh(1) (fixed here) and in the NetBSD make(1) (fixed here). Unfortunately we can't depend on the shell to obey the logical path so we decided to go with: 

    ${MAKE} -C other/dir
instead of: 
    cd other/dir && ${MAKE}

This works because make(1) is a tool (part of the toolchain we provide) whereas sh(1) is not.

Another weird issue popped up on sparc64 where a single file in the whole source tree does not build reproducibly. This file is asn1_krb5_asn1.c which is generated in here. The problem is that when profiling on RISC machines gcc uses the PROFILE_HOOK macro which in turn uses the "function number" to generate labels. This number is assigned to each function in a source file as it is being compiled. Unfortunately this number is not deterministic because of optimization (a bug?), but fortunately turning optimization off fixes the problem.

Status and future work

As of 2017-02-20 we have fully reproducible builds on amd64 and sparc64. We are planning to work on the following areas:

  • Vary more parameters on the system build (filesystem types, build OS's)
  • Verify that cross building is reproducible
  • Verify that unprivileged builds work
  • Test on all the platforms
Posted late Monday afternoon, February 20th, 2017 Tags:
snj (NetBSD Blog) NetBSD 7.1_RC2 available

The second release candidate of NetBSD 7.1 is now available for download at:

http://cdn.NetBSD.org/pub/NetBSD/NetBSD-7.1_RC2/

Those of you who prefer to build from source can continue to follow the netbsd-7 branch or use the netbsd-7-1-RC2 tag.

Most changes made since 7.1_RC1 have been security fixes. See src/doc/CHANGES-7.1 for the full list.

Please help us out by testing 7.1_RC2. We love any and all feedback. Report problems through the usual channels (submit a PR or write to the appropriate list). More general feedback is welcome at releng@NetBSD.org.

Posted terribly early Friday morning, February 24th, 2017 Tags:
snj (NetBSD Blog) NetBSD 7.1_RC2 available

The second release candidate of NetBSD 7.1 is now available for download at:

http://cdn.NetBSD.org/pub/NetBSD/NetBSD-7.1_RC2/

Those of you who prefer to build from source can continue to follow the netbsd-7 branch or use the netbsd-7-1-RC2 tag.

Most changes made since 7.1_RC1 have been security fixes. See src/doc/CHANGES-7.1 for the full list.

Please help us out by testing 7.1_RC2. We love any and all feedback. Report problems through the usual channels (submit a PR or write to the appropriate list). More general feedback is welcome at releng@NetBSD.org.

Posted terribly early Friday morning, February 24th, 2017 Tags:
Add a comment
Last edited Tuesday afternoon, May 30th, 2023
Preferences | Logout