This report was written by Naveen Narayanan as part of Google Summer of Code 2019.

This report encompasses the progress of the project during the second coding period.

WINE on amd64

As getting Wine to work with WoW64 support was of foremost importance, my focus was on compat32 dependency packages without which Wine's functionality would be limited and more importantly untestable. Initially, being unaware of what to expect, I just wanted Wine to run, at the earliest. So, with outmost support from mentors, the consensus was to install libs from 32-bit packages to ${PREFIX}/lib/32 and ignore everything else that came with the respective packages.

I had most of the compat32 packages ready after a couple of days. And it was time we gave Wine a whirl. Well, the build was successful. However, I had problems with 32-bit Xorg. The applications which came packaged with Wine worked fine, but, other Microsoft Windows applications like notepad++, Mario etc had a hard time running. Additionally, I noticed that fontconfig went wild and crashed spewing errors symptomatic of Wine (32-bit) not playing nice with the fontconfig lib that came with 32-bit Xorg package. On top of this, I found that build failed on certain machines due to unavailability of headers. This made us reconsider our decision to install 32-bit libs to ${PREFIX}/lib/32 and ignore everything else which included headers and binaries.

Ultimately, we realized that it was time we found a proper solution to the problem of where 32-bit packages should be installed on NetBSD amd64 and then, we eventually settled on ${PREFIX}/emul/netbsd32/. It seemed logical, and I got on with the task of adapting the respective Makefiles of dependency packages to install to ${PREFIX}/emul/netbsd32/. Additionally, I packaged fontconfig (32-bit) in high hopes that Wine would behave appropriately with the former. Wine build was successful. However, I noticed that Wine wasn't linking against the fontconfig libs from my package but against the ones which came with 32-bit Xorg package. Later, I realized, after consulting with mentors, that pkgsrc doesn't search for pkgconfig files (*.pc) in ${PREFIX}/emul/netbsd32/lib by default. pkgsrc sets _PKG_CONFIG_LIBDIR appropriately based on ${LIBABISUFFIX}. As you can see, it doesn't search for .pc files in ${PREFIX}/emul/netbsd32/lib and hence, pkgconfig couldn't get the right flags for fontconfig package. On the other hand, Wine configure script found fontconfig libs provided by 32-bit Xorg package against which it linked wine. A kludgy workaround was to use configure args for the respective libs thereby, obviating configure from finding the wrong libs. Finally, with the help of aforementioned kludgy workarounds, we were able to build Wine and successfully run Mario and Lua (32-bit binaries).

How to run Wine on NetBSD -current

• Build -current distribution.
• Compile -current kernel with USER_LDT enabled and SVS disabled.
• Install kernel and distribution.
• Clone wip repo.
• cd /usr/pkgsrc/wip/wine64; make install
• export LD_LIBRARY_PATH=/usr/pkg/emul/netbsd32/lib; wine mario.exe

Future Plans

Wine requires the kernel option USER_LDT to be able to run 32-bit applications on amd64 - facilitated by WoW64. Presently, this feature isn't enabled by default on NetBSD and hence, the kernel has to be compiled with USER_LDT enabled. This also warrants the kernel option SVS to be disabled currently owing to compatability issues. Work is being done to resolve the conflict which would apparently allow USER_LDT to be enabled by default in the GENERIC kernel.

pkgsrc can be made to search for pkgconf config files in ${PREFIX}/emul/netbsd32/lib by setting _PKG_CONFIG_LIBDIR to ${BUILDLINK_DIR}${LIBABIPREFIX}/lib${LIBABISUFFIX} as per @maya's suggestion. It might take some more thought and time before we finalize it.

Summary

Presently, Wine on amd64 is in test phase. It seems to work fine with caveats like LD_LIBRARY_PATH which has to be set as 32-bit Xorg libs don't have ${PREFIX}/emul/netbsd32/lib in its rpath section. The latter is due to us extracting 32-bit libs from tarballs in lieu of building 32-bit Xorg on amd64. As previously stated, pkgsrc doesn't search for pkgconfig files in ${PREFIX}/emul/netbsd32/lib which might have inadvertent effects that I am unaware of as of now. I shall be working on these issues during the final coding period. I would like to thank @leot, @maya and @christos for saving me from shooting myself in the foot many a time. I, admittedly, have had times when multiple approaches, which all seemed right at that time, perplexed me. I believe those are times when having a mentor counts, and I have been lucky enough to have really good ones. Once again, thanks to Google for this wonderful opportunity.

This report was written by Naveen Narayanan as part of Google Summer of Code 2019.

This report encompasses the progress of the project during the second coding period.

WINE on amd64

As getting Wine to work with WoW64 support was of foremost importance, my focus was on compat32 dependency packages without which Wine's functionality would be limited and more importantly untestable. Initially, being unaware of what to expect, I just wanted Wine to run, at the earliest. So, with outmost support from mentors, the consensus was to install libs from 32-bit packages to ${PREFIX}/lib/32 and ignore everything else that came with the respective packages.

I had most of the compat32 packages ready after a couple of days. And it was time we gave Wine a whirl. Well, the build was successful. However, I had problems with 32-bit Xorg. The applications which came packaged with Wine worked fine, but, other Microsoft Windows applications like notepad++, Mario etc had a hard time running. Additionally, I noticed that fontconfig went wild and crashed spewing errors symptomatic of Wine (32-bit) not playing nice with the fontconfig lib that came with 32-bit Xorg package. On top of this, I found that build failed on certain machines due to unavailability of headers. This made us reconsider our decision to install 32-bit libs to ${PREFIX}/lib/32 and ignore everything else which included headers and binaries.

Ultimately, we realized that it was time we found a proper solution to the problem of where 32-bit packages should be installed on NetBSD amd64 and then, we eventually settled on ${PREFIX}/emul/netbsd32/. It seemed logical, and I got on with the task of adapting the respective Makefiles of dependency packages to install to ${PREFIX}/emul/netbsd32/. Additionally, I packaged fontconfig (32-bit) in high hopes that Wine would behave appropriately with the former. Wine build was successful. However, I noticed that Wine wasn't linking against the fontconfig libs from my package but against the ones which came with 32-bit Xorg package. Later, I realized, after consulting with mentors, that pkgsrc doesn't search for pkgconfig files (*.pc) in ${PREFIX}/emul/netbsd32/lib by default. pkgsrc sets _PKG_CONFIG_LIBDIR appropriately based on ${LIBABISUFFIX}. As you can see, it doesn't search for .pc files in ${PREFIX}/emul/netbsd32/lib and hence, pkgconfig couldn't get the right flags for fontconfig package. On the other hand, Wine configure script found fontconfig libs provided by 32-bit Xorg package against which it linked wine. A kludgy workaround was to use configure args for the respective libs thereby, obviating configure from finding the wrong libs. Finally, with the help of aforementioned kludgy workarounds, we were able to build Wine and successfully run Mario and Lua (32-bit binaries).

How to run Wine on NetBSD -current

• Build -current distribution.
• Compile -current kernel with USER_LDT enabled and SVS disabled.
• Install kernel and distribution.
• Clone wip repo.
• cd /usr/pkgsrc/wip/wine64; make install
• export LD_LIBRARY_PATH=/usr/pkg/emul/netbsd32/lib; wine mario.exe

Future Plans

Wine requires the kernel option USER_LDT to be able to run 32-bit applications on amd64 - facilitated by WoW64. Presently, this feature isn't enabled by default on NetBSD and hence, the kernel has to be compiled with USER_LDT enabled. This also warrants the kernel option SVS to be disabled currently owing to compatability issues. Work is being done to resolve the conflict which would apparently allow USER_LDT to be enabled by default in the GENERIC kernel.

pkgsrc can be made to search for pkgconf config files in ${PREFIX}/emul/netbsd32/lib by setting _PKG_CONFIG_LIBDIR to ${BUILDLINK_DIR}${LIBABIPREFIX}/lib${LIBABISUFFIX} as per @maya's suggestion. It might take some more thought and time before we finalize it.

Summary

Presently, Wine on amd64 is in test phase. It seems to work fine with caveats like LD_LIBRARY_PATH which has to be set as 32-bit Xorg libs don't have ${PREFIX}/emul/netbsd32/lib in its rpath section. The latter is due to us extracting 32-bit libs from tarballs in lieu of building 32-bit Xorg on amd64. As previously stated, pkgsrc doesn't search for pkgconfig files in ${PREFIX}/emul/netbsd32/lib which might have inadvertent effects that I am unaware of as of now. I shall be working on these issues during the final coding period. I would like to thank @leot, @maya and @christos for saving me from shooting myself in the foot many a time. I, admittedly, have had times when multiple approaches, which all seemed right at that time, perplexed me. I believe those are times when having a mentor counts, and I have been lucky enough to have really good ones. Once again, thanks to Google for this wonderful opportunity.

I have been working on adapting TriforceAFL for NetBSD kernel syscall fuzzing. This blog post summarizes the work done until the second evaluation.

For work done during the first coding period, check out this post.

Input Generation

For a feedback driven mutation based fuzzer that is TriforceAFL, fuzzing can be greatly improved by providing it with proper input test cases. The fuzzer can then alter parts of the valid input leading to more coverage and hopefully more bugs. The TriforceNetBSDSyscallFuzzer itself was a working fuzzer at the end of the first evaluation, but it was missing some proper input generation for most of the syscalls.
A greater part of the time during this coding period was spent adding and testing basic templates for a majority of NetBSD syscalls, scripts have also been added for cases where more complex input generation was required. This should now allow the fuzzer to find bugs it previously could not have.

Templates for 160 of the 483 syscalls in NetBSD have been added, below is the complete list:

A separate script (targ/gen2.py) generating tricker input cases was added for the following:

Reproducibility

The fuzzer uses the simplest way to reproduce a crash which is by storing the exact input for the test case that resulted in a crash. This input can then be passed to the driver program, which will be able to parse the input and execute the syscalls that were executed in the order that they were executed.
A better prototype reproducer generator has now been added to the fuzzer that provides us with human readable and executable C code. This C code can be compiled and executed to reproduce the crash.
To generate the reproducers simply run the genRepro script from the targ directory. If everything goes right, the C reproducers will now be available in targ/reproducers, in separate files as follows:

// id:000009,sig:00,src:005858+002155,op:splice,rep:8
#include <sys/syscall.h>
#include <unistd.h>

int main() {
    __syscall( SYS_mincore, 0x/* removed from the report due to security concerns */, 0x/* ... */, 0x/* ... */);
    return 0;
}

// id:000010,sig:00,src:005859+004032,op:splice,rep:2
#include <sys/syscall.h>
#include <unistd.h>

int main() {
    __syscall( SYS_mprotect, 0x/* ... */, 0x/* ... */, 0x/* ... */);
    __syscall( SYS_mincore, 0x/* ... */, 0x/* ... */, 0x/* ... */);
    return 0;
}

// id:000011,sig:00,src:005859+004032,op:splice,rep:4
#include <sys/syscall.h>
#include <unistd.h>

int main() {
    __syscall( SYS_mprotect, 0x/* ... */, 0x/* ... */, 0x/* ... */);
    __syscall( SYS_mincore, 0x/* ... */, 0x/* ... */, 0x/* ... */);
    return 0;
}

Fuzzing

The fuzzer was run for ~4 days. We were getting ~50 execs/sec on a single core. Please note that this is using qemu with softemu (TCG) as hardware acceleration cannot be used here. During this period the fuzzer detected 23 crashes that it marked as unique. Not all of these were truly unique, there is the scope of adding a secondary filter here to detect truly unique crashes. Most of them were duplicates of the crashes highlighted below:

compat_43_osendmsg - tcp_output: no template

call 114 - compat_43_osendmsg
arg 0: argStdFile 4 - type 12
arg 1: argVec64 77d0549cc000 - size 4
arg 2: argNum 8003

read 83 bytes, parse result 0 nrecs 1
syscall 114 (4, 77d0549cc000, 8003)

[ 191.8169124] panic: tcp_output: no template
[ 191.8169124] cpu0: Begin traceback...
[ 191.8269174] vpanic() at netbsd:vpanic+0x160
[ 191.8269174] snprintf() at netbsd:snprintf
[ 191.8269174] tcp_output() at netbsd:tcp_output+0x2869
[ 191.8385864] tcp_sendoob_wrapper() at netbsd:tcp_sendoob_wrapper+0xfe
[ 191.8469824] sosend() at netbsd:sosend+0x6e3
[ 191.8469824] do_sys_sendmsg_so() at netbsd:do_sys_sendmsg_so+0x231
[ 191.8469824] do_sys_sendmsg() at netbsd:do_sys_sendmsg+0xac
[ 191.8569944] compat_43_sys_sendmsg() at netbsd:compat_43_sys_sendmsg+0xea
[ 191.8569944] sys___syscall() at netbsd:sys___syscall+0x74
[ 191.8655484] syscall() at netbsd:syscall+0x181
[ 191.8655484] --- syscall (number 198) ---
[ 191.8655484] 40261a:
[ 191.8655484] cpu0: End traceback...
[ 191.8655484] fatal breakpoint trap in supervisor mode
[ 191.8655484] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x202 cr2
 0x7f7795402000 ilevel 0x4 rsp 0xffffc58032d589f0
[ 191.8655484] curlwp 0xfffff7e8b1c63220 pid 700.1 lowest kstack 0xffffc58032d55
2c0
Stopped in pid 700.1 (driver) at        netbsd:breakpoint+0x5:  leave

mincore - uvm_fault_unwire_locked: address not in map

call 78 - mincore
arg 0: argNum d000000
arg 1: argNum 7600000000000000
arg 2: argNum 1b0000000000
read 65 bytes, parse result 0 nrecs 1
syscall 78 (d000000, 7600000000000000, 1b0000000000)

[ 141.0578675] panic: uvm_fault_unwire_locked: address not in map
[ 141.0578675] cpu0: Begin traceback...
[ 141.0691345] vpanic() at netbsd:vpanic+0x160
[ 141.0691345] snprintf() at netbsd:snprintf
[ 141.0774205] uvm_fault_unwire() at netbsd:uvm_fault_unwire
[ 141.0774205] uvm_fault_unwire() at netbsd:uvm_fault_unwire+0x29
[ 141.0774205] sys_mincore() at netbsd:sys_mincore+0x23c
[ 141.0884435] sys___syscall() at netbsd:sys___syscall+0x74
[ 141.0884435] syscall() at netbsd:syscall+0x181
[ 141.0884435] --- syscall (number 198) ---
[ 141.0996065] 40261a:
[ 141.0996065] cpu0: End traceback...
[ 141.0996065] fatal breakpoint trap in supervisor mode
[ 141.0996065] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x202 cr2
 0x761548094000 ilevel 0 rsp 0xffff870032e48d90
[ 141.0996065] curlwp 0xffff829094e51b00 pid 646.1 lowest kstack 0xffff870032e45
2c0
Stopped in pid 646.1 (driver) at        netbsd:breakpoint+0x5:  leave

extattrctl - KASSERT fail

call 360 - extattrctl
arg 0: argBuf 7b762b514044 from 2 bytes
arg 1: argNum ff8001
arg 2: argFilename 7b762b515020 - 2 bytes from /tmp/file0
arg 3: argNum 0
arg 4: argNum 0
arg 5: argNum 2100000000
read 59 bytes, parse result 0 nrecs 1
syscall 360 (7b762b514044, ff8001, 7b762b515020, 0, 0, 2100000000)

[ 386.4528838] panic: kernel diagnostic assertion "fli->fli_trans_cnt == 0" fail
ed: file "src/sys/kern/vfs_trans.c", line 201
[ 386.4528838] cpu0: Begin traceback...
[ 386.4528838] vpanic() at netbsd:vpanic+0x160
[ 386.4648968] stge_eeprom_wait.isra.4() at netbsd:stge_eeprom_wait.isra.4
[ 386.4724138] fstrans_lwp_dtor() at netbsd:fstrans_lwp_dtor+0xbd
[ 386.4724138] exit1() at netbsd:exit1+0x1fa
[ 386.4724138] sys_exit() at netbsd:sys_exit+0x3d
[ 386.4832968] syscall() at netbsd:syscall+0x181
[ 386.4832968] --- syscall (number 1) ---
[ 386.4832968] 421b6a:
[ 386.4832968] cpu0: End traceback...
[ 386.4832968] fatal breakpoint trap in supervisor mode
[ 386.4944688] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x202 cr2
 0xffffc100324bd000 ilevel 0 rsp 0xffffc10032ce9dc0
[ 386.4944688] curlwp 0xfffff6278e2fc240 pid 105.1 lowest kstack 0xffffc10032ce6
2c0
Stopped in pid 105.1 (driver) at        netbsd:breakpoint+0x5:  leave

pkgsrc Package

Lastly, the TriforceNetBSDSyscallFuzzer has now been made available in the form of a pkgsrc package in pkgsrc/wip as triforcenetbsdsyscallfuzzer. The package will require wip/triforceafl which was ported earlier.
All other changes mentioned can be found in the github repo.

script(1) recording

A typescript recording of a functional TriforceAFL fuzzer setup and execution is available here. Download it and replay it with script -p.

Future Work

Work that remains to be done include:

• Restructuring of files
  The file structure needs to be modified to suit the specific case of the Host and Target being the same OS. Right now, files are separated into Host and Target directories, this is not required.
• Testing with Sanitizers enabled
  Until now the fuzzing done was without using KASAN or kUBSAN. Testing with them enabled and fuzzing with them will be of major focus in the third coding period.
• Improving the 'reproducer generator'
  There is some scope of improvement for the prototype that was added. Incremental updates to it are to be expected.
• Analysis of crash reports and fixing bugs
• Documentation

Summary

So far, the TriforceNetBSDSyscallFuzzer has been made available in the form of a pkgsrc package with the ability to fuzz most of NetBSD syscalls. In the final coding period of GSoC. I plan to analyse the crashes that were found until now. Integrate sanitizers, try and find more bugs and finally wrap up neatly with detailed documentation.

Last but not least, I would like to thank my mentor, Kamil Rytarowski for helping me through the process and guiding me. It has been a wonderful learning experience so far!

I have been working on adapting TriforceAFL for NetBSD kernel syscall fuzzing. This blog post summarizes the work done until the second evaluation.

For work done during the first coding period, check out this post.

Input Generation

For a feedback driven mutation based fuzzer that is TriforceAFL, fuzzing can be greatly improved by providing it with proper input test cases. The fuzzer can then alter parts of the valid input leading to more coverage and hopefully more bugs. The TriforceNetBSDSyscallFuzzer itself was a working fuzzer at the end of the first evaluation, but it was missing some proper input generation for most of the syscalls.
A greater part of the time during this coding period was spent adding and testing basic templates for a majority of NetBSD syscalls, scripts have also been added for cases where more complex input generation was required. This should now allow the fuzzer to find bugs it previously could not have.

Templates for 160 of the 483 syscalls in NetBSD have been added, below is the complete list:

A separate script (targ/gen2.py) generating tricker input cases was added for the following:

Reproducibility

The fuzzer uses the simplest way to reproduce a crash which is by storing the exact input for the test case that resulted in a crash. This input can then be passed to the driver program, which will be able to parse the input and execute the syscalls that were executed in the order that they were executed.
A better prototype reproducer generator has now been added to the fuzzer that provides us with human readable and executable C code. This C code can be compiled and executed to reproduce the crash.
To generate the reproducers simply run the genRepro script from the targ directory. If everything goes right, the C reproducers will now be available in targ/reproducers, in separate files as follows:

// id:000009,sig:00,src:005858+002155,op:splice,rep:8
#include <sys/syscall.h>
#include <unistd.h>

int main() {
    __syscall( SYS_mincore, 0x/* removed from the report due to security concerns */, 0x/* ... */, 0x/* ... */);
    return 0;
}

// id:000010,sig:00,src:005859+004032,op:splice,rep:2
#include <sys/syscall.h>
#include <unistd.h>

int main() {
    __syscall( SYS_mprotect, 0x/* ... */, 0x/* ... */, 0x/* ... */);
    __syscall( SYS_mincore, 0x/* ... */, 0x/* ... */, 0x/* ... */);
    return 0;
}

// id:000011,sig:00,src:005859+004032,op:splice,rep:4
#include <sys/syscall.h>
#include <unistd.h>

int main() {
    __syscall( SYS_mprotect, 0x/* ... */, 0x/* ... */, 0x/* ... */);
    __syscall( SYS_mincore, 0x/* ... */, 0x/* ... */, 0x/* ... */);
    return 0;
}

Fuzzing

The fuzzer was run for ~4 days. We were getting ~50 execs/sec on a single core. Please note that this is using qemu with softemu (TCG) as hardware acceleration cannot be used here. During this period the fuzzer detected 23 crashes that it marked as unique. Not all of these were truly unique, there is the scope of adding a secondary filter here to detect truly unique crashes. Most of them were duplicates of the crashes highlighted below:

compat_43_osendmsg - tcp_output: no template

call 114 - compat_43_osendmsg
arg 0: argStdFile 4 - type 12
arg 1: argVec64 77d0549cc000 - size 4
arg 2: argNum 8003

read 83 bytes, parse result 0 nrecs 1
syscall 114 (4, 77d0549cc000, 8003)

[ 191.8169124] panic: tcp_output: no template
[ 191.8169124] cpu0: Begin traceback...
[ 191.8269174] vpanic() at netbsd:vpanic+0x160
[ 191.8269174] snprintf() at netbsd:snprintf
[ 191.8269174] tcp_output() at netbsd:tcp_output+0x2869
[ 191.8385864] tcp_sendoob_wrapper() at netbsd:tcp_sendoob_wrapper+0xfe
[ 191.8469824] sosend() at netbsd:sosend+0x6e3
[ 191.8469824] do_sys_sendmsg_so() at netbsd:do_sys_sendmsg_so+0x231
[ 191.8469824] do_sys_sendmsg() at netbsd:do_sys_sendmsg+0xac
[ 191.8569944] compat_43_sys_sendmsg() at netbsd:compat_43_sys_sendmsg+0xea
[ 191.8569944] sys___syscall() at netbsd:sys___syscall+0x74
[ 191.8655484] syscall() at netbsd:syscall+0x181
[ 191.8655484] --- syscall (number 198) ---
[ 191.8655484] 40261a:
[ 191.8655484] cpu0: End traceback...
[ 191.8655484] fatal breakpoint trap in supervisor mode
[ 191.8655484] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x202 cr2
 0x7f7795402000 ilevel 0x4 rsp 0xffffc58032d589f0
[ 191.8655484] curlwp 0xfffff7e8b1c63220 pid 700.1 lowest kstack 0xffffc58032d55
2c0
Stopped in pid 700.1 (driver) at        netbsd:breakpoint+0x5:  leave

mincore - uvm_fault_unwire_locked: address not in map

call 78 - mincore
arg 0: argNum d000000
arg 1: argNum 7600000000000000
arg 2: argNum 1b0000000000
read 65 bytes, parse result 0 nrecs 1
syscall 78 (d000000, 7600000000000000, 1b0000000000)

[ 141.0578675] panic: uvm_fault_unwire_locked: address not in map
[ 141.0578675] cpu0: Begin traceback...
[ 141.0691345] vpanic() at netbsd:vpanic+0x160
[ 141.0691345] snprintf() at netbsd:snprintf
[ 141.0774205] uvm_fault_unwire() at netbsd:uvm_fault_unwire
[ 141.0774205] uvm_fault_unwire() at netbsd:uvm_fault_unwire+0x29
[ 141.0774205] sys_mincore() at netbsd:sys_mincore+0x23c
[ 141.0884435] sys___syscall() at netbsd:sys___syscall+0x74
[ 141.0884435] syscall() at netbsd:syscall+0x181
[ 141.0884435] --- syscall (number 198) ---
[ 141.0996065] 40261a:
[ 141.0996065] cpu0: End traceback...
[ 141.0996065] fatal breakpoint trap in supervisor mode
[ 141.0996065] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x202 cr2
 0x761548094000 ilevel 0 rsp 0xffff870032e48d90
[ 141.0996065] curlwp 0xffff829094e51b00 pid 646.1 lowest kstack 0xffff870032e45
2c0
Stopped in pid 646.1 (driver) at        netbsd:breakpoint+0x5:  leave

extattrctl - KASSERT fail

call 360 - extattrctl
arg 0: argBuf 7b762b514044 from 2 bytes
arg 1: argNum ff8001
arg 2: argFilename 7b762b515020 - 2 bytes from /tmp/file0
arg 3: argNum 0
arg 4: argNum 0
arg 5: argNum 2100000000
read 59 bytes, parse result 0 nrecs 1
syscall 360 (7b762b514044, ff8001, 7b762b515020, 0, 0, 2100000000)

[ 386.4528838] panic: kernel diagnostic assertion "fli->fli_trans_cnt == 0" fail
ed: file "src/sys/kern/vfs_trans.c", line 201
[ 386.4528838] cpu0: Begin traceback...
[ 386.4528838] vpanic() at netbsd:vpanic+0x160
[ 386.4648968] stge_eeprom_wait.isra.4() at netbsd:stge_eeprom_wait.isra.4
[ 386.4724138] fstrans_lwp_dtor() at netbsd:fstrans_lwp_dtor+0xbd
[ 386.4724138] exit1() at netbsd:exit1+0x1fa
[ 386.4724138] sys_exit() at netbsd:sys_exit+0x3d
[ 386.4832968] syscall() at netbsd:syscall+0x181
[ 386.4832968] --- syscall (number 1) ---
[ 386.4832968] 421b6a:
[ 386.4832968] cpu0: End traceback...
[ 386.4832968] fatal breakpoint trap in supervisor mode
[ 386.4944688] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x202 cr2
 0xffffc100324bd000 ilevel 0 rsp 0xffffc10032ce9dc0
[ 386.4944688] curlwp 0xfffff6278e2fc240 pid 105.1 lowest kstack 0xffffc10032ce6
2c0
Stopped in pid 105.1 (driver) at        netbsd:breakpoint+0x5:  leave

pkgsrc Package

Lastly, the TriforceNetBSDSyscallFuzzer has now been made available in the form of a pkgsrc package in pkgsrc/wip as triforcenetbsdsyscallfuzzer. The package will require wip/triforceafl which was ported earlier.
All other changes mentioned can be found in the github repo.

script(1) recording

A typescript recording of a functional TriforceAFL fuzzer setup and execution is available here. Download it and replay it with script -p.

Future Work

Work that remains to be done include:

• Restructuring of files
  The file structure needs to be modified to suit the specific case of the Host and Target being the same OS. Right now, files are separated into Host and Target directories, this is not required.
• Testing with Sanitizers enabled
  Until now the fuzzing done was without using KASAN or kUBSAN. Testing with them enabled and fuzzing with them will be of major focus in the third coding period.
• Improving the 'reproducer generator'
  There is some scope of improvement for the prototype that was added. Incremental updates to it are to be expected.
• Analysis of crash reports and fixing bugs
• Documentation

Summary

So far, the TriforceNetBSDSyscallFuzzer has been made available in the form of a pkgsrc package with the ability to fuzz most of NetBSD syscalls. In the final coding period of GSoC. I plan to analyse the crashes that were found until now. Integrate sanitizers, try and find more bugs and finally wrap up neatly with detailed documentation.

Last but not least, I would like to thank my mentor, Kamil Rytarowski for helping me through the process and guiding me. It has been a wonderful learning experience so far!

Prepared by Siddharth Muralee(@R3x) as a part of Google Summer of Code’19

As a part of Google Summer of Code’19, I am working on improving the support for Syzkaller kernel fuzzer. Syzkaller is an unsupervised coverage-guided kernel fuzzer, that supports a variety of operating systems including NetBSD. This report details the work done during the second coding period.

You can also take a look at the first report to learn more about the initial support that we added.

Network Packet Injection

As a part of improving the fuzzing support for the NetBSD kernel. We decided to add support for fuzzing the Network stack. This feature already exists for operating systems such as Linux and OpenBSD.

Motivation

The aim is to fuzz the Network stack by sending packets with malformed/random data and see if it causes any kernel anomalies. We aim to send packets in such a way that the code paths in the kernel which would usually get triggered normally (during ordinary usage of the networking system) would also get triggered here. This is achieved using the TAP/TUN interface.

TAP/TUN Interface

TAP/TUN interface is a software-only interface - which means there are no hardware links involved. This makes it an ideal option to be used for interacting with the kernel networking code.

Userspace programs can create TAP/TUN interfaces and then write properly formatted data into it which will then be sent to the kernel. Also reading from the TAP/TUN interface would give us the data that the interface is sending out.

Basic Design

We create a virtual interface using TAP/TUN and send packets through it. We add two syscalls - syz_emit_ethernet and syz_extract_tcp_res. The former does the job of sending a packet into the kernel and the latter receives the response.

We need the response from the kernel to be read because we need the TCP acknowledgement number to be used for the next packet that we send. So syz_extract_tcp_res also extracts the acknowledgement number from the reply the kernel sent. This article explains the concept of TCP acknowledgement and sequence numbers very well.

Parallelizing Network fuzzing

In the syzkaller config you can define the number of processes that syzkaller can run in a single VM instance. And since we can have multiple instances of the fuzzer(executor) running at the same time we need to make sure that there is no collision between the interfaces. For solving this we create a seperate interface per fuzzer and assign it with a different IP address (both IPv4 and IPv6) to create an isolated network.

Filesystem Image fuzzing

A very less explored area in fuzzing is Filesystem image fuzzing. Syzkaller supports filesystem fuzzing only for the Linux kernel. We are currently working on porting the existing support that Linux has and then improve it.

Motivation

The aim is to fuzz filesystem specific code by mounting custom images and then perform operations on these images. This would lead to execution of the kernel components of filesystem code and allow us to find out potential bugs.

Existing Design

As I mentioned in the previous report - syzkaller generats inputs based on a pseudo formal grammar. This allows us to also write grammar to generate filesystem images on the fly. This is what the current implementation does - generate random images, write the segments into memory and then mount the resulting image.

Miscellaneous work

I have also been fine tuning the syzkaller fuzzer whenever necessary. This involves adding new features and fixing issues.

Coverage Display

Syzkaller has a utility which uses the coverage from KCoV and marks the corresponding lines on the source code and displays the same. This feature wasn’t working for some of the operating systems.

The issue was that syzkaller was trying to strip off the common prefix on all the file paths retrieved from Kcov and then appending the directory where the source code is present to access the files.

Stripping the common prefix made sense for Linux since its files were distributed across multiple folders in the src/ directory. This created issues for NetBSD since almost all of the kernel files are present in src/sys/ which lead to `sys` also being removed from the resultant path leading to an invalid file name.

I worked on revamping the syz-manager such that it removes prefix computation altogether and take the prefix we need to strip as a part of the config for syz-manager. Then we add the filepath of the kernel sources to get the path to the file.

The coverage for NetBSD can be viewed on the NetBSD dashboard.

TODO

The Filesystem fuzzing code isn’t upstream as of now and will be done shortly.

I have added a basic support for fuzzing both the Filesystem and the Network stack. Now there are a lot of improvements to be done for the same. This mainly involves adding more detailed and improved descriptions.

Relevant Links

Summary

So far, we have found around 70 unique crashes with syzkaller. During the final coding period - I would be working on improving the support for the Filesystem fuzzing.

Last but not least, I want to thank my mentors, @kamil and @cryo for their useful suggestions and guidance. I would also like to thank Dmitry Vyukov, Google for helping with any issues faced with regard to Syzkaller. Finally, thanks to Google to give me a good chance to work with NetBSD community.

Prepared by Siddharth Muralee(@R3x) as a part of Google Summer of Code’19

As a part of Google Summer of Code’19, I am working on improving the support for Syzkaller kernel fuzzer. Syzkaller is an unsupervised coverage-guided kernel fuzzer, that supports a variety of operating systems including NetBSD. This report details the work done during the second coding period.

You can also take a look at the first report to learn more about the initial support that we added.

Network Packet Injection

As a part of improving the fuzzing support for the NetBSD kernel. We decided to add support for fuzzing the Network stack. This feature already exists for operating systems such as Linux and OpenBSD.

Motivation

The aim is to fuzz the Network stack by sending packets with malformed/random data and see if it causes any kernel anomalies. We aim to send packets in such a way that the code paths in the kernel which would usually get triggered normally (during ordinary usage of the networking system) would also get triggered here. This is achieved using the TAP/TUN interface.

TAP/TUN Interface

TAP/TUN interface is a software-only interface - which means there are no hardware links involved. This makes it an ideal option to be used for interacting with the kernel networking code.

Userspace programs can create TAP/TUN interfaces and then write properly formatted data into it which will then be sent to the kernel. Also reading from the TAP/TUN interface would give us the data that the interface is sending out.

Basic Design

We create a virtual interface using TAP/TUN and send packets through it. We add two syscalls - syz_emit_ethernet and syz_extract_tcp_res. The former does the job of sending a packet into the kernel and the latter receives the response.

We need the response from the kernel to be read because we need the TCP acknowledgement number to be used for the next packet that we send. So syz_extract_tcp_res also extracts the acknowledgement number from the reply the kernel sent. This article explains the concept of TCP acknowledgement and sequence numbers very well.

Parallelizing Network fuzzing

In the syzkaller config you can define the number of processes that syzkaller can run in a single VM instance. And since we can have multiple instances of the fuzzer(executor) running at the same time we need to make sure that there is no collision between the interfaces. For solving this we create a seperate interface per fuzzer and assign it with a different IP address (both IPv4 and IPv6) to create an isolated network.

Filesystem Image fuzzing

A very less explored area in fuzzing is Filesystem image fuzzing. Syzkaller supports filesystem fuzzing only for the Linux kernel. We are currently working on porting the existing support that Linux has and then improve it.

Motivation

The aim is to fuzz filesystem specific code by mounting custom images and then perform operations on these images. This would lead to execution of the kernel components of filesystem code and allow us to find out potential bugs.

Existing Design

As I mentioned in the previous report - syzkaller generats inputs based on a pseudo formal grammar. This allows us to also write grammar to generate filesystem images on the fly. This is what the current implementation does - generate random images, write the segments into memory and then mount the resulting image.

Miscellaneous work

I have also been fine tuning the syzkaller fuzzer whenever necessary. This involves adding new features and fixing issues.

Coverage Display

Syzkaller has a utility which uses the coverage from KCoV and marks the corresponding lines on the source code and displays the same. This feature wasn’t working for some of the operating systems.

The issue was that syzkaller was trying to strip off the common prefix on all the file paths retrieved from Kcov and then appending the directory where the source code is present to access the files.

Stripping the common prefix made sense for Linux since its files were distributed across multiple folders in the src/ directory. This created issues for NetBSD since almost all of the kernel files are present in src/sys/ which lead to `sys` also being removed from the resultant path leading to an invalid file name.

I worked on revamping the syz-manager such that it removes prefix computation altogether and take the prefix we need to strip as a part of the config for syz-manager. Then we add the filepath of the kernel sources to get the path to the file.

The coverage for NetBSD can be viewed on the NetBSD dashboard.

TODO

The Filesystem fuzzing code isn’t upstream as of now and will be done shortly.

I have added a basic support for fuzzing both the Filesystem and the Network stack. Now there are a lot of improvements to be done for the same. This mainly involves adding more detailed and improved descriptions.

Relevant Links

Summary

So far, we have found around 70 unique crashes with syzkaller. During the final coding period - I would be working on improving the support for the Filesystem fuzzing.

Last but not least, I want to thank my mentors, @kamil and @cryo for their useful suggestions and guidance. I would also like to thank Dmitry Vyukov, Google for helping with any issues faced with regard to Syzkaller. Finally, thanks to Google to give me a good chance to work with NetBSD community.

Upstream describes LLDB as a next generation, high-performance debugger. It is built on top of LLVM/Clang toolchain, and features great integration with it. At the moment, it primarily supports debugging C, C++ and ObjC code, and there is interest in extending it to more languages.

In February, I have started working on LLDB, as contracted by the NetBSD Foundation. So far I've been working on reenabling continuous integration, squashing bugs, improving NetBSD core file support, extending NetBSD's ptrace interface to cover more register types and fix compat32 issues, and lately fixing watchpoint support. You can read more about that in my June 2019 report.

My July's work has been focused on improving support for NetBSD threads in LLDB. This involved a lot of debugging and fighting hanging tests, and I have decided to delay committing the results until I manage to provide fixes for all the immediate issues.

Buildbot updates

During July, upstream has made two breaking changes to the build system:

1. Automatic switching to libc++abi when present in ebuild tree was initially removed in D63883. I needed to force it explicitly because of this. However, upstream has eventually reverted the change.

2. LLDB has enabled Python 3 support, and started requiring SWIG 2+ in the process (D64782). We had to upgrade SWIG on the build host, and eventually switched to Python 3 as well.

As a result of earlier watchpoint fixes, a number of new tests started running. Due to lacking multithreading support, I had to XFAIL a number of LLDB tests in r365338.

A few days later, upstream has fixed the issue causing TestFormattersSBAPI to fail. I un-XFAILED it in r365991.

The breaking xfer:libraries-svr4:read change has been reapplied and broke NetBSD process plugin again. And I've reapplied my earlier fix as r366889.

Lit maintainers have broken NetBSD support in tests by starting to use env -u VAR syntax in r366980. The -u switch is not specified by POSIX, and not supported by NetBSD env(1). In order to fix the problem, I've changed FileCheck's behavior to consider empty envvar as equivalent to disabled (r367122), and then switched lit to set both envvars to empty instead (r367123).

Finally, I've investigated a number of new test failures by the end of the month:

1. New functionalities/signal/handle-abrt test was added in r366580. Since trampolines are not properly supported at the moment, I've marked it XFAIL in r367228.

2. Two functionalities/exec tests started failing since upstream fixed @skipIfSanitized that previously caused the test to be skipped unconditionally (r366903). Since it's not a regression, I've marked it XFAIL in r367228.

3. Same happened for one of the python_api/hello_world tests. It was clearly related to another failing test, so I've marked it XFAIL in r367285.

4. Two tools/lldb-vscode tests were failing since upstream compared realpath'd path with normal path, and our build path happens to include symlinks as one of the parent directories. I've fixed the test to compare realpath in r367291. While at it, I've replaced weird os.path.split(...)[0] with clearer os.path.dirname(...) as suggested by Pavel Labath, in r367290.

NetBSD ptrace() interfaces for thread control

NetBSD currently provides two different methods for thread-related operations.

The legacy method consists of the following requests:

• PT_CONTINUE with negative data argument. It is used to resume execution of a single thread while suspending all other threads.

• PT_STEP with positive data argument. It is used to single-step the specified thread, while all other threads continue execution.

• PT_STEP with negative data argument. It is used to single-step the specified thread, while all other threads remain suspended.

This means that using those methods, you can effectively either:

• run all threads, and optionally send signal to the process,

• run one thread, while keeping other threads suspended,

• single-step one thread, with all other threads either running or being suspended as a whole.

Furthermore, it is impossible to combine single-stepping with syscall tracing via PT_SYSCALL.

The new method introduced by Kamil Rytarowski during his ptrace(2) work is more flexible, and includes the following requests:

• PT_RESUME that sets the specified thread to continue running after PT_CONTINUE.

• PT_SUSPEND that sets the specified thread to remain suspended after PT_CONTINUE.

• PT_SETSTEP that enables single-stepping for the specified thread after PT_CONTINUE.

• PT_CLEARSTEP that disables single-stepping for the specified thread.

Using the new API, it is possible to control both execution and single- stepping per thread, and to combine syscall tracing with that. It is also possible to deliver a single signal either to the whole process or to one of the threads.

Implementing threading in LLDB NetBSD process plugin

When I started my work, the support for threads in the NetBSD plugin was minimal. Technically, the code had structures needed to keep the threads and filled it in at start. However, it did not register new or terminated threads, and it did not support per-thread execution control.

The first change necessary was therefore to implement support for reporting new and terminated threads. I've prepared an initial patch in D65555. With this patch enabled, the thread list command now correctly reports the list of threads at any moment.

The second change necessary is to fix process resuming routine to support multiple threads properly. The routine is passed a data structure containing requested action for each thread. The old code simply took the action for the first thread, and applied it to the whole process. D64647 is my work-in-progress attempt at using the new ptrace calls to apply correct action for each thread.

However, the patch is currently buggy as it assumed that LLDB should provide explicit eStateSuspended action for each thread that is supposed to be supposed. The current LLDB implementation, on the other hand, assumes that thread should be suspended if no action is specified for it. I am currently discussing with upstream whether the current approach is correct, or should be changed to the explicit eStateSuspended usage.

The third change necessary is that we need to explicitly copy debug registers to newly created threads, in order to enable watchpoints on them. However, I haven't gotten to writing a patch for this yet.

Fixing nasty process interrupt bug

While debugging my threading code, I've hit a nasty bug in LLDB. After issuing process interrupt command from remote LLDB session, the server terminated. After putting a lot of effort into debugging why the server terminates with no obvious error, I've discovered that it's terminating because… the client has disconnected.

Further investigation with help of Pavel Labath uncovered that the client is silently disconnecting because it expects a packet indicating that the process has stopped and times out waiting for it. In order to make the server send this packet, NetBSD process plugin needed to explicitly mark process as stopped in the SIGSTOP handler. I've fixed it in r367047.

Future plans

The initial 6 months of my LLDB contract have passed. I am currently taking a month's break from the work, then I will resume it for 3 more months. During that time, I will continue working on threading support and my remaining goals.

The remaining TODO items are:

1. Add support to backtrace through signal trampoline and extend the support to libexecinfo, unwind implementations (LLVM, nongnu). Examine adding CFI support to interfaces that need it to provide more stable backtraces (both kernel and userland).

2. Add support for i386 and aarch64 targets.

3. Stabilize LLDB and address breaking tests from the test suite.

4. Merge LLDB with the base system (under LLVM-style distribution).

This work is sponsored by The NetBSD Foundation

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL to chip in what you can:

https://netbsd.org/donations/#how-to-donate

Upstream describes LLDB as a next generation, high-performance debugger. It is built on top of LLVM/Clang toolchain, and features great integration with it. At the moment, it primarily supports debugging C, C++ and ObjC code, and there is interest in extending it to more languages.

In February, I have started working on LLDB, as contracted by the NetBSD Foundation. So far I've been working on reenabling continuous integration, squashing bugs, improving NetBSD core file support, extending NetBSD's ptrace interface to cover more register types and fix compat32 issues, and lately fixing watchpoint support. You can read more about that in my June 2019 report.

My July's work has been focused on improving support for NetBSD threads in LLDB. This involved a lot of debugging and fighting hanging tests, and I have decided to delay committing the results until I manage to provide fixes for all the immediate issues.

Buildbot updates

During July, upstream has made two breaking changes to the build system:

1. Automatic switching to libc++abi when present in ebuild tree was initially removed in D63883. I needed to force it explicitly because of this. However, upstream has eventually reverted the change.

2. LLDB has enabled Python 3 support, and started requiring SWIG 2+ in the process (D64782). We had to upgrade SWIG on the build host, and eventually switched to Python 3 as well.

As a result of earlier watchpoint fixes, a number of new tests started running. Due to lacking multithreading support, I had to XFAIL a number of LLDB tests in r365338.

A few days later, upstream has fixed the issue causing TestFormattersSBAPI to fail. I un-XFAILED it in r365991.

The breaking xfer:libraries-svr4:read change has been reapplied and broke NetBSD process plugin again. And I've reapplied my earlier fix as r366889.

Lit maintainers have broken NetBSD support in tests by starting to use env -u VAR syntax in r366980. The -u switch is not specified by POSIX, and not supported by NetBSD env(1). In order to fix the problem, I've changed FileCheck's behavior to consider empty envvar as equivalent to disabled (r367122), and then switched lit to set both envvars to empty instead (r367123).

Finally, I've investigated a number of new test failures by the end of the month:

1. New functionalities/signal/handle-abrt test was added in r366580. Since trampolines are not properly supported at the moment, I've marked it XFAIL in r367228.

2. Two functionalities/exec tests started failing since upstream fixed @skipIfSanitized that previously caused the test to be skipped unconditionally (r366903). Since it's not a regression, I've marked it XFAIL in r367228.

3. Same happened for one of the python_api/hello_world tests. It was clearly related to another failing test, so I've marked it XFAIL in r367285.

4. Two tools/lldb-vscode tests were failing since upstream compared realpath'd path with normal path, and our build path happens to include symlinks as one of the parent directories. I've fixed the test to compare realpath in r367291. While at it, I've replaced weird os.path.split(...)[0] with clearer os.path.dirname(...) as suggested by Pavel Labath, in r367290.

NetBSD ptrace() interfaces for thread control

NetBSD currently provides two different methods for thread-related operations.

The legacy method consists of the following requests:

• PT_CONTINUE with negative data argument. It is used to resume execution of a single thread while suspending all other threads.

• PT_STEP with positive data argument. It is used to single-step the specified thread, while all other threads continue execution.

• PT_STEP with negative data argument. It is used to single-step the specified thread, while all other threads remain suspended.

This means that using those methods, you can effectively either:

• run all threads, and optionally send signal to the process,

• run one thread, while keeping other threads suspended,

• single-step one thread, with all other threads either running or being suspended as a whole.

Furthermore, it is impossible to combine single-stepping with syscall tracing via PT_SYSCALL.

The new method introduced by Kamil Rytarowski during his ptrace(2) work is more flexible, and includes the following requests:

• PT_RESUME that sets the specified thread to continue running after PT_CONTINUE.

• PT_SUSPEND that sets the specified thread to remain suspended after PT_CONTINUE.

• PT_SETSTEP that enables single-stepping for the specified thread after PT_CONTINUE.

• PT_CLEARSTEP that disables single-stepping for the specified thread.

Using the new API, it is possible to control both execution and single- stepping per thread, and to combine syscall tracing with that. It is also possible to deliver a single signal either to the whole process or to one of the threads.

Implementing threading in LLDB NetBSD process plugin

When I started my work, the support for threads in the NetBSD plugin was minimal. Technically, the code had structures needed to keep the threads and filled it in at start. However, it did not register new or terminated threads, and it did not support per-thread execution control.

The first change necessary was therefore to implement support for reporting new and terminated threads. I've prepared an initial patch in D65555. With this patch enabled, the thread list command now correctly reports the list of threads at any moment.

The second change necessary is to fix process resuming routine to support multiple threads properly. The routine is passed a data structure containing requested action for each thread. The old code simply took the action for the first thread, and applied it to the whole process. D64647 is my work-in-progress attempt at using the new ptrace calls to apply correct action for each thread.

However, the patch is currently buggy as it assumed that LLDB should provide explicit eStateSuspended action for each thread that is supposed to be supposed. The current LLDB implementation, on the other hand, assumes that thread should be suspended if no action is specified for it. I am currently discussing with upstream whether the current approach is correct, or should be changed to the explicit eStateSuspended usage.

The third change necessary is that we need to explicitly copy debug registers to newly created threads, in order to enable watchpoints on them. However, I haven't gotten to writing a patch for this yet.

Fixing nasty process interrupt bug

While debugging my threading code, I've hit a nasty bug in LLDB. After issuing process interrupt command from remote LLDB session, the server terminated. After putting a lot of effort into debugging why the server terminates with no obvious error, I've discovered that it's terminating because… the client has disconnected.

Further investigation with help of Pavel Labath uncovered that the client is silently disconnecting because it expects a packet indicating that the process has stopped and times out waiting for it. In order to make the server send this packet, NetBSD process plugin needed to explicitly mark process as stopped in the SIGSTOP handler. I've fixed it in r367047.

Future plans

The initial 6 months of my LLDB contract have passed. I am currently taking a month's break from the work, then I will resume it for 3 more months. During that time, I will continue working on threading support and my remaining goals.

The remaining TODO items are:

1. Add support to backtrace through signal trampoline and extend the support to libexecinfo, unwind implementations (LLVM, nongnu). Examine adding CFI support to interfaces that need it to provide more stable backtraces (both kernel and userland).

2. Add support for i386 and aarch64 targets.

3. Stabilize LLDB and address breaking tests from the test suite.

4. Merge LLDB with the base system (under LLVM-style distribution).

This work is sponsored by The NetBSD Foundation

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL to chip in what you can:

https://netbsd.org/donations/#how-to-donate

Parameter Tuning

Evaluation and Results

m2# echo -n 'password'|argon2 somesalt -id -p 2 -t 1 -m 19
Type:           Argon2id
Iterations:     1
Memory:         524288 KiB
Parallelism:    2
Hash:           7b9618bf35b02c00cfef32cb4455206dc400b140116710a6c02732e068021609
Encoded:        $argon2id$v=19$m=524288,t=1,p=2$c29tZXNhbHQ$e5YYvzWwLADP7zLLRFUgbcQAsUARZxCmwCcy4GgCFgk
0.950 seconds
Verification ok

Having established our performance baseline to obtain a best guess for our memory m value, we can turn our attention to parallelism. As touched upon earlier, common suggestions for parallelism p is twice the dedicated core count. Threads in Argon2 form computational lanes which allow work on the memory matrix to be partitioned into independent slices. As such, intuition tells us that by increasing parallelism we should see a decrease in our overall run-time until we are computationally bound. In the graph below, we see the affects of increasing parallelism p over our selected t and m values.
Our baseline memory m value initially exceed our desired upper bound of 1sec without parallelism. Fortunately, we found that increasing the thread count sufficiently parallelizes the work until it starts to settle around p=8. To see if we can further increase our baseline memory m value, we can look at following the graph for argon2id with t=1, p=8. We note that our initial baseline value of m=20 is the maximum m value falling within our bound.

m2# echo -n 'password'|argon2 somesalt -id -m 20 -p 8 -t 1  
Type:           Argon2id
Iterations:     1
Memory:         524288 KiB
Parallelism:    8
Hash:           c62dbebec4a2da3a37dcfa2d82bd2f55541fce80992cd2c1cb887910e859589f
Encoded:        $argon2id$v=19$m=524288,t=1,p=8$c29tZXNhbHQ$xi2+vsSi2jo33Potgr0vVVQfzoCZLNLBy4h5EOhZWJ8
0.858 seconds
Verification ok

Summary

Remaining Tasks

References

Parameter Tuning

Evaluation and Results

m2# echo -n 'password'|argon2 somesalt -id -p 2 -t 1 -m 19
Type:           Argon2id
Iterations:     1
Memory:         524288 KiB
Parallelism:    2
Hash:           7b9618bf35b02c00cfef32cb4455206dc400b140116710a6c02732e068021609
Encoded:        $argon2id$v=19$m=524288,t=1,p=2$c29tZXNhbHQ$e5YYvzWwLADP7zLLRFUgbcQAsUARZxCmwCcy4GgCFgk
0.950 seconds
Verification ok

Having established our performance baseline to obtain a best guess for our memory m value, we can turn our attention to parallelism. As touched upon earlier, common suggestions for parallelism p is twice the dedicated core count. Threads in Argon2 form computational lanes which allow work on the memory matrix to be partitioned into independent slices. As such, intuition tells us that by increasing parallelism we should see a decrease in our overall run-time until we are computationally bound. In the graph below, we see the affects of increasing parallelism p over our selected t and m values.
Our baseline memory m value initially exceed our desired upper bound of 1sec without parallelism. Fortunately, we found that increasing the thread count sufficiently parallelizes the work until it starts to settle around p=8. To see if we can further increase our baseline memory m value, we can look at following the graph for argon2id with t=1, p=8. We note that our initial baseline value of m=20 is the maximum m value falling within our bound.

m2# echo -n 'password'|argon2 somesalt -id -m 20 -p 8 -t 1  
Type:           Argon2id
Iterations:     1
Memory:         524288 KiB
Parallelism:    8
Hash:           c62dbebec4a2da3a37dcfa2d82bd2f55541fce80992cd2c1cb887910e859589f
Encoded:        $argon2id$v=19$m=524288,t=1,p=8$c29tZXNhbHQ$xi2+vsSi2jo33Potgr0vVVQfzoCZLNLBy4h5EOhZWJ8
0.858 seconds
Verification ok

Summary

Remaining Tasks

References

This report was prepared by Manikishan Ghantasala as a part of Google Summer of Code 2019

This report encloses the progress of the project Add KNF (NetBSD style) clang-format configuration during the second coding period of GSoC 2019.

Clang-format

Clang-format is a powerful code formatter which is a part of clang. Clang-format formats the code either by a configuration file .clang-format or can be chosen from some predefined coding styles namely LLVM, Google, Chromium, Mozilla, WebKit.

The final goal of the project is to add a new style NetBSD along with them by patching the libFormat to support the missing styles and add the configuration according to NetBSD KNF.

Style options introduced in the first coding period:

1. BitFieldDeclarationsOnePerLine
2. SortNetBSDIncludes

You can also take a look at the first report to learn more about these style options.

Work done in the second coding period

I have worked on the following styles during this second coding period.

1. Withdrawn SortNetBSDIncludes and modified existing sortIncludes.
2. Modified spacesBeforeTralingComments to support block comments.
3. New style option alignConsecutiveListElements introduced.

Sortincludes:

The native SortIncludes sorts the includes/headers in alphabetical order. And we also have IncludeCategories which allows setting custom priorities to group the includes matching via a regex after sorting.

Example:

Configuration:

    
    IncludeCategories:
        -Regex:	        ^<(a|b|c)/
         Priority:	1

        -Regex:	        ^<(foo)/
         Priority:	2

        -Regex:	        .*
         Priority:	3
    

Input

    
        #include exe.h
        #include gaz.h
        #include <a/dee.h>
        #include <foo/b.h>
        #include <a/bee.h>
        #include <exc.h>
        #include <b/dee.h>
        #include <c/abc.h>
        #include <foo/a.h>

    

Output

    
        #include <a/bee.h>
        #include <a/dee.h>
        #include <b/dee.h>
        #include <c/abc.h>

        #include <foo/a.h>
        #include <foo/b.h>

        #include <exc.h>
        #include gaz.h>


    

Modified SortIncludes

The new sortIncludes supports to give custom priority for sorting in addition to grouping priority. Now the IncludeCategories have a new field named SortPriority along with Priority to set the priority while sorting the includes, and the default priority will be alphabetical. The usage and working of the new IncludeCategories style shown in the following example.

Example

    
    IncludeCategories:
        -Regex:	        <^c/
         Priority:	1
         SortPriority:	0

        -Regex:	        ^<(a|b)/
         Priority:	1
         SortPriority:	1

        -Regex:	        ^<(foo)/
         Priority:	2

        -Regex:	        .*
         Priority:	3   
    

Input

    
        #include exe.h
        #include <a/dee.h>
        #include <foo/b.h>
        #include <a/bee.h>
        #include <exc.h>
        #include <b/dee.h>
        #include <c/abc.h>
        #include <foo/a.h>
    

Output

    
        #include <c/abc.h>
        #include <a/bee.h>
        #include <a/dee.h>
        #include <b/dee.h>
        
        #include <foo/a.h>
        #include <foo/b.h>
        
        #include <exc.h>
        #include gaz.h
        
    

As we observe in the above example, the includes having the same Priority are grouped, and SortPriority defines the sort priority.

The patch was accepted and ready to merge, and you can find the patch here -> SortIncludesPatch. This patch also introduces the NetBSD style to clang-format with configurations to supported styles.

Spaces Before Trailing Comments

This option is also a native style option in clang-format which enables a user to decide the number of spaces to be given before trailing comments (// - comments) but not block comments (/* - comments). The reason for this is that block comments have different usage patterns and different exceptional cases. The following is an example of the native style option.

    
    SpacesBeforeTrailingComments: 3

    void f() {
    	if (true) {  //foo
            f();     //bar
    	}            //foo1
    }

    

Modifications to spaceBeforeTrailingComments:

I am working on modifying this style option to support block comments by covering cases where block comments are used. There are some cases yet to be covered. Once the patch is ready, this is the deliverable

The initial revision can be found here -> patch

    
    SpacesBeforeTrailingComments: 3


    void f() {
        if (true) {  /*foo */
            f();     /*bar */
    	}            /*foo1*/
    }
    

AlignConsecutiveListElements

AlignConsecutiveListElements is a new style option that I am going to introduce to clang-format. This style option aligns elements in consecutive definitions, declarations of lists. The style is not yet ready to patch, and I have to modify a lot of functionalities in alignTokens() which aligns tokens to support this style.

Example:

Input

    
    keys[] = {
        	{"all", f_all, 0 },
        	{ "cbreak", f_cbreak, F_OFFOK },
        	{"cols", f_columns, F_NEEDARG },
        	{ "columns", f_columns, F_NEEDARG },
        };
    

Output

    
    keys[] = {
        	{ "all",        f_all,      0 },
        	{ "cbreak",     _cbreak,    F_OFFOK },
        	{ "cols",       f_columns,  F_NEEDARG },
        	{ "columns",    f_columns,  F_NEEDARG },
        };
    

The blocker to this style is the nested structure for list declarations, alignTokens() should be equipped to parse the nested list declarations to support this style option. I will make sure this style option is available by the next report.

Further plans

For the next phase, I will make all the style options that were modified or introduced till now during the first two phases mergeable to upstream along with required unit tests. With these style options ready, I will test the new clang-format with NetBSD style patched across NetBSD source and check for bugs. After the testing will try to fix the bugs and get the NetBSDStyle ready for the final evaluation.

Summary

In the final coding period, the main focus will be on testing the new/modified style options and fix them. Add any missing styles for NetBSD and get the NetBSDStyle by the final evaluation.

I want to thank my mentors Michal, Christos and developers of both LLVM and NetBSD for supporting to complete this project.

This report was prepared by Manikishan Ghantasala as a part of Google Summer of Code 2019

This report encloses the progress of the project Add KNF (NetBSD style) clang-format configuration during the second coding period of GSoC 2019.

Clang-format

Clang-format is a powerful code formatter which is a part of clang. Clang-format formats the code either by a configuration file .clang-format or can be chosen from some predefined coding styles namely LLVM, Google, Chromium, Mozilla, WebKit.

The final goal of the project is to add a new style NetBSD along with them by patching the libFormat to support the missing styles and add the configuration according to NetBSD KNF.

Style options introduced in the first coding period:

1. BitFieldDeclarationsOnePerLine
2. SortNetBSDIncludes

You can also take a look at the first report to learn more about these style options.

Work done in the second coding period

I have worked on the following styles during this second coding period.

1. Withdrawn SortNetBSDIncludes and modified existing sortIncludes.
2. Modified spacesBeforeTralingComments to support block comments.
3. New style option alignConsecutiveListElements introduced.

Sortincludes:

The native SortIncludes sorts the includes/headers in alphabetical order. And we also have IncludeCategories which allows setting custom priorities to group the includes matching via a regex after sorting.

Example:

Configuration:

    
    IncludeCategories:
        -Regex:	        ^<(a|b|c)/
         Priority:	1

        -Regex:	        ^<(foo)/
         Priority:	2

        -Regex:	        .*
         Priority:	3
    

Input

    
        #include exe.h
        #include gaz.h
        #include <a/dee.h>
        #include <foo/b.h>
        #include <a/bee.h>
        #include <exc.h>
        #include <b/dee.h>
        #include <c/abc.h>
        #include <foo/a.h>

    

Output

    
        #include <a/bee.h>
        #include <a/dee.h>
        #include <b/dee.h>
        #include <c/abc.h>

        #include <foo/a.h>
        #include <foo/b.h>

        #include <exc.h>
        #include gaz.h>


    

Modified SortIncludes

The new sortIncludes supports to give custom priority for sorting in addition to grouping priority. Now the IncludeCategories have a new field named SortPriority along with Priority to set the priority while sorting the includes, and the default priority will be alphabetical. The usage and working of the new IncludeCategories style shown in the following example.

Example

    
    IncludeCategories:
        -Regex:	        <^c/
         Priority:	1
         SortPriority:	0

        -Regex:	        ^<(a|b)/
         Priority:	1
         SortPriority:	1

        -Regex:	        ^<(foo)/
         Priority:	2

        -Regex:	        .*
         Priority:	3   
    

Input

    
        #include exe.h
        #include <a/dee.h>
        #include <foo/b.h>
        #include <a/bee.h>
        #include <exc.h>
        #include <b/dee.h>
        #include <c/abc.h>
        #include <foo/a.h>
    

Output

    
        #include <c/abc.h>
        #include <a/bee.h>
        #include <a/dee.h>
        #include <b/dee.h>
        
        #include <foo/a.h>
        #include <foo/b.h>
        
        #include <exc.h>
        #include gaz.h
        
    

As we observe in the above example, the includes having the same Priority are grouped, and SortPriority defines the sort priority.

The patch was accepted and ready to merge, and you can find the patch here -> SortIncludesPatch. This patch also introduces the NetBSD style to clang-format with configurations to supported styles.

Spaces Before Trailing Comments

This option is also a native style option in clang-format which enables a user to decide the number of spaces to be given before trailing comments (// - comments) but not block comments (/* - comments). The reason for this is that block comments have different usage patterns and different exceptional cases. The following is an example of the native style option.

    
    SpacesBeforeTrailingComments: 3

    void f() {
    	if (true) {  //foo
            f();     //bar
    	}            //foo1
    }

    

Modifications to spaceBeforeTrailingComments:

I am working on modifying this style option to support block comments by covering cases where block comments are used. There are some cases yet to be covered. Once the patch is ready, this is the deliverable

The initial revision can be found here -> patch

    
    SpacesBeforeTrailingComments: 3


    void f() {
        if (true) {  /*foo */
            f();     /*bar */
    	}            /*foo1*/
    }
    

AlignConsecutiveListElements

AlignConsecutiveListElements is a new style option that I am going to introduce to clang-format. This style option aligns elements in consecutive definitions, declarations of lists. The style is not yet ready to patch, and I have to modify a lot of functionalities in alignTokens() which aligns tokens to support this style.

Example:

Input

    
    keys[] = {
        	{"all", f_all, 0 },
        	{ "cbreak", f_cbreak, F_OFFOK },
        	{"cols", f_columns, F_NEEDARG },
        	{ "columns", f_columns, F_NEEDARG },
        };
    

Output

    
    keys[] = {
        	{ "all",        f_all,      0 },
        	{ "cbreak",     _cbreak,    F_OFFOK },
        	{ "cols",       f_columns,  F_NEEDARG },
        	{ "columns",    f_columns,  F_NEEDARG },
        };
    

The blocker to this style is the nested structure for list declarations, alignTokens() should be equipped to parse the nested list declarations to support this style option. I will make sure this style option is available by the next report.

Further plans

For the next phase, I will make all the style options that were modified or introduced till now during the first two phases mergeable to upstream along with required unit tests. With these style options ready, I will test the new clang-format with NetBSD style patched across NetBSD source and check for bugs. After the testing will try to fix the bugs and get the NetBSDStyle ready for the final evaluation.

Summary

In the final coding period, the main focus will be on testing the new/modified style options and fix them. Add any missing styles for NetBSD and get the NetBSDStyle by the final evaluation.

I want to thank my mentors Michal, Christos and developers of both LLVM and NetBSD for supporting to complete this project.

Recently I started working on Fuzzing Filesystems on NetBSD using AFL.
You can take a look at the previous post to learn more details about background of this project.
This post summarizes the work that has been done in this area, and is divided into 3 sections:

1. Porting AFL kernel mode to work with NetBSD
2. Running kernel fuzzing benchmark
3. Example howto fuzzing particular Filesystem

AFL Port for NetBSD

AFL is a well known fuzzer for user space programs and libraries, but with some changes it can be used for fuzzing the kernel binary itself.

For the first step to fuzz the NetBSD kernel via AFL, I needed to modify it to use coverage data provided by the kernel instead of compiled instrumentations. My initial plan was to replace the coverage data gathered via afl-as with that provided by kcov(4). In this scenario, AFL would just run a wrapper and see the real coverage from the kernel.

I also saw previous work done by Oracle in this area, where instead of running the wrapper as a binary, the wrapper code was included in a custom library (.so object).

Both approaches have some pros and cons. One thing that convinced me to use a solution based on the shared library with initialization code was the potentially easier integration with remote fork server. AFL has some constraints in the way of managing fuzzed binary, and keeping it on a remote VM is less portable than fuzzing using a shared library and avoiding introducing changes to the original binary fuzzing.
Porting AFL kernel fuzzing mode to be compatible with NetBSD kernel mainly relied on how the operating system manages the coverage data. The port can be found currently on github.

Writing a kernel fuzzing benchmark

Performance is one of the key factors of fuzzing. If performance of the fuzzing process is not good enough, it's likely that the entire solution won't be useful in practice. In this section we will evaluate our fuzzer with a practice benchmark.

One exercise that I want to perform to check the AFL kernel fuzzing in practice is similar to a password cracking benchmark. The high level idea is that a fuzzer based on coverage should be much smarter than bruteforce or random generation.

To do this, we can write a simple program that will take a text input and compare it with a hardcoded value. If the values match, then the fuzzer cracked the password. Otherwise, it will perform another iteration with a modified input.

Instead of "password cracker", I called my kernel program "lottery dev". It's a character device that takes an input and compares it with a string.

The chance to find one 6 byte combination (or the "lucky byte" combination, because of the name) are similar to winning big in the lottery: every byte contains 8 bits, thus we have 2**(8*6) => 281,474,976,710,656 combinations. The coverage based fuzzer should be able to do this much quicker and in fewer iterations, as feedback from code instrumentations will show, compared to blindly guessing.

I performed a similar test using a simple C program: the program read stdio and compared it with the hardcoded pattern. If the pattern matches, the program panics, and otherwise returns zero. Such a test took an AFL about a few hours on my local laptop to break the challenge (some important details can make it faster). The curious reader who wants to learn some basics of AFL should try to do run a similar test on their machine.

I ran the fuzzer on my lottery dev for several days, and after almost a week it was still not able to find the combination. So something was fundamentally not right. The kernel module with the wrapper code can be found here.

Measuring Coverage for a particular function

In the previous article, I mentioned that the NetBSD kernel seems to be 'more verbose' in terms of coverage reporting. I ran my lottery dev wrapper code (the code that writes a given input to the char device) to check the coverage data using standard kcov(4) without the AFL module. My idea was to check the ratio between entries of my code that I wanted to track and other kernel functions that can be considered noise from other subsystems. These operations are caused by the context services, such as Memory Management, File Systems or Power Management, that are executed in the same process.

To my surprise, there was a lot of data, but I could not find any of functions from lottery dev. I quickly noticed that the amount of addresses is equal to the size of kcov(4) buffer, so maybe my data didn't fit in the buffer inside kernel space?

I changed the size of the coverage buffer to make it significantly larger and recompiled the kernel. With this change I reran the test. Now, with the buffer being large enough, I collected the data and printed top 20 entries with a number of occurrences. There were 30578 entries in total.

1544 /usr/netbsd/src/sys/uvm/uvm_page.c:847
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:869
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:890
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:880
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:858
1281 /usr/netbsd/src/sys/arch/amd64/compile/obj/GENERIC/./machine/cpu.h:70
1281 /usr/netbsd/src/sys/arch/amd64/compile/obj/GENERIC/./machine/cpu.h:71
 478 /usr/netbsd/src/sys/kern/kern_mutex.c:840
 456 /usr/netbsd/src/sys/arch/x86/x86/pmap.c:3046
 438 /usr/netbsd/src/sys/kern/kern_mutex.c:837
 438 /usr/netbsd/src/sys/kern/kern_mutex.c:835
 398 /usr/netbsd/src/sys/kern/kern_mutex.c:838
 383 /usr/netbsd/src/sys/uvm/uvm_page.c:186
 308 /usr/netbsd/src/sys/lib/libkern/../../../common/lib/libc/gen/rb.c:129
 307 /usr/netbsd/src/sys/lib/libkern/../../../common/lib/libc/gen/rb.c:130
 307 /usr/netbsd/src/sys/uvm/uvm_page.c:178
 307 /usr/netbsd/src/sys/uvm/uvm_page.c:1568
 231 /usr/netbsd/src/sys/lib/libkern/../../../common/lib/libc/gen/rb.c:135
 230 /usr/netbsd/src/sys/uvm/uvm_page.c:1567
 228 /usr/netbsd/src/sys/kern/kern_synch.c:416

It should not be a surprise that the coverage data does not much help our fuzzing with AFL. Most of the information that the fuzzer sees is related to UVM page management and machine-dependent code.

I decided to remove instrumentation from these most common functions to check the difference. Using the attribute no_instrument_function tells the compiler to not put instrumentation for coverage tracing inside these functions.

Unfortunately, after recompiling the kernel, the most common functions did not disappear from the list. As I figured out, the support in GCC 7 may not be fully in place.

GCC 8 for help

To solve this issue, I decided to work on reusing GCC 8 for building the NetBSD kernel. After fixing basic build warnings, I got my basic kernel working. This still needs more work to get kcov(4) fully functional. Hopefully, in the next report, I will be able to share these results.

Fuzzing Filesystem

Given what we already know, we can run Filesystem fuzzing. As a target I chose FFS as it is a default FS that is delivered with NetBSD.

The reader may ask the question: why would you run coverage based fuzzer if the data is not 100% accurate? So here is the trick: For coverage based fuzzers, it is usually recommended to leave the input format as is, as genetic algorithms can do a pretty good job here. There is great post on Michal Zalewski's Blog about this process applied for the JPEG format: "Pulling JPEGs out of thin air". But what will AFL do if we provide an input format that is already correct? We already know what a valid FS image should look like (or we can simply just generate one). As it turns out, AFL will start performing operations on the input in a similar way as mutation fuzzers do - another great source that explains this process can be found here: "Binary fuzzing strategies: what works, what doesn't"

Writing mount wrapper

As we discussed in the previous paragraph, to fuzz the kernel itself, we need some code to run operations inside the kernel. We will call it a wrapper, as it wraps the operations of every cycle of fuzzing. The first step to write a wrapper for AFL is to describe it as a sequence of operations. Bash style scripting is usually good enough to do that.
We need to have an input that will be modified by the fuzzer, and be able to mount it. NetBSD comes with vnd(4) that allows exposing regular files as block devices. The simplest sequence can be described as:

# Expose file from tmpfs as block device
vndconfig vnd0 /tmp/rand.tmp

# Create a new FS image on the blk dev that we created
newfs /dev/vnd0

# Mount our fresh FS
mount /dev/vnd0 /mnt

# Check if FS works fine
echo "FFS mounted!" > /mnt/test

# Undo mount
umount /mnt

# Last undo step
vndconfig -u vnd0

From bash to C and system calls

At this point, the reader has probably figured out that a shell script won't be the best approach for fuzzer usage. We need to change it to C code and use proper syscall/libc interfaces.

vndconfig uses the opendisk(3) combined with vnd_ioctl. mount(2) is a simple system call which can operate directly after file is added to vnd(4)

Below is an example conceptual code for mounting an FS:

	// Structure required by mount()
	struct ufs_args ufs_args;

	// VNConfigure step
	rv = run_config(VND_CONFIG, dev, fpath);
	if (rv) 
		printf("VND_CONFIG failed: rv: %d\n", rv);

	// Mount FS
	if (mount(FS_TYPE, fs_name, mntflags, &ufs_args, 8) == -1) {
		printf("Mount failed: %s", strerror(errno));
	} else {
		// Here FS is mounted
		// We can perform any other operations on it
	
		// Umount FS
		if (unmount(fs_name, 0) == -1) printf("#: Umount failed!\n");
	}

	// VNC-unconfigure
	rv = run_config(VND_UNCONFIG, dev, fpath);
	if (rv) {
		printf("VND_UNCONFIG failed: rv: %d\n", rv);
	}

The complete code can be found here

Ready to fuzz FFS! aka Running FS Fuzzing with a predifined corpus

The first thing that we need to do is to have a wrapper that provides mount/umount functionality. In the previous section, we have already shown how that can be done. For now, we will be fuzzing the same kernel that we are running on. Isn't that dangerous? Taking a saw to the branch we are sitting on? Of course it is! In this exercise I want to illustrate an idea from a technical perspective so the curious reader is able to understand it better and do any modifications by their own. The take away from this exercise is that the fuzzing target is the kernel itself, the same binary that is running the fuzzing process.

Let's come back to the wrapper code. We've already discussed how it works. Now we need to compile it as a shared library. This is not obvious, but should be easy to understand after we already brought this sawing-off metaphor.

To compile the so object:

gcc -fPIC -lutil  -g -shared ./wrapper_mount.c -o wrapper_mount.so 

Now we need to create the input corpus, for the first attempt we will use a large enough empty file.

dd if=/dev/zero of=./in/test1 bs=10k count=8

And finally run. The @@ tells AFL to put here the name of input file that will be used for fuzzing.

./afl-fuzz -k -i ./in -o ./out -- /mypath/wrapper_mount.so @@

Now, as we described earlier, we need a proper FS image to allow AFL perform mutations on it. The only difference is the additional newfs(8) command.

# We need a file, big enough to fit FS image but not too big
dd if=/dev/zero of=./in/test1 bs=10k count=8

# A block is already inside fuzzer ./in
vndconfig vnd0 ./in/test1

# Create new FFS filesystem
newfs /dev/vnd0

vndconfig -u vnd0

Now we are ready for another run!

./afl-fuzz -k -i ./in -o ./out -- /mypath/wrapper_mount.so @@



                  american fuzzy lop 2.35b (wrapper_mount.so)

┌─ process timing ─────────────────────────────────────┬─ overall results ─────┐
│        run time : 0 days, 0 hrs, 0 min, 17 sec       │  cycles done : 0      │
│   last new path : none seen yet                      │  total paths : 1      │
│ last uniq crash : none seen yet                      │ uniq crashes : 0      │
│  last uniq hang : none seen yet                      │   uniq hangs : 0      │
├─ cycle progress ────────────────────┬─ map coverage ─┴───────────────────────┤
│  now processing : 0 (0.00%)         │    map density : 17.28% / 17.31%       │
│ paths timed out : 0 (0.00%)         │ count coverage : 3.53 bits/tuple       │
├─ stage progress ────────────────────┼─ findings in depth ────────────────────┤
│  now trying : trim 512/512          │ favored paths : 1 (100.00%)            │
│ stage execs : 15/160 (9.38%)        │  new edges on : 1 (100.00%)            │
│ total execs : 202                   │ total crashes : 0 (0 unique)           │
│  exec speed : 47.74/sec (slow!)     │   total hangs : 0 (0 unique)           │
├─ fuzzing strategy yields ───────────┴───────────────┬─ path geometry ────────┤
│   bit flips : 0/0, 0/0, 0/0                         │    levels : 1          │
│  byte flips : 0/0, 0/0, 0/0                         │   pending : 1          │
│ arithmetics : 0/0, 0/0, 0/0                         │  pend fav : 1          │
│  known ints : 0/0, 0/0, 0/0                         │ own finds : 0          │
│  dictionary : 0/0, 0/0, 0/0                         │  imported : n/a        │
│       havoc : 0/0, 0/0                              │ stability : 23.66%     │
│        trim : n/a, n/a                              ├────────────────────────┘
┴─────────────────────────────────────────────────────┘             [cpu:  0%]

Future work

Support for the NetBSD kernel fuzzing was developed as a part of the AFL FileSystems Fuzzing project, which aims to improve quality of filesystems and catch various issues.
The very next thing that I have on my todo list is to provide support for kernel tracing on GCC 8 to turn off coverage data from other functions that generate a lot of noise.
During the FFS fuzzing, I found a few issues that I need to analyze in detail.
Last but not least, for the next report I plan to show a remote setup of AFL running on a VM, reporting crashes, and being remotely rebooted by the master controller.

Recently I started working on Fuzzing Filesystems on NetBSD using AFL.
You can take a look at the previous post to learn more details about background of this project.
This post summarizes the work that has been done in this area, and is divided into 3 sections:

1. Porting AFL kernel mode to work with NetBSD
2. Running kernel fuzzing benchmark
3. Example howto fuzzing particular Filesystem

AFL Port for NetBSD

AFL is a well known fuzzer for user space programs and libraries, but with some changes it can be used for fuzzing the kernel binary itself.

For the first step to fuzz the NetBSD kernel via AFL, I needed to modify it to use coverage data provided by the kernel instead of compiled instrumentations. My initial plan was to replace the coverage data gathered via afl-as with that provided by kcov(4). In this scenario, AFL would just run a wrapper and see the real coverage from the kernel.

I also saw previous work done by Oracle in this area, where instead of running the wrapper as a binary, the wrapper code was included in a custom library (.so object).

Both approaches have some pros and cons. One thing that convinced me to use a solution based on the shared library with initialization code was the potentially easier integration with remote fork server. AFL has some constraints in the way of managing fuzzed binary, and keeping it on a remote VM is less portable than fuzzing using a shared library and avoiding introducing changes to the original binary fuzzing.
Porting AFL kernel fuzzing mode to be compatible with NetBSD kernel mainly relied on how the operating system manages the coverage data. The port can be found currently on github.

Writing a kernel fuzzing benchmark

Performance is one of the key factors of fuzzing. If performance of the fuzzing process is not good enough, it's likely that the entire solution won't be useful in practice. In this section we will evaluate our fuzzer with a practice benchmark.

One exercise that I want to perform to check the AFL kernel fuzzing in practice is similar to a password cracking benchmark. The high level idea is that a fuzzer based on coverage should be much smarter than bruteforce or random generation.

To do this, we can write a simple program that will take a text input and compare it with a hardcoded value. If the values match, then the fuzzer cracked the password. Otherwise, it will perform another iteration with a modified input.

Instead of "password cracker", I called my kernel program "lottery dev". It's a character device that takes an input and compares it with a string.

The chance to find one 6 byte combination (or the "lucky byte" combination, because of the name) are similar to winning big in the lottery: every byte contains 8 bits, thus we have 2**(8*6) => 281,474,976,710,656 combinations. The coverage based fuzzer should be able to do this much quicker and in fewer iterations, as feedback from code instrumentations will show, compared to blindly guessing.

I performed a similar test using a simple C program: the program read stdio and compared it with the hardcoded pattern. If the pattern matches, the program panics, and otherwise returns zero. Such a test took an AFL about a few hours on my local laptop to break the challenge (some important details can make it faster). The curious reader who wants to learn some basics of AFL should try to do run a similar test on their machine.

I ran the fuzzer on my lottery dev for several days, and after almost a week it was still not able to find the combination. So something was fundamentally not right. The kernel module with the wrapper code can be found here.

Measuring Coverage for a particular function

In the previous article, I mentioned that the NetBSD kernel seems to be 'more verbose' in terms of coverage reporting. I ran my lottery dev wrapper code (the code that writes a given input to the char device) to check the coverage data using standard kcov(4) without the AFL module. My idea was to check the ratio between entries of my code that I wanted to track and other kernel functions that can be considered noise from other subsystems. These operations are caused by the context services, such as Memory Management, File Systems or Power Management, that are executed in the same process.

To my surprise, there was a lot of data, but I could not find any of functions from lottery dev. I quickly noticed that the amount of addresses is equal to the size of kcov(4) buffer, so maybe my data didn't fit in the buffer inside kernel space?

I changed the size of the coverage buffer to make it significantly larger and recompiled the kernel. With this change I reran the test. Now, with the buffer being large enough, I collected the data and printed top 20 entries with a number of occurrences. There were 30578 entries in total.

1544 /usr/netbsd/src/sys/uvm/uvm_page.c:847
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:869
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:890
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:880
1536 /usr/netbsd/src/sys/uvm/uvm_page.c:858
1281 /usr/netbsd/src/sys/arch/amd64/compile/obj/GENERIC/./machine/cpu.h:70
1281 /usr/netbsd/src/sys/arch/amd64/compile/obj/GENERIC/./machine/cpu.h:71
 478 /usr/netbsd/src/sys/kern/kern_mutex.c:840
 456 /usr/netbsd/src/sys/arch/x86/x86/pmap.c:3046
 438 /usr/netbsd/src/sys/kern/kern_mutex.c:837
 438 /usr/netbsd/src/sys/kern/kern_mutex.c:835
 398 /usr/netbsd/src/sys/kern/kern_mutex.c:838
 383 /usr/netbsd/src/sys/uvm/uvm_page.c:186
 308 /usr/netbsd/src/sys/lib/libkern/../../../common/lib/libc/gen/rb.c:129
 307 /usr/netbsd/src/sys/lib/libkern/../../../common/lib/libc/gen/rb.c:130
 307 /usr/netbsd/src/sys/uvm/uvm_page.c:178
 307 /usr/netbsd/src/sys/uvm/uvm_page.c:1568
 231 /usr/netbsd/src/sys/lib/libkern/../../../common/lib/libc/gen/rb.c:135
 230 /usr/netbsd/src/sys/uvm/uvm_page.c:1567
 228 /usr/netbsd/src/sys/kern/kern_synch.c:416

It should not be a surprise that the coverage data does not much help our fuzzing with AFL. Most of the information that the fuzzer sees is related to UVM page management and machine-dependent code.

I decided to remove instrumentation from these most common functions to check the difference. Using the attribute no_instrument_function tells the compiler to not put instrumentation for coverage tracing inside these functions.

Unfortunately, after recompiling the kernel, the most common functions did not disappear from the list. As I figured out, the support in GCC 7 may not be fully in place.

GCC 8 for help

To solve this issue, I decided to work on reusing GCC 8 for building the NetBSD kernel. After fixing basic build warnings, I got my basic kernel working. This still needs more work to get kcov(4) fully functional. Hopefully, in the next report, I will be able to share these results.

Fuzzing Filesystem

Given what we already know, we can run Filesystem fuzzing. As a target I chose FFS as it is a default FS that is delivered with NetBSD.

The reader may ask the question: why would you run coverage based fuzzer if the data is not 100% accurate? So here is the trick: For coverage based fuzzers, it is usually recommended to leave the input format as is, as genetic algorithms can do a pretty good job here. There is great post on Michal Zalewski's Blog about this process applied for the JPEG format: "Pulling JPEGs out of thin air". But what will AFL do if we provide an input format that is already correct? We already know what a valid FS image should look like (or we can simply just generate one). As it turns out, AFL will start performing operations on the input in a similar way as mutation fuzzers do - another great source that explains this process can be found here: "Binary fuzzing strategies: what works, what doesn't"

Writing mount wrapper

As we discussed in the previous paragraph, to fuzz the kernel itself, we need some code to run operations inside the kernel. We will call it a wrapper, as it wraps the operations of every cycle of fuzzing. The first step to write a wrapper for AFL is to describe it as a sequence of operations. Bash style scripting is usually good enough to do that.
We need to have an input that will be modified by the fuzzer, and be able to mount it. NetBSD comes with vnd(4) that allows exposing regular files as block devices. The simplest sequence can be described as:

# Expose file from tmpfs as block device
vndconfig vnd0 /tmp/rand.tmp

# Create a new FS image on the blk dev that we created
newfs /dev/vnd0

# Mount our fresh FS
mount /dev/vnd0 /mnt

# Check if FS works fine
echo "FFS mounted!" > /mnt/test

# Undo mount
umount /mnt

# Last undo step
vndconfig -u vnd0

From bash to C and system calls

At this point, the reader has probably figured out that a shell script won't be the best approach for fuzzer usage. We need to change it to C code and use proper syscall/libc interfaces.

vndconfig uses the opendisk(3) combined with vnd_ioctl. mount(2) is a simple system call which can operate directly after file is added to vnd(4)

Below is an example conceptual code for mounting an FS:

	// Structure required by mount()
	struct ufs_args ufs_args;

	// VNConfigure step
	rv = run_config(VND_CONFIG, dev, fpath);
	if (rv) 
		printf("VND_CONFIG failed: rv: %d\n", rv);

	// Mount FS
	if (mount(FS_TYPE, fs_name, mntflags, &ufs_args, 8) == -1) {
		printf("Mount failed: %s", strerror(errno));
	} else {
		// Here FS is mounted
		// We can perform any other operations on it
	
		// Umount FS
		if (unmount(fs_name, 0) == -1) printf("#: Umount failed!\n");
	}

	// VNC-unconfigure
	rv = run_config(VND_UNCONFIG, dev, fpath);
	if (rv) {
		printf("VND_UNCONFIG failed: rv: %d\n", rv);
	}

The complete code can be found here

Ready to fuzz FFS! aka Running FS Fuzzing with a predifined corpus

The first thing that we need to do is to have a wrapper that provides mount/umount functionality. In the previous section, we have already shown how that can be done. For now, we will be fuzzing the same kernel that we are running on. Isn't that dangerous? Taking a saw to the branch we are sitting on? Of course it is! In this exercise I want to illustrate an idea from a technical perspective so the curious reader is able to understand it better and do any modifications by their own. The take away from this exercise is that the fuzzing target is the kernel itself, the same binary that is running the fuzzing process.

Let's come back to the wrapper code. We've already discussed how it works. Now we need to compile it as a shared library. This is not obvious, but should be easy to understand after we already brought this sawing-off metaphor.

To compile the so object:

gcc -fPIC -lutil  -g -shared ./wrapper_mount.c -o wrapper_mount.so 

Now we need to create the input corpus, for the first attempt we will use a large enough empty file.

dd if=/dev/zero of=./in/test1 bs=10k count=8

And finally run. The @@ tells AFL to put here the name of input file that will be used for fuzzing.

./afl-fuzz -k -i ./in -o ./out -- /mypath/wrapper_mount.so @@

Now, as we described earlier, we need a proper FS image to allow AFL perform mutations on it. The only difference is the additional newfs(8) command.

# We need a file, big enough to fit FS image but not too big
dd if=/dev/zero of=./in/test1 bs=10k count=8

# A block is already inside fuzzer ./in
vndconfig vnd0 ./in/test1

# Create new FFS filesystem
newfs /dev/vnd0

vndconfig -u vnd0

Now we are ready for another run!

./afl-fuzz -k -i ./in -o ./out -- /mypath/wrapper_mount.so @@



                  american fuzzy lop 2.35b (wrapper_mount.so)

┌─ process timing ─────────────────────────────────────┬─ overall results ─────┐
│        run time : 0 days, 0 hrs, 0 min, 17 sec       │  cycles done : 0      │
│   last new path : none seen yet                      │  total paths : 1      │
│ last uniq crash : none seen yet                      │ uniq crashes : 0      │
│  last uniq hang : none seen yet                      │   uniq hangs : 0      │
├─ cycle progress ────────────────────┬─ map coverage ─┴───────────────────────┤
│  now processing : 0 (0.00%)         │    map density : 17.28% / 17.31%       │
│ paths timed out : 0 (0.00%)         │ count coverage : 3.53 bits/tuple       │
├─ stage progress ────────────────────┼─ findings in depth ────────────────────┤
│  now trying : trim 512/512          │ favored paths : 1 (100.00%)            │
│ stage execs : 15/160 (9.38%)        │  new edges on : 1 (100.00%)            │
│ total execs : 202                   │ total crashes : 0 (0 unique)           │
│  exec speed : 47.74/sec (slow!)     │   total hangs : 0 (0 unique)           │
├─ fuzzing strategy yields ───────────┴───────────────┬─ path geometry ────────┤
│   bit flips : 0/0, 0/0, 0/0                         │    levels : 1          │
│  byte flips : 0/0, 0/0, 0/0                         │   pending : 1          │
│ arithmetics : 0/0, 0/0, 0/0                         │  pend fav : 1          │
│  known ints : 0/0, 0/0, 0/0                         │ own finds : 0          │
│  dictionary : 0/0, 0/0, 0/0                         │  imported : n/a        │
│       havoc : 0/0, 0/0                              │ stability : 23.66%     │
│        trim : n/a, n/a                              ├────────────────────────┘
┴─────────────────────────────────────────────────────┘             [cpu:  0%]

Future work

Support for the NetBSD kernel fuzzing was developed as a part of the AFL FileSystems Fuzzing project, which aims to improve quality of filesystems and catch various issues.
The very next thing that I have on my todo list is to provide support for kernel tracing on GCC 8 to turn off coverage data from other functions that generate a lot of noise.
During the FFS fuzzing, I found a few issues that I need to analyze in detail.
Last but not least, for the next report I plan to show a remote setup of AFL running on a VM, reporting crashes, and being remotely rebooted by the master controller.

As NetBSD-9 has finally branched and I keep receiving requests to finish the integration of LLVM sanitizers, I have pushed this task forward too. I have also completed a few leftover tasks from my previous months that still needed fixes.

GNU GDB

Originally the GDB debugger was meant for local debugging. However with the request of tracing remote setups, it has been extended to remote debugging capabilities and this move standardized the protocol for many debuggers.

In fact the native / local / non-remote debugging is redundant with the remote capabilities, however this blog entry is not the right place to discuss the technical merits in comparision between them trying to convince someone. What matters: the developers of LLDB (LLVM debugger) already removed their local debugging variation (for Linux) and implements only the remote process plugin. The NetBSD support for LLDB started with this mode of copying the Linux approach. Bypassing no longer exists for Linux local-debugging-only support plugin.

Inside the GNU world, the transition from local process debugging to remote process debugging is progressing slowly. In the current state of affairs there is a need to support two separate (however whenever possible, with code sharing) plugins for each OS. Also, there is intention from some GNU debugger developers to completely drop the native-only process plugin to debugging code. The state in NetBSD one month ago handled only local-debugging and no code for remote process debugging. The state is the same with other BSD derived systems (FreeBSD, DragonFlyBSD...) and this state was in general kept alive with a reduced set of features compared to Linux.

I have decided to refresh the GDB support for NetBSD as this is still the primary debugger on NetBSD (LLDB support is still work-in-progress) for two primary reasons:

• Slight ptrace(2) changes in the kernel code can alter support in GDB/NetBSD. With proper NetBSD handling in GDB and ability to run its regression tests the fallout will be minimized.
• Certain ptrace(2) reported bugs are in fact specific GDB bugs. In order to close them as resolved I need to revamp and validate the GDB support.

I have managed to get a basic GDB client/server session to work for tracing a simple application. The following features have been implemented targetting as of now NetBSD/amd64:

• creating inferior (debuggee in GDB terminology)
• attaching
• killing
• detaching
• resuming (with optional: single step, signal, resuming single thread)
• waiting
• checking whether a thread is alive
• fetching registers
• storing registers
• reading memory
• writing memory
• requesting interruption
• auxv reading
• software breakpoint support
• hardware single step support
• svr4 ELF loader tracing
• retrieving exec file name from pid
• reading thread name

• i386 support
• multilib support amd64/i386
• reading TLS BASE (x86) register (there is still missing a ptrace(2) interface)
• EXEC events handling improvement in the generic code (that maps Linux restricted semantics only)
• hardware watchpoint/breakpoint support
• FPU support
• FORK, VFORK, POSIX_SPAWN events
• reading loadmap
• research for other features: multiprocess, mulifs, async, btrace, tracepoints, new gdb connections..

The main difficulty was with overly elaborated version of Linux proccess tracing in GDB that contains various workaround for various kernel bugs, handles differences between behavior of the Linux kernel (like different set of ptrace(2) calls or.. swapped arguments) between CPUs.. Finally, I have decided to base my work on unfortunately dated (and probably broken today) gdbserver for lynxos/i386 and then keep adding missing features that are implemented for Linux. I have passed most of the past month on debugging spurious crashes and anomalies in my GDB remote port to NetBSD. After much work, I have finally managed to get gdbserver to work and perform successful operations of spawning a process, inserting a breakpoint, single-stepping, detaching and running a process until its natural completion.

I plan to validate the native tracing support for NetBSD, checking whether it needs upgrading and planning to run the GDB regression tests as soon as possible.

LLVM changes

I have updated the list of ioctl(2) operations to the state as of 9.99.3 (+36 ioctls). The support for NVMM ioctl(2)s has been enabled and restricted as of now to amd64 only.

I have also finally managed to resolve the crash of dynamic (as DSO library) Address Sanitizer that blocked the integration with base in Januray 2019. My original suspect on the root cause was finally verified to be false (mismatch in the source code files composing .so libraries). The real reason to ship with broken dynamic ASan was inability to reuse the same implementation of TSD (Thread-Specific Data) in asan-dynamic. The static (default in LLVM) version of ASan can use in early initialization TLS (Thread-Local Storage), while TSD (Thread-Specific Data) is broken. It happened that the state with dynamic ASan is inversed and TLS breaks and TSD works.

I have also landed build rules for LLVM sanitizers into src/, for:

• asan
• ubsan
• tsan
• msan
• xray
• safestack
• libfuzzer

MAXRSS changes

During the work on libfuzzer last year for GSoC-2018 one of the changes that landed the kernel was the restoration of the MAXRSS option. MAXRSS presents the maxiumum resident set size of the process during its lifetime. In order to finalize this work, there was still need to enable printing it in ps(1). I have pushed a kernel fix to update the RSS related values for sysctl(3) query of a process statistics and with help of Krzysztof Lasocki reenabled printing MAXRSS, IDRSS (integral unshared data), ISRSS (integral unshared stack) and IXRSS (integral shared memory size) in ps(1). These values were commented out in the ps(1) program since the inception of NetBSD, the first registered commit.

$ ps -O maxrss,idrss,isrss,ixrss|head -3
 PID MAXRSS IDRSS ISRSS IXRSS TTY    STAT    TIME COMMAND
 910   5844    20    12   216 pts/0  Is   0:00.01 -ksh
1141   5844    20    12   152 pts/0  I+   0:00.01 mg event.h

These change have been backported to the NetBSD-9 branch and will land 9.0. Welcome back!

Plan for the next milestone

Start executing GDB regression tests. Keep enhancing GDB support. Keep detecting ptrace(2) bugs and addressing them.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL to chip in what you can:

http://netbsd.org/donations/#how-to-donate

As NetBSD-9 has finally branched and I keep receiving requests to finish the integration of LLVM sanitizers, I have pushed this task forward too. I have also completed a few leftover tasks from my previous months that still needed fixes.

GNU GDB

Originally the GDB debugger was meant for local debugging. However with the request of tracing remote setups, it has been extended to remote debugging capabilities and this move standardized the protocol for many debuggers.

In fact the native / local / non-remote debugging is redundant with the remote capabilities, however this blog entry is not the right place to discuss the technical merits in comparision between them trying to convince someone. What matters: the developers of LLDB (LLVM debugger) already removed their local debugging variation (for Linux) and implements only the remote process plugin. The NetBSD support for LLDB started with this mode of copying the Linux approach. Bypassing no longer exists for Linux local-debugging-only support plugin.

Inside the GNU world, the transition from local process debugging to remote process debugging is progressing slowly. In the current state of affairs there is a need to support two separate (however whenever possible, with code sharing) plugins for each OS. Also, there is intention from some GNU debugger developers to completely drop the native-only process plugin to debugging code. The state in NetBSD one month ago handled only local-debugging and no code for remote process debugging. The state is the same with other BSD derived systems (FreeBSD, DragonFlyBSD...) and this state was in general kept alive with a reduced set of features compared to Linux.

I have decided to refresh the GDB support for NetBSD as this is still the primary debugger on NetBSD (LLDB support is still work-in-progress) for two primary reasons:

• Slight ptrace(2) changes in the kernel code can alter support in GDB/NetBSD. With proper NetBSD handling in GDB and ability to run its regression tests the fallout will be minimized.
• Certain ptrace(2) reported bugs are in fact specific GDB bugs. In order to close them as resolved I need to revamp and validate the GDB support.

I have managed to get a basic GDB client/server session to work for tracing a simple application. The following features have been implemented targetting as of now NetBSD/amd64:

• creating inferior (debuggee in GDB terminology)
• attaching
• killing
• detaching
• resuming (with optional: single step, signal, resuming single thread)
• waiting
• checking whether a thread is alive
• fetching registers
• storing registers
• reading memory
• writing memory
• requesting interruption
• auxv reading
• software breakpoint support
• hardware single step support
• svr4 ELF loader tracing
• retrieving exec file name from pid
• reading thread name

• i386 support
• multilib support amd64/i386
• reading TLS BASE (x86) register (there is still missing a ptrace(2) interface)
• EXEC events handling improvement in the generic code (that maps Linux restricted semantics only)
• hardware watchpoint/breakpoint support
• FPU support
• FORK, VFORK, POSIX_SPAWN events
• reading loadmap
• research for other features: multiprocess, mulifs, async, btrace, tracepoints, new gdb connections..

The main difficulty was with overly elaborated version of Linux proccess tracing in GDB that contains various workaround for various kernel bugs, handles differences between behavior of the Linux kernel (like different set of ptrace(2) calls or.. swapped arguments) between CPUs.. Finally, I have decided to base my work on unfortunately dated (and probably broken today) gdbserver for lynxos/i386 and then keep adding missing features that are implemented for Linux. I have passed most of the past month on debugging spurious crashes and anomalies in my GDB remote port to NetBSD. After much work, I have finally managed to get gdbserver to work and perform successful operations of spawning a process, inserting a breakpoint, single-stepping, detaching and running a process until its natural completion.

I plan to validate the native tracing support for NetBSD, checking whether it needs upgrading and planning to run the GDB regression tests as soon as possible.

LLVM changes

I have updated the list of ioctl(2) operations to the state as of 9.99.3 (+36 ioctls). The support for NVMM ioctl(2)s has been enabled and restricted as of now to amd64 only.

I have also finally managed to resolve the crash of dynamic (as DSO library) Address Sanitizer that blocked the integration with base in Januray 2019. My original suspect on the root cause was finally verified to be false (mismatch in the source code files composing .so libraries). The real reason to ship with broken dynamic ASan was inability to reuse the same implementation of TSD (Thread-Specific Data) in asan-dynamic. The static (default in LLVM) version of ASan can use in early initialization TLS (Thread-Local Storage), while TSD (Thread-Specific Data) is broken. It happened that the state with dynamic ASan is inversed and TLS breaks and TSD works.

I have also landed build rules for LLVM sanitizers into src/, for:

• asan
• ubsan
• tsan
• msan
• xray
• safestack
• libfuzzer

MAXRSS changes

During the work on libfuzzer last year for GSoC-2018 one of the changes that landed the kernel was the restoration of the MAXRSS option. MAXRSS presents the maxiumum resident set size of the process during its lifetime. In order to finalize this work, there was still need to enable printing it in ps(1). I have pushed a kernel fix to update the RSS related values for sysctl(3) query of a process statistics and with help of Krzysztof Lasocki reenabled printing MAXRSS, IDRSS (integral unshared data), ISRSS (integral unshared stack) and IXRSS (integral shared memory size) in ps(1). These values were commented out in the ps(1) program since the inception of NetBSD, the first registered commit.

$ ps -O maxrss,idrss,isrss,ixrss|head -3
 PID MAXRSS IDRSS ISRSS IXRSS TTY    STAT    TIME COMMAND
 910   5844    20    12   216 pts/0  Is   0:00.01 -ksh
1141   5844    20    12   152 pts/0  I+   0:00.01 mg event.h

These change have been backported to the NetBSD-9 branch and will land 9.0. Welcome back!

Plan for the next milestone

Start executing GDB regression tests. Keep enhancing GDB support. Keep detecting ptrace(2) bugs and addressing them.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL to chip in what you can:

http://netbsd.org/donations/#how-to-donate

Once upon a time a developer wrote a temperature sensor driver for one system and decided to port it to another, similar, system. For days and then weeks, the developer waited and waited for the other, similar, system to arrive.

One day, joy of joys, the other, similar, system arrived. A National Day of Celebration commenced, and the porting effort began. Over hours, perhaps even as many as five hours of time, the sensors were finally able to tell you whether they were hot or not.

This other, similar, system suddenly was purposeless and was pondering what else life could provide when the Remote Server task popped up and said "Hey, first media file is free", and sadly, this other, similar, system tried its first media file, and then purchased many, many more media files, and suddenly this other, similar, system was suddenly hooked.

Unfortunately, this other, similar, system had a problem talking on the network without falling asleep on the job, so the developer says "let's try a USB network instead!", and initially this seemed like a good idea. Many bits were transferred over USB, but soon whispers of a lurking monster, known to developers, experience or otherwise, as KASSERT, were heard and soon found common.

The developer attempted other USB network as the other, similar, system was destined to be flown many thousands of miles away soon, but the only other option was similarly plagued by the KASSERT monster. The developer reached into his Bag Of Holding and pulled out a magical weapon known capable of slaying the KASSERT monster. The mighty blade of MPSAFE was free again!

After much planning and many failed attacks, the developer was able to slay the KASSERT monster and all the bits wanting to be transferred were able to be.

For a day and for a night there were celebrations. Much food and ale was consumed until finally everyone was asleep, and the music and lights were finally turned off. In the morning a great clean up was needed and while the developer was cleaning off the shiny, happy and working USB network the original USB network was accidentally reconnected. Oh no, the KASSERT monster has returned! Lock your doors and hide your children.

The developer quickly pulled out MPSAFE again, and once again the KASSERT monster was slain, though the face of this monster was different to the previous monster. The developer then searched and searched for surely they were more KASSERT monsters to be found. Indeed, many many others were found, though they retreated to safety after two more of their number were slain by the mighty MPSAFE.

The developer called upon his friends Shared and Code and with them forged a new weapon against the KASSERT monster, using the mighty MPSAFE in ways unheard of before. After much research and planning, and with the help of some friends, the USBNET was born. All the angels and all the birds of the world were said to sing all at once at this moment, for the USBNET would bring happiness to both the Code Deletionist and the Code Sharers, bound to war against each other from time immemorial.

With this new USBNET the developer was able to blaze a trail across the landscape, searching out each KASSERT monster lurking in every USB network corner. All told, fourteen faces of KASSERT monster were found and the developer and his friends have slain seven of these faces, with the remaining seven under attack, life looks grim for them.

The other, similar, system is safe now. Turns out that MPSAFE also has cleared up the sleeping problem using the cousins, NET and FDT in a tight, dual-blade combination.

Let the world rejoice, for soon the KASSERT monster will be no more!

--mrg @ 2019-08-11

tl;dr:

i fixed many bugs across several USB ethernet adapters and got sick of fixing the same bug across multiple drivers so made common code for them to use instead. the original 4 drivers fixed were axen(4), axe(4), cdce(4), and ure(4). the current list of fixed drivers, at time of writing, includes smsc(4), udav(4) and urndis(4). all drivers except umb(4) are ported but either not tested or not yet working with the usbnet(9) framework.

update 2019-09-02:

all 13 known working drivers converted and will be present in netbsd 9.

Once upon a time a developer wrote a temperature sensor driver for one system and decided to port it to another, similar, system. For days and then weeks, the developer waited and waited for the other, similar, system to arrive.

One day, joy of joys, the other, similar, system arrived. A National Day of Celebration commenced, and the porting effort began. Over hours, perhaps even as many as five hours of time, the sensors were finally able to tell you whether they were hot or not.

This other, similar, system suddenly was purposeless and was pondering what else life could provide when the Remote Server task popped up and said "Hey, first media file is free", and sadly, this other, similar, system tried its first media file, and then purchased many, many more media files, and suddenly this other, similar, system was suddenly hooked.

Unfortunately, this other, similar, system had a problem talking on the network without falling asleep on the job, so the developer says "let's try a USB network instead!", and initially this seemed like a good idea. Many bits were transferred over USB, but soon whispers of a lurking monster, known to developers, experience or otherwise, as KASSERT, were heard and soon found common.

The developer attempted other USB network as the other, similar, system was destined to be flown many thousands of miles away soon, but the only other option was similarly plagued by the KASSERT monster. The developer reached into his Bag Of Holding and pulled out a magical weapon known capable of slaying the KASSERT monster. The mighty blade of MPSAFE was free again!

After much planning and many failed attacks, the developer was able to slay the KASSERT monster and all the bits wanting to be transferred were able to be.

For a day and for a night there were celebrations. Much food and ale was consumed until finally everyone was asleep, and the music and lights were finally turned off. In the morning a great clean up was needed and while the developer was cleaning off the shiny, happy and working USB network the original USB network was accidentally reconnected. Oh no, the KASSERT monster has returned! Lock your doors and hide your children.

The developer quickly pulled out MPSAFE again, and once again the KASSERT monster was slain, though the face of this monster was different to the previous monster. The developer then searched and searched for surely they were more KASSERT monsters to be found. Indeed, many many others were found, though they retreated to safety after two more of their number were slain by the mighty MPSAFE.

The developer called upon his friends Shared and Code and with them forged a new weapon against the KASSERT monster, using the mighty MPSAFE in ways unheard of before. After much research and planning, and with the help of some friends, the USBNET was born. All the angels and all the birds of the world were said to sing all at once at this moment, for the USBNET would bring happiness to both the Code Deletionist and the Code Sharers, bound to war against each other from time immemorial.

With this new USBNET the developer was able to blaze a trail across the landscape, searching out each KASSERT monster lurking in every USB network corner. All told, fourteen faces of KASSERT monster were found and the developer and his friends have slain seven of these faces, with the remaining seven under attack, life looks grim for them.

The other, similar, system is safe now. Turns out that MPSAFE also has cleared up the sleeping problem using the cousins, NET and FDT in a tight, dual-blade combination.

Let the world rejoice, for soon the KASSERT monster will be no more!

--mrg @ 2019-08-11

tl;dr:

i fixed many bugs across several USB ethernet adapters and got sick of fixing the same bug across multiple drivers so made common code for them to use instead. the original 4 drivers fixed were axen(4), axe(4), cdce(4), and ure(4). the current list of fixed drivers, at time of writing, includes smsc(4), udav(4) and urndis(4). all drivers except umb(4) are ported but either not tested or not yet working with the usbnet(9) framework.

update 2019-09-02:

all 13 known working drivers converted and will be present in netbsd 9.

This report was written by Naveen Narayanan as part of Google Summer of Code 2019.

This report encompasses the progress of the project during the third coding period. You can make sense of the overall progress of the project by going through the first evaluation report and second evaluation report.

WINE on amd64

Wine-4.4 (released on Mar 2019) is working fine on amd64 and i386. I have been able to use a script as a workaround for the problem of setting LD_LIBRARY_PATH. My patch for setting guard size to 0 and hence, precluding Wine from segfaulting, that got upstreamed, can be found here. I have updated the package to the latest development version of Wine which is Wine-4.13 (released on Aug 2019). I have added support to Wine pkgsrc packages to run tests using make test, and at the time of writing, they are failing. I have also noticed them fail on Linux non-pkgsrc environment and hence, will require further investigation. Initially, they were disabled owing to pkgsrc setting FORTIFY_SOURCE which is a macro that provides support for detecting buffer overflows. In pkgsrc, the wip/wine* packages honor PKGSRC_USE_FORTIFY variable passing _FORTIFY_SOURCE macro accordingly. Programs compiled with FORTIFY_SOURCE substitute wrappers for commonly used libc functions that don't do bounds checking regularly, but could in some cases. Wine unconditionally disables that via their configure script because for some platforms that triggered false positives in the past. However, in my experience, no false positive were found.

Running tests on Wine-4.13 throws errors as you can see below:

mode64# make test
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M adsldp.dll -p adsldp_test.exe.so sysinfo && touch sysinfo.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
002a:err:winediag:SECUR32_initNTLMSP ntlm_auth was not found or is outdated. Make sure that ntlm_auth >= 3.0.25 is in your path. Usually, you can find it in the winbind package of your distribution.
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f150,0x00000001,0x22f120) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022EEA0 000000000002B028 000000000022EE9C): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f0e0,0x00000001,0x22f0b0) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022EE30 000000000002B318 000000000022EE2C): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f0e0,0x00000001,0x22f0b0) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022EE30 000000000002B318 000000000022EE2C): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:adsldp:sysinfo_get_UserName 0000000000021670,000000000022F0C8: stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f980,0x00000001,0x22f950) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022F6D0 000000000002B318 000000000022F6CC): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f980,0x00000001,0x22f950) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022F6D0 000000000002B318 000000000022F6CC): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:adsldp:sysinfo_get_UserName 0000000000021670,000000000022FB48: stub
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so cred && touch cred.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
002e:fixme:cred:CredReadW unhandled type -1
002e:fixme:cred:CredReadW unhandled flags 0xdeadbeef
002e:err:cred:CredWriteW bad username L"winetest"
002e:err:cred:CredWriteW bad username (null)
002e:err:cred:CredWriteW bad username (null)
002e:fixme:cred:CredDeleteW unhandled type -1
002e:fixme:cred:CredDeleteW unhandled flags 0xdeadbeef
002e:fixme:cred:CredReadDomainCredentialsW (0x1b1d0, 0x0, 0x22fa50, 0x22f908) stub
002e:fixme:cred:CredReadDomainCredentialsW (0x1b1d0, 0x0, 0x22fa50, 0x22f908) stub
002e:fixme:cred:CredReadDomainCredentialsW (0x1b3c0, 0x0, 0x22fa50, 0x22f908) stub
cred.c:820: Tests skipped: CRED_TYPE_DOMAIN_VISIBLE_PASSWORD credentials are not supported or are disabled. Skipping
002e:fixme:cred:CredIsMarshaledCredentialW BinaryBlobCredential not checked
002e:fixme:cred:CredIsMarshaledCredentialW BinaryBlobCredential not checked
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt && touch crypt.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_lmhash && touch crypt_lmhash.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_md4 && touch crypt_md4.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_md5 && touch crypt_md5.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_sha && touch crypt_sha.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so eventlog && touch eventlog.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:OpenEventLogW ((null),(null)) stub
0046:fixme:advapi:OpenEventLogW (L"IDontExist",(null)) stub
0046:fixme:advapi:OpenEventLogW (L"IDontExist",L"deadbeef") stub
0046:fixme:advapi:OpenEventLogW Remote server not supported
0046:fixme:advapi:OpenEventLogW ((null),L"deadbeef") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW (L"",L"Application") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:GetEventLogInformation (0x0, 1, 0x0, 0, 0x0) stub
0046:fixme:advapi:GetEventLogInformation (0x0, 0, 0x0, 0, 0x0) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x0, 0, 0x0) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x0, 0, 0x22fb40) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x22fb59, 0, 0x0) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x22fb59, 0, 0x22fb40) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x22fb59, 8, 0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0x0,0x0) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0x0,0x22fb40) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x0) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0x0,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:GetOldestEventLogRecord (0x0,0x0) stub
0046:fixme:advapi:GetOldestEventLogRecord (0x0,0x22fb40) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x0) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:GetOldestEventLogRecord (0x0,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:BackupEventLogW (0x0,(null)) stub
0046:fixme:advapi:BackupEventLogW (0x0,L"backup.evt") stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,(null)) stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:BackupEventLogW (0x0,L"backup2.evt") stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),(null)) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"idontexist.evt") stub
0046:fixme:advapi:OpenBackupEventLogW (L"IDontExist",(null)) stub
0046:fixme:advapi:OpenBackupEventLogW (L"IDontExist",L"idontexist.evt") stub
0046:fixme:advapi:OpenBackupEventLogW Remote server not supported
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
eventlog.c:546: Tests skipped: We don't have a backup eventlog to work with
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x0,0x0) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x22fa88,0x0) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x0,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x0,0x00000000,0x0,0x0) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x0,0x00000000,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000000,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000001,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000002,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x0000000d,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x0000000e,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000007,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa84) stub
eventlog.c:479: Tests skipped: No records in the 'Application' log
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:ClearEventLogW (0x0,(null)) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:ClearEventLogW (0x0,L"backup.evt") stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:ClearEventLogW (0x0,L"backup.evt") stub
0046:fixme:advapi:ClearEventLogW (0x0,L"backup2.evt") stub
0046:fixme:advapi:ClearEventLogW (0x0,(null)) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Wine") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0020,0x0000,0x00000000,0x0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000005,0x00000000,0x1b1e0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ClearEventLogW (0xcafe4242,(null)) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"Wine"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"Wine"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0004,0x0001,0x00000001,0x0,0x0001,0x00000000,0x7f7ff72beb00,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0004,0x0001,0x00000001,0x0,0x0001,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0002,0x0001,0x00000002,0x0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc1"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc1"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0010,0x0001,0x00000003,0x0,0x0002,0x00000000,0x7f7ff72beaf0,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0010,0x0001,0x00000003,0x0,0x0002,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc20") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0001,0x0001,0x00000004,0x0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc300"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc300"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0002,0x0001,0x00000005,0x0,0x0001,0x00000000,0x7f7ff72beb00,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0002,0x0001,0x00000005,0x0,0x0001,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Wine") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0000,0x0002,0x00000006,0x1b3d0,0x0002,0x00000000,0x7f7ff72beaf0,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0000,0x0002,0x00000006,0x1b3d0,0x0002,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0010,0x0002,0x00000007,0x1b3d0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc1") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0008,0x0002,0x00000008,0x1b3d0,0x0002,0x00000000,0x7f7ff72beaf0,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0008,0x0002,0x00000008,0x1b3d0,0x0002,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc20"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc20"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0002,0x0002,0x00000009,0x1b3d0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc300") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0001,0x0002,0x0000000a,0x1b3d0,0x0001,0x00000000,0x7f7ff72beb00,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0001,0x0002,0x0000000a,0x1b3d0,0x0001,0x00000000,0x1b1e0,0x0): stub
0046:err:eventlog:ReportEventW L"First string"
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Wine") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
eventlog.c:889: Tests skipped: No events were written to the eventlog
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "this name is too long", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x0) stub
0046:fixme:advapi:StartTraceA (0x0, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:ControlTraceA (cafe4242, "wine", 0x1b1e0, 1) stub
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so lsa && touch lsa.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
004a:fixme:advapi:LsaOpenPolicy ((null),0x22f960,0x000f0fff,0x22f930) stub
004a:fixme:security:GetWindowsAccountDomainSid (000000000022F690 0000000000019078 000000000022F68C): semi-stub
004a:fixme:advapi:LsaEnumerateAccountRights (0xcafe,0x22fa20,0x22f990,0x22f92c) stub
004a:fixme:advapi:LsaClose (0xcafe) stub
004a:fixme:advapi:LsaOpenPolicy ((null),0x22fb10,0x000f0fff,0x22fae8) stub
004a:fixme:advapi:LsaClose (0xcafe) stub
004a:fixme:advapi:LsaOpenPolicy ((null),0x22fb40,0x00000800,0x22fb00) stub
004a:fixme:advapi:LsaClose (0xcafe) stub
004a:fixme:advapi:LsaOpenPolicy ((null),0x22fb40,0x00000800,0x22fb08) stub
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so registry && touch registry.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
004e:fixme:reg:RegQueryInfoKeyA security argument not supported.
004e:fixme:reg:RegQueryInfoKeyW security argument not supported.
004e:fixme:reg:RegQueryInfoKeyA security argument not supported.
004e:fixme:reg:RegQueryInfoKeyW security argument not supported.
004e:fixme:reg:RegQueryInfoKeyA security argument not supported.
004e:fixme:reg:RegQueryInfoKeyW security argument not supported.
registry.c:2850: Tests skipped: HKCR key merging not supported
registry.c:3146: Tests skipped: HKCR key merging not supported
004e:fixme:winspool:PerfOpen (null): stub
004e:fixme:winspool:PerfCollect L"Global", 0x22ead8, 0x22eabc, 0x22eac0: stub
004e:fixme:winspool:PerfClose stub
004e:fixme:winspool:PerfOpen (null): stub
004e:fixme:winspool:PerfCollect L"invalid counter name", 0x22ead8, 0x22eabc, 0x22eac0: stub
004e:fixme:winspool:PerfClose stub
registry.c:4032: Test failed: [ 9] expected 1168, got 2
registry.c:4032: Test failed: [10] expected 1814, got 2
*** Error code 2

Stop.
make[1]: stopped in /usr/pkgsrc/wip/wine64-dev/work/wine-4.13/wine64/dlls/advapi32/tests
*** Error code 1

Stop.
make: stopped in /usr/pkgsrc/wip/wine64-dev/work/wine-4.13/wine64

Running programs on Wine

You can find obligatory screenshots of Wine-4.4/4.13 on amd64/i386 below:

010 Editor (Professional Text/Hex Editor) on Wine-4.13 (amd64)

Notepad++ on Wine-4.4 (i386)

Pinball on Wine-4.13 (amd64)

How to run Wine on NetBSD/amd64

• Compile kernel with USER_LDT enabled.
• Install kernel.
• Clone wip repo.
• cd /usr/pkgsrc/wip/wine64; make install (for 4.4)
or
• cd /usr/pkgsrc/wip/wine64-dev; make install (for 4.13)
• wine notepad

Future Plans

Wine requires the kernel option USER_LDT to be able to run 32-bit applications on amd64 - facilitated by WoW64. Presently, this feature isn't enabled by default on NetBSD and hence, the kernel has to be compiled with USER_LDT enabled. I will work on getting the tests to pass and finding a better approach to deal with LD_LIBRARY_PATH issue. In addition to that, I shall work on incorporating a NetBSD VM to Wine Test Bot infrastructure so we can preclude Wine from getting out of shape on NetBSD in the future (work in progress).

Summary

Presently, Wine-4.4 and Wine-4.13 are working fine on amd64/i386. I have been able to meet all the goals as per my original plan. I would like to attribute the success to the support provided by my mentors @leot, @maya and @christos. I would also like to thank my mentor @maxv for solving the conflict between SVS and USER_LDT kernel options. I intend to maintain the packages wine64/wine64-dev/wine32 for the foreseeable future. As stated above, I shall try to set up a NetBSD VM as Wine Test Bot for purpose of CI. Once again, thanks to Google for enabling me to do what I love.

Source Code

This report was written by Naveen Narayanan as part of Google Summer of Code 2019.

This report encompasses the progress of the project during the third coding period. You can make sense of the overall progress of the project by going through the first evaluation report and second evaluation report.

WINE on amd64

Wine-4.4 (released on Mar 2019) is working fine on amd64 and i386. I have been able to use a script as a workaround for the problem of setting LD_LIBRARY_PATH. My patch for setting guard size to 0 and hence, precluding Wine from segfaulting, that got upstreamed, can be found here. I have updated the package to the latest development version of Wine which is Wine-4.13 (released on Aug 2019). I have added support to Wine pkgsrc packages to run tests using make test, and at the time of writing, they are failing. I have also noticed them fail on Linux non-pkgsrc environment and hence, will require further investigation. Initially, they were disabled owing to pkgsrc setting FORTIFY_SOURCE which is a macro that provides support for detecting buffer overflows. In pkgsrc, the wip/wine* packages honor PKGSRC_USE_FORTIFY variable passing _FORTIFY_SOURCE macro accordingly. Programs compiled with FORTIFY_SOURCE substitute wrappers for commonly used libc functions that don't do bounds checking regularly, but could in some cases. Wine unconditionally disables that via their configure script because for some platforms that triggered false positives in the past. However, in my experience, no false positive were found.

Running tests on Wine-4.13 throws errors as you can see below:

mode64# make test
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M adsldp.dll -p adsldp_test.exe.so sysinfo && touch sysinfo.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
002a:err:winediag:SECUR32_initNTLMSP ntlm_auth was not found or is outdated. Make sure that ntlm_auth >= 3.0.25 is in your path. Usually, you can find it in the winbind package of your distribution.
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f150,0x00000001,0x22f120) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022EEA0 000000000002B028 000000000022EE9C): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f0e0,0x00000001,0x22f0b0) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022EE30 000000000002B318 000000000022EE2C): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f0e0,0x00000001,0x22f0b0) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022EE30 000000000002B318 000000000022EE2C): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:adsldp:sysinfo_get_UserName 0000000000021670,000000000022F0C8: stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f980,0x00000001,0x22f950) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022F6D0 000000000002B318 000000000022F6CC): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:advapi:LsaOpenPolicy ((null),0x22f980,0x00000001,0x22f950) stub
002a:fixme:security:GetWindowsAccountDomainSid (000000000022F6D0 000000000002B318 000000000022F6CC): semi-stub
002a:fixme:advapi:LsaClose (0xcafe) stub
002a:fixme:adsldp:sysinfo_get_UserName 0000000000021670,000000000022FB48: stub
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so cred && touch cred.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
002e:fixme:cred:CredReadW unhandled type -1
002e:fixme:cred:CredReadW unhandled flags 0xdeadbeef
002e:err:cred:CredWriteW bad username L"winetest"
002e:err:cred:CredWriteW bad username (null)
002e:err:cred:CredWriteW bad username (null)
002e:fixme:cred:CredDeleteW unhandled type -1
002e:fixme:cred:CredDeleteW unhandled flags 0xdeadbeef
002e:fixme:cred:CredReadDomainCredentialsW (0x1b1d0, 0x0, 0x22fa50, 0x22f908) stub
002e:fixme:cred:CredReadDomainCredentialsW (0x1b1d0, 0x0, 0x22fa50, 0x22f908) stub
002e:fixme:cred:CredReadDomainCredentialsW (0x1b3c0, 0x0, 0x22fa50, 0x22f908) stub
cred.c:820: Tests skipped: CRED_TYPE_DOMAIN_VISIBLE_PASSWORD credentials are not supported or are disabled. Skipping
002e:fixme:cred:CredIsMarshaledCredentialW BinaryBlobCredential not checked
002e:fixme:cred:CredIsMarshaledCredentialW BinaryBlobCredential not checked
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt && touch crypt.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_lmhash && touch crypt_lmhash.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_md4 && touch crypt_md4.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_md5 && touch crypt_md5.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so crypt_sha && touch crypt_sha.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so eventlog && touch eventlog.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:OpenEventLogW ((null),(null)) stub
0046:fixme:advapi:OpenEventLogW (L"IDontExist",(null)) stub
0046:fixme:advapi:OpenEventLogW (L"IDontExist",L"deadbeef") stub
0046:fixme:advapi:OpenEventLogW Remote server not supported
0046:fixme:advapi:OpenEventLogW ((null),L"deadbeef") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW (L"",L"Application") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:GetEventLogInformation (0x0, 1, 0x0, 0, 0x0) stub
0046:fixme:advapi:GetEventLogInformation (0x0, 0, 0x0, 0, 0x0) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x0, 0, 0x0) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x0, 0, 0x22fb40) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x22fb59, 0, 0x0) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x22fb59, 0, 0x22fb40) stub
0046:fixme:advapi:GetEventLogInformation (0xcafe4242, 0, 0x22fb59, 8, 0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0x0,0x0) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0x0,0x22fb40) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x0) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0x0,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:GetOldestEventLogRecord (0x0,0x0) stub
0046:fixme:advapi:GetOldestEventLogRecord (0x0,0x22fb40) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x0) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:GetOldestEventLogRecord (0x0,0x22fb40) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:BackupEventLogW (0x0,(null)) stub
0046:fixme:advapi:BackupEventLogW (0x0,L"backup.evt") stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,(null)) stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:BackupEventLogW (0x0,L"backup2.evt") stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),(null)) stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"idontexist.evt") stub
0046:fixme:advapi:OpenBackupEventLogW (L"IDontExist",(null)) stub
0046:fixme:advapi:OpenBackupEventLogW (L"IDontExist",L"idontexist.evt") stub
0046:fixme:advapi:OpenBackupEventLogW Remote server not supported
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
eventlog.c:546: Tests skipped: We don't have a backup eventlog to work with
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x0,0x0) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x22fa88,0x0) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x0,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000000,0x00000000,0x0,0x00000000,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x0,0x00000000,0x0,0x0) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x0,0x00000000,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0x0,0x00000005,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000000,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000001,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000002,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x0000000d,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x0000000e,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000007,0x00000000,0x1b3d0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa84) stub
eventlog.c:479: Tests skipped: No records in the 'Application' log
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:ClearEventLogW (0x0,(null)) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Application") stub
0046:fixme:advapi:BackupEventLogW (0xcafe4242,L"backup.evt") stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:ClearEventLogW (0x0,L"backup.evt") stub
0046:fixme:advapi:OpenBackupEventLogW ((null),L"backup.evt") stub
0046:fixme:advapi:ClearEventLogW (0x0,L"backup.evt") stub
0046:fixme:advapi:ClearEventLogW (0x0,L"backup2.evt") stub
0046:fixme:advapi:ClearEventLogW (0x0,(null)) stub
0046:fixme:advapi:CloseEventLog (0x0) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Wine") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0020,0x0000,0x00000000,0x0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:ReadEventLogA (0xcafe4242,0x00000005,0x00000000,0x1b1e0,0x00000038,0x22fa88,0x22fa8c) stub
0046:fixme:advapi:ClearEventLogW (0xcafe4242,(null)) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"Wine"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"Wine"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0004,0x0001,0x00000001,0x0,0x0001,0x00000000,0x7f7ff72beb00,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0004,0x0001,0x00000001,0x0,0x0001,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0002,0x0001,0x00000002,0x0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc1"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc1"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0010,0x0001,0x00000003,0x0,0x0002,0x00000000,0x7f7ff72beaf0,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0010,0x0001,0x00000003,0x0,0x0002,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc20") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0001,0x0001,0x00000004,0x0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc300"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc300"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0002,0x0001,0x00000005,0x0,0x0001,0x00000000,0x7f7ff72beb00,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0002,0x0001,0x00000005,0x0,0x0001,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Wine") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0000,0x0002,0x00000006,0x1b3d0,0x0002,0x00000000,0x7f7ff72beaf0,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0000,0x0002,0x00000006,0x1b3d0,0x0002,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0010,0x0002,0x00000007,0x1b3d0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc1") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0008,0x0002,0x00000008,0x1b3d0,0x0002,0x00000000,0x7f7ff72beaf0,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0008,0x0002,0x00000008,0x1b3d0,0x0002,0x00000000,0x1b1e0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:RegisterEventSourceA ((null),"WineSrc20"): stub
0046:fixme:advapi:RegisterEventSourceW (L"",L"WineSrc20"): stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0002,0x0002,0x00000009,0x1b3d0,0x0000,0x00000000,0x0,0x0): stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:DeregisterEventSource (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"WineSrc300") stub
0046:fixme:advapi:ReportEventA (0xcafe4242,0x0001,0x0002,0x0000000a,0x1b3d0,0x0001,0x00000000,0x7f7ff72beb00,0x0): stub
0046:fixme:advapi:ReportEventW (0xcafe4242,0x0001,0x0002,0x0000000a,0x1b3d0,0x0001,0x00000000,0x1b1e0,0x0): stub
0046:err:eventlog:ReportEventW L"First string"
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:GetOldestEventLogRecord (0xcafe4242,0x22fa8c) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
0046:fixme:advapi:OpenEventLogW ((null),L"Wine") stub
0046:fixme:advapi:GetNumberOfEventLogRecords (0xcafe4242,0x22fa80) stub
0046:fixme:advapi:CloseEventLog (0xcafe4242) stub
eventlog.c:889: Tests skipped: No events were written to the eventlog
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "this name is too long", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x0) stub
0046:fixme:advapi:StartTraceA (0x0, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:StartTraceA (0x22fb40, "wine", 0x1b1e0) stub
0046:fixme:advapi:ControlTraceA (cafe4242, "wine", 0x1b1e0, 1) stub
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so lsa && touch lsa.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
004a:fixme:advapi:LsaOpenPolicy ((null),0x22f960,0x000f0fff,0x22f930) stub
004a:fixme:security:GetWindowsAccountDomainSid (000000000022F690 0000000000019078 000000000022F68C): semi-stub
004a:fixme:advapi:LsaEnumerateAccountRights (0xcafe,0x22fa20,0x22f990,0x22f92c) stub
004a:fixme:advapi:LsaClose (0xcafe) stub
004a:fixme:advapi:LsaOpenPolicy ((null),0x22fb10,0x000f0fff,0x22fae8) stub
004a:fixme:advapi:LsaClose (0xcafe) stub
004a:fixme:advapi:LsaOpenPolicy ((null),0x22fb40,0x00000800,0x22fb00) stub
004a:fixme:advapi:LsaClose (0xcafe) stub
004a:fixme:advapi:LsaOpenPolicy ((null),0x22fb40,0x00000800,0x22fb08) stub
/usr/pkgsrc/wip/wine64-dev/work/wine-4.13/tools/runtest -q -P wine -T ../../.. -M advapi32.dll -p advapi32_test.exe.so registry && touch registry.ok
/usr/pkg/emul/netbsd32/lib/wine/libwine.so.1: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/ntdll.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernel32.dll.so: text relocations
/usr/pkg/emul/netbsd32/lib/wine/wine/kernelbase.dll.so: text relocations
004e:fixme:reg:RegQueryInfoKeyA security argument not supported.
004e:fixme:reg:RegQueryInfoKeyW security argument not supported.
004e:fixme:reg:RegQueryInfoKeyA security argument not supported.
004e:fixme:reg:RegQueryInfoKeyW security argument not supported.
004e:fixme:reg:RegQueryInfoKeyA security argument not supported.
004e:fixme:reg:RegQueryInfoKeyW security argument not supported.
registry.c:2850: Tests skipped: HKCR key merging not supported
registry.c:3146: Tests skipped: HKCR key merging not supported
004e:fixme:winspool:PerfOpen (null): stub
004e:fixme:winspool:PerfCollect L"Global", 0x22ead8, 0x22eabc, 0x22eac0: stub
004e:fixme:winspool:PerfClose stub
004e:fixme:winspool:PerfOpen (null): stub
004e:fixme:winspool:PerfCollect L"invalid counter name", 0x22ead8, 0x22eabc, 0x22eac0: stub
004e:fixme:winspool:PerfClose stub
registry.c:4032: Test failed: [ 9] expected 1168, got 2
registry.c:4032: Test failed: [10] expected 1814, got 2
*** Error code 2

Stop.
make[1]: stopped in /usr/pkgsrc/wip/wine64-dev/work/wine-4.13/wine64/dlls/advapi32/tests
*** Error code 1

Stop.
make: stopped in /usr/pkgsrc/wip/wine64-dev/work/wine-4.13/wine64

Running programs on Wine

You can find obligatory screenshots of Wine-4.4/4.13 on amd64/i386 below:

010 Editor (Professional Text/Hex Editor) on Wine-4.13 (amd64)

Notepad++ on Wine-4.4 (i386)

Pinball on Wine-4.13 (amd64)

How to run Wine on NetBSD/amd64

• Compile kernel with USER_LDT enabled.
• Install kernel.
• Clone wip repo.
• cd /usr/pkgsrc/wip/wine64; make install (for 4.4)
or
• cd /usr/pkgsrc/wip/wine64-dev; make install (for 4.13)
• wine notepad

Future Plans

Wine requires the kernel option USER_LDT to be able to run 32-bit applications on amd64 - facilitated by WoW64. Presently, this feature isn't enabled by default on NetBSD and hence, the kernel has to be compiled with USER_LDT enabled. I will work on getting the tests to pass and finding a better approach to deal with LD_LIBRARY_PATH issue. In addition to that, I shall work on incorporating a NetBSD VM to Wine Test Bot infrastructure so we can preclude Wine from getting out of shape on NetBSD in the future (work in progress).

Summary

Presently, Wine-4.4 and Wine-4.13 are working fine on amd64/i386. I have been able to meet all the goals as per my original plan. I would like to attribute the success to the support provided by my mentors @leot, @maya and @christos. I would also like to thank my mentor @maxv for solving the conflict between SVS and USER_LDT kernel options. I intend to maintain the packages wine64/wine64-dev/wine32 for the foreseeable future. As stated above, I shall try to set up a NetBSD VM as Wine Test Bot for purpose of CI. Once again, thanks to Google for enabling me to do what I love.

Source Code

This report was prepared by Manikishan Ghantasala as a part of Google Summer of Code 2019

This is the third and final report of the project Add KNF (NetBSD style) clang-format configuration that I have been doing as a part of Google Summer of Code (GSoC) ‘19 with the NetBSD.

You can refer to the first and second reports here:

1. Adding NetBSD KNF to clang-format, Part 1
2. Adding NetBSD KNF to clang-format, Part 2

About the project

ClangFormat is a set of tools to format C/C++/Java/JavaScript/Objective-C/Protobuf code. It is built on top of LibFormat to support workflow in various ways including a standalone tool called clang-format, and editor integrations. It supports a few built-in CodingStyles that include: LLVM, Google, Chromium, Mozilla, Webkit. When the desired code formatting style is different from the available options, the style can be customized using a configuration file. The aim of this project is to add NetBSD KNF support to clang-format and new styles to libFormat that support NetBSD’s style of coding. This would allow us to format NetBSD code by passing `-style=NetBSD` as an argument.

How to use clang-format

While using clang-format one can choose a style from the predefined styles or create a custom style by configuring specific style options.

Use the following command if you are using one of the predefined styles

Configuring style with clang-format

clang-format supports two ways to provide custom style options: directly specify style name in the -style= command line option or use -style=file and put style configuration in a .clang-format or _clang-format file in the project’s top directory.

Check Clang-Format Style Options to know how different Style Options works and how to use them.

When specifying configuration in the -style= option, the same configuration is applied for all input files. The format of the configuration is:

The .clang-format file uses YAML format. An easy way to get a valid .clang-format file containing all configuration options of a certain predefined style is:

After making required changes to the .clang-format file it can be used as a custom Style by:

Changes made to clang-format

The following changes were made to the clang-format as a part of adding NetBSD KNF:

New Style options added:

Modifications made to existing styles:

1. Modified SortIncludes  and IncludeCategories to support NetBSD like includes.
2. Modified SpacesBeforeTrailingComments to support block comments.

 The new NetBSD Style Configurations:

This is the final configurations for clang-format with modified changes to support NetBSD KNF.

    
        AlignTrailingComments: true
        AlwaysBreakAfterReturnType: All
        AlignConsecutiveMacros: true
        AlignConsecutiveLists: true
        BitFieldDeclarationsOnePerLine: true
        BreakBeforeBraces: Mozilla
        ColumnLimit: 80
        ContinuationIndentWidth: 4
        Cpp11BracedListStyle: false
        FixNamespaceComments: true
        IndentCaseLabels: false
        IndentWidth: 8
        IncludeBlocks: Regroup
        IncludeCategories:
         - Regex: '^<sys/param\.h>'
           Priority: 1
           SortPriority: 0
         - Regex: '^<sys/types\.h>'
           Priority: 1
           SortPriority: 1
         - Regex: '^<sys.*/'
           Priority: 1
           SortPriority: 2
         - Regex: '^<uvm/'
           Priority: 2
           SortPriority: 3
         - Regex: '^<machine/'
           Priority: 3
           SortPriority: 4
         - Regex: '^<dev/'
           Priority: 4
           SortPriority: 5
         - Regex: '^<net.*/'
           Priority: 5
           SortPriority: 6
         - Regex: '^<protocols/'
           Priority: 5
           SortPriority: 7
         - Regex: '^<(fs|miscfs|msdosfs|nfs|ufs)/'
           Priority: 6
           SortPriority: 8
         - Regex: '^<(x86|amd64|i386|xen)/'
           Priority: 7
           SortPriority: 8
         - Regex: '^<path'
           Priority: 9
           SortPriority: 11
         - Regex: '^<[^/].*\.h'
           Priority: 8
           SortPriority: 10
         - Regex: '^\".*\.h\"'
           Priority: 10
           SortPriority: 12   
        SortIncludes: true
        SpacesBeforeCpp11BracedList: true
        SpacesBeforeTrailingComments: 4
        TabWidth: 8
        UseTab: Always

    

Status of each Style Option

Styles Ready to Merge:

1.Modified SortIncludes and IncludeCategories:

     Patch: https://reviews.llvm.org/D64695

Styles needing revision

1.BitFieldDeclarationsOnePerLine:

    Patch: https://reviews.llvm.org/D63062

    Bugs: 1 

 2.SpacesBeforeTrailingComments supports Block Comments:

    Patch: https://reviews.llvm.org/D65648

    Remark:  I have to discuss more on the cases that Block comments are used but where the Spaces are not to be added before them.

WIP Style

1.AlignConsecutiveListElements:

    Commit: https://github.com/sh4nnu/clang/commit/4b4cd45a5f3d211008763f1c0235a22352faa81e

    Bugs: 1

About Styles

BitfieldDeclarationsOnePerLine:

    Patch: https://reviews.llvm.org/D63062 

    This style lines up BitField declarations on consecutive lines with correct indentation.

    
            unsigned int bas :3, hh : 4, jjj : 8;
            
            
            unsigned int baz:1,
                            fuz:5,
                zap:2;
    

            unsigned  int bas : 3, 
                        hh  : 4, 
                        jjj : 8;
            
            
            unsigned int baz:1,
                        fuz:5,
                        zap:2;

Bug: Indentation breaks in the presence of block comments in between.

Input:

        
                    unsigned  int bas : 3, /* foo */
                                  hh  : 4, /* bar */
                                  jjj : 8;
                    
        
        

Output:

        
                    unsigned  int bas : 3, /* foo */
                        hh  : 4, /* bar */
                        jjj : 8;

        
        

Modification for SortIncludes and IncludeCategories:

    Patch: https://reviews.llvm.org/D64695

    Status: Accepted, and ready to land.

Clang-format has a style option named SortIncludes which sorts the includes in alphabetical order.The IncludeCategories Style allows us to define a custom order for sorting the includes.

It supports POSIX extended regular expressions to assign Categories for includes.

The SortIncludes then sorts the #includes first according to increasing category number and then lexically within each category. When IncludeBlocks  is set to Regroup  merge multiple #includes blocks together and sort as one. Then split into groups based on category priority.

The problem arises when you want to define the order within each category, which is not supported. In this modification a new field named SortPriority is added, which is optional.

The #includes matching the regexs are sorted according to the values of SortPriority, and Regrouping after sorting is done according to the values of Priority. If SortPriority is not defined it is set to the value of Priority as a default value.

    
        IncludeCategories:
         - Regex: ‘<^c/’
           Priority: 1
           SortPriority: 0
        
         - Regex: ‘^<(a|b)/’
           Priority: 1
           SortPriority: 1
        
         - Regex: ‘^<(foo)/’
           Priority: 2
      
         - Regex: ‘.*’
           Priority: 3
           

    

    
            #include "exe.h"
            #include <a/dee.h>
            #include <foo/b.h>
            #include <a/bee.h>
            #include <exc.h>
            #include <b/dee.h>
            #include <c/abc.h>
            #include <foo/a.h>
    

    
            #include <c/abc.h>
            #include <a/bee.h>
            #include <a/dee.h>
            #include <b/dee.h>
            
            #include <foo/a.h>
            #include <foo/b.h>
            
            #include <exc.h>
            #include "exe.h"

    

As you can observe in the above example the #includes are grouped by different priority and were sorted by different priority. Introduction of this new patch doesn’t affect the old configurations as it will work as the same old SortIncludes if sortPriority is not defined.

Refer to Report 2 for detailed examples on this.

Modification for SpacesBeforeTrailingComments

    Patch: https://reviews.llvm.org/D64695

The SpacesBeforeTrailingComments is modified to support Block Comments which was used to support only line comments. The reason for this is block comments have different usage patterns and different exceptional cases. I have tried to exclude cases where some tests doesn’t support spaces before block comments. I have been discussing in the community for getting to know which cases should be included, and which to exclude.

Cases that were excluded due to failing tests:

• If it is a Preprocessor directive,
• If it is followed by a LeftParenthesis
• And if it is after a Template closer

AlignConsecutiveListElements

    Status: Work In Progress

This is a new style that aligns elements of consecutive lists in a nested list. The Style is still in work in progress. There are few cases that needed to be covered and fix few bugs.

    
            keys[] = {
                {"all", f_all, 0 },
                { "cbreak", f_cbreak, F_OFFOK },
                {"cols", f_columns, F_NEEDARG },
                { "columns", f_columns, F_NEEDARG },
             };
    

    
            keys[] = { { "all",      f_all,        0 },
                       { "cbreak",   _cbreak,      F_OFFOK },
                       { "cols",     f_columns,    F_NEEDARG }, 
                       { "columns",  f_columns,    F_NEEDARG },
                     };

    

This style option aligns list declarations that are nested inside a list, I would also like to extend this style to align individual single line list declarations that are consecutive.

The problem with this style is the case in which there can be different number of elements for each individual.

    
            keys[] =  { "all",        f_all,        0 };
            keys2[] = { "cbreak",     _cbreak,      F_OFFOK };
            keys3[] = { "cols",       f_columns,    F_NEEDARG,     7 };
            keys4[] = { "columns",    f_columns };
    

Future Work

Some Style Options that were introduced during this GSoC were made in order to meet all the cases in NetBSD KNF. So they may need some revisions with respect to other languages and coding styles that clang-format supports. I will continue working on this project even after the GSoC period on the style options that are yet to be merged and add new style options if necessary and will get the NetBSD Style merged with upstream which is the final deliverable for the project. I would like to take up the responsibility of maintaining the “NetBSD KNF” support to clang-format.

Summary

Even though officially the GSoC’19 coding period is over, I definitely look forward to keep contributing to this project. This summer has had me digging a lot into the code from CLANG and NetBSD for references for creating or modifying the Style Options. I am pretty much interested to work with NetBSD again, I like being in the community and I would like to improve my skills and learn more about Operating Systems by contributing to this organisation.

I would like to thank my mentors Michal and Christos for their constant support and patient guidance. A huge thanks to both the NetBSD and LLVM community who have been supportive and have helped me whenever I have had trouble. Finally a huge thanks to Google for providing me this opportunity.

This report was prepared by Manikishan Ghantasala as a part of Google Summer of Code 2019

This is the third and final report of the project Add KNF (NetBSD style) clang-format configuration that I have been doing as a part of Google Summer of Code (GSoC) ‘19 with the NetBSD.

You can refer to the first and second reports here:

1. Adding NetBSD KNF to clang-format, Part 1
2. Adding NetBSD KNF to clang-format, Part 2

About the project

ClangFormat is a set of tools to format C/C++/Java/JavaScript/Objective-C/Protobuf code. It is built on top of LibFormat to support workflow in various ways including a standalone tool called clang-format, and editor integrations. It supports a few built-in CodingStyles that include: LLVM, Google, Chromium, Mozilla, Webkit. When the desired code formatting style is different from the available options, the style can be customized using a configuration file. The aim of this project is to add NetBSD KNF support to clang-format and new styles to libFormat that support NetBSD’s style of coding. This would allow us to format NetBSD code by passing `-style=NetBSD` as an argument.

How to use clang-format

While using clang-format one can choose a style from the predefined styles or create a custom style by configuring specific style options.

Use the following command if you are using one of the predefined styles

Configuring style with clang-format

clang-format supports two ways to provide custom style options: directly specify style name in the -style= command line option or use -style=file and put style configuration in a .clang-format or _clang-format file in the project’s top directory.

Check Clang-Format Style Options to know how different Style Options works and how to use them.

When specifying configuration in the -style= option, the same configuration is applied for all input files. The format of the configuration is:

The .clang-format file uses YAML format. An easy way to get a valid .clang-format file containing all configuration options of a certain predefined style is:

After making required changes to the .clang-format file it can be used as a custom Style by:

Changes made to clang-format

The following changes were made to the clang-format as a part of adding NetBSD KNF:

New Style options added:

Modifications made to existing styles:

1. Modified SortIncludes  and IncludeCategories to support NetBSD like includes.
2. Modified SpacesBeforeTrailingComments to support block comments.

 The new NetBSD Style Configurations:

This is the final configurations for clang-format with modified changes to support NetBSD KNF.

    
        AlignTrailingComments: true
        AlwaysBreakAfterReturnType: All
        AlignConsecutiveMacros: true
        AlignConsecutiveLists: true
        BitFieldDeclarationsOnePerLine: true
        BreakBeforeBraces: Mozilla
        ColumnLimit: 80
        ContinuationIndentWidth: 4
        Cpp11BracedListStyle: false
        FixNamespaceComments: true
        IndentCaseLabels: false
        IndentWidth: 8
        IncludeBlocks: Regroup
        IncludeCategories:
         - Regex: '^<sys/param\.h>'
           Priority: 1
           SortPriority: 0
         - Regex: '^<sys/types\.h>'
           Priority: 1
           SortPriority: 1
         - Regex: '^<sys.*/'
           Priority: 1
           SortPriority: 2
         - Regex: '^<uvm/'
           Priority: 2
           SortPriority: 3
         - Regex: '^<machine/'
           Priority: 3
           SortPriority: 4
         - Regex: '^<dev/'
           Priority: 4
           SortPriority: 5
         - Regex: '^<net.*/'
           Priority: 5
           SortPriority: 6
         - Regex: '^<protocols/'
           Priority: 5
           SortPriority: 7
         - Regex: '^<(fs|miscfs|msdosfs|nfs|ufs)/'
           Priority: 6
           SortPriority: 8
         - Regex: '^<(x86|amd64|i386|xen)/'
           Priority: 7
           SortPriority: 8
         - Regex: '^<path'
           Priority: 9
           SortPriority: 11
         - Regex: '^<[^/].*\.h'
           Priority: 8
           SortPriority: 10
         - Regex: '^\".*\.h\"'
           Priority: 10
           SortPriority: 12   
        SortIncludes: true
        SpacesBeforeCpp11BracedList: true
        SpacesBeforeTrailingComments: 4
        TabWidth: 8
        UseTab: Always

    

Status of each Style Option

Styles Ready to Merge:

1.Modified SortIncludes and IncludeCategories:

     Patch: https://reviews.llvm.org/D64695

Styles needing revision

1.BitFieldDeclarationsOnePerLine:

    Patch: https://reviews.llvm.org/D63062

    Bugs: 1 

 2.SpacesBeforeTrailingComments supports Block Comments:

    Patch: https://reviews.llvm.org/D65648

    Remark:  I have to discuss more on the cases that Block comments are used but where the Spaces are not to be added before them.

WIP Style

1.AlignConsecutiveListElements:

    Commit: https://github.com/sh4nnu/clang/commit/4b4cd45a5f3d211008763f1c0235a22352faa81e

    Bugs: 1

About Styles

BitfieldDeclarationsOnePerLine:

    Patch: https://reviews.llvm.org/D63062 

    This style lines up BitField declarations on consecutive lines with correct indentation.

    
            unsigned int bas :3, hh : 4, jjj : 8;
            
            
            unsigned int baz:1,
                            fuz:5,
                zap:2;
    

            unsigned  int bas : 3, 
                        hh  : 4, 
                        jjj : 8;
            
            
            unsigned int baz:1,
                        fuz:5,
                        zap:2;

Bug: Indentation breaks in the presence of block comments in between.

Input:

        
                    unsigned  int bas : 3, /* foo */
                                  hh  : 4, /* bar */
                                  jjj : 8;
                    
        
        

Output:

        
                    unsigned  int bas : 3, /* foo */
                        hh  : 4, /* bar */
                        jjj : 8;

        
        

Modification for SortIncludes and IncludeCategories:

    Patch: https://reviews.llvm.org/D64695

    Status: Accepted, and ready to land.

Clang-format has a style option named SortIncludes which sorts the includes in alphabetical order.The IncludeCategories Style allows us to define a custom order for sorting the includes.

It supports POSIX extended regular expressions to assign Categories for includes.

The SortIncludes then sorts the #includes first according to increasing category number and then lexically within each category. When IncludeBlocks  is set to Regroup  merge multiple #includes blocks together and sort as one. Then split into groups based on category priority.

The problem arises when you want to define the order within each category, which is not supported. In this modification a new field named SortPriority is added, which is optional.

The #includes matching the regexs are sorted according to the values of SortPriority, and Regrouping after sorting is done according to the values of Priority. If SortPriority is not defined it is set to the value of Priority as a default value.

    
        IncludeCategories:
         - Regex: ‘<^c/’
           Priority: 1
           SortPriority: 0
        
         - Regex: ‘^<(a|b)/’
           Priority: 1
           SortPriority: 1
        
         - Regex: ‘^<(foo)/’
           Priority: 2
      
         - Regex: ‘.*’
           Priority: 3
           

    

    
            #include "exe.h"
            #include <a/dee.h>
            #include <foo/b.h>
            #include <a/bee.h>
            #include <exc.h>
            #include <b/dee.h>
            #include <c/abc.h>
            #include <foo/a.h>
    

    
            #include <c/abc.h>
            #include <a/bee.h>
            #include <a/dee.h>
            #include <b/dee.h>
            
            #include <foo/a.h>
            #include <foo/b.h>
            
            #include <exc.h>
            #include "exe.h"

    

As you can observe in the above example the #includes are grouped by different priority and were sorted by different priority. Introduction of this new patch doesn’t affect the old configurations as it will work as the same old SortIncludes if sortPriority is not defined.

Refer to Report 2 for detailed examples on this.

Modification for SpacesBeforeTrailingComments

    Patch: https://reviews.llvm.org/D64695

The SpacesBeforeTrailingComments is modified to support Block Comments which was used to support only line comments. The reason for this is block comments have different usage patterns and different exceptional cases. I have tried to exclude cases where some tests doesn’t support spaces before block comments. I have been discussing in the community for getting to know which cases should be included, and which to exclude.

Cases that were excluded due to failing tests:

• If it is a Preprocessor directive,
• If it is followed by a LeftParenthesis
• And if it is after a Template closer

AlignConsecutiveListElements

    Status: Work In Progress

This is a new style that aligns elements of consecutive lists in a nested list. The Style is still in work in progress. There are few cases that needed to be covered and fix few bugs.

    
            keys[] = {
                {"all", f_all, 0 },
                { "cbreak", f_cbreak, F_OFFOK },
                {"cols", f_columns, F_NEEDARG },
                { "columns", f_columns, F_NEEDARG },
             };
    

    
            keys[] = { { "all",      f_all,        0 },
                       { "cbreak",   _cbreak,      F_OFFOK },
                       { "cols",     f_columns,    F_NEEDARG }, 
                       { "columns",  f_columns,    F_NEEDARG },
                     };

    

This style option aligns list declarations that are nested inside a list, I would also like to extend this style to align individual single line list declarations that are consecutive.

The problem with this style is the case in which there can be different number of elements for each individual.

    
            keys[] =  { "all",        f_all,        0 };
            keys2[] = { "cbreak",     _cbreak,      F_OFFOK };
            keys3[] = { "cols",       f_columns,    F_NEEDARG,     7 };
            keys4[] = { "columns",    f_columns };
    

Future Work

Some Style Options that were introduced during this GSoC were made in order to meet all the cases in NetBSD KNF. So they may need some revisions with respect to other languages and coding styles that clang-format supports. I will continue working on this project even after the GSoC period on the style options that are yet to be merged and add new style options if necessary and will get the NetBSD Style merged with upstream which is the final deliverable for the project. I would like to take up the responsibility of maintaining the “NetBSD KNF” support to clang-format.

Summary

Even though officially the GSoC’19 coding period is over, I definitely look forward to keep contributing to this project. This summer has had me digging a lot into the code from CLANG and NetBSD for references for creating or modifying the Style Options. I am pretty much interested to work with NetBSD again, I like being in the community and I would like to improve my skills and learn more about Operating Systems by contributing to this organisation.

I would like to thank my mentors Michal and Christos for their constant support and patient guidance. A huge thanks to both the NetBSD and LLVM community who have been supportive and have helped me whenever I have had trouble. Finally a huge thanks to Google for providing me this opportunity.

This article was prepared by Surya P as a part of Google Summer of Code 2019

To begin with where we left off last time, we were able to fix the suse131 package with this commit.This commit adds the GPU-specific bits to the package. And with that we had direct rendering enabled and working.I tested it out with glxinfo and glxgears applications.

Testing

In order to make sure that applications did not break with this commit,I tried Libreoffice and to no surprise everything ran as expected without any hiccups.

Then I had to make a choice between porting steam and implementing compat_netbsd32 but since steam had lot of dependencies which needed to be resolved and since implementation of compat_netbsd32 had much more priority I started with the implementation of compat_netbsd32.

Implementing compat_netbsd32 DRM ioctls - The Setup

For the Setup I downloaded i386 sets from the official NetBSD site and extracted it in the /emul directory. I ran some arbitrary programs like cat and ls from the emulated netbsd32 directory to make sure everything ran perfectly without any problems. I then tried running the 32bit glxinfo and glxgears application and to no surprise it kept segfaulting. I ktraced the application and identified the DRM ioctl that needed to be implemented.

Implementing compat_netbsd32 DRM ioctls - The Code

There were several functions which were required for the complete working of the compat_netbsd32 DRM ioctl. We implemented each and every function and had the code compiled. We then made sure that the code compiled both as a module and as well as a non module option with which the kernel can be built.I initially tested the code with 32bit glxinfo and glxgears , and the program didn't segfault and ran as expected.

Implementing compat_netbsd32 DRM ioctls - Testing

In order to test the code I built a test application leveraging the api’s provided in libdrm. It is a very simple application which initializes the DRM connection, setup and draws a gradient on screen and exits. I initially ran it against the native amd64 architecture, but to my surprise the application didn't work as expected. After some hours of debugging I realized that there can be only one DRM master and X was already a master. After exiting the X session and running the application, everything ran perfectly for both amd64 as well as i386 architectures.

What is done

• The Drm Ioctls implementation of Netbsd has been tested and verified
• The suse131 package has patched and updated (committed)
• Compat_netbsd32 DRM ioctls has been implemented (Merged)
• Subsequently DRM ioctls for emulated 32bit linux as well
• Created a Test GUI Application for the code (yet to PR)

TODO

• Create an ATF for the code and merge it into the tree
• Read the code, look for bugs and clean it up
• Port Steam and make it available in NetBSD

Conclusion

Completing the tasks listed in the TODO is of highest priority and would be carried over even if it exceeds the GSOC time period.

Last but not the least I would like to thank my mentors @christos and @maya for helping me out and guiding me throughout the process and Google for providing me with such a wonderful opportunity to work with NetBSD community.

This article was prepared by Surya P as a part of Google Summer of Code 2019

To begin with where we left off last time, we were able to fix the suse131 package with this commit.This commit adds the GPU-specific bits to the package. And with that we had direct rendering enabled and working.I tested it out with glxinfo and glxgears applications.

Testing

In order to make sure that applications did not break with this commit,I tried Libreoffice and to no surprise everything ran as expected without any hiccups.

Then I had to make a choice between porting steam and implementing compat_netbsd32 but since steam had lot of dependencies which needed to be resolved and since implementation of compat_netbsd32 had much more priority I started with the implementation of compat_netbsd32.

Implementing compat_netbsd32 DRM ioctls - The Setup

For the Setup I downloaded i386 sets from the official NetBSD site and extracted it in the /emul directory. I ran some arbitrary programs like cat and ls from the emulated netbsd32 directory to make sure everything ran perfectly without any problems. I then tried running the 32bit glxinfo and glxgears application and to no surprise it kept segfaulting. I ktraced the application and identified the DRM ioctl that needed to be implemented.

Implementing compat_netbsd32 DRM ioctls - The Code

There were several functions which were required for the complete working of the compat_netbsd32 DRM ioctl. We implemented each and every function and had the code compiled. We then made sure that the code compiled both as a module and as well as a non module option with which the kernel can be built.I initially tested the code with 32bit glxinfo and glxgears , and the program didn't segfault and ran as expected.

Implementing compat_netbsd32 DRM ioctls - Testing

In order to test the code I built a test application leveraging the api’s provided in libdrm. It is a very simple application which initializes the DRM connection, setup and draws a gradient on screen and exits. I initially ran it against the native amd64 architecture, but to my surprise the application didn't work as expected. After some hours of debugging I realized that there can be only one DRM master and X was already a master. After exiting the X session and running the application, everything ran perfectly for both amd64 as well as i386 architectures.

What is done

• The Drm Ioctls implementation of Netbsd has been tested and verified
• The suse131 package has patched and updated (committed)
• Compat_netbsd32 DRM ioctls has been implemented (Merged)
• Subsequently DRM ioctls for emulated 32bit linux as well
• Created a Test GUI Application for the code (yet to PR)

TODO

• Create an ATF for the code and merge it into the tree
• Read the code, look for bugs and clean it up
• Port Steam and make it available in NetBSD

Conclusion

Completing the tasks listed in the TODO is of highest priority and would be carried over even if it exceeds the GSOC time period.

Last but not the least I would like to thank my mentors @christos and @maya for helping me out and guiding me throughout the process and Google for providing me with such a wonderful opportunity to work with NetBSD community.

This is the third report summarising the work done in the third coding period for the GSoC project of Adapting TriforceAFL for NetBSD kernel syscall fuzzing.
Please also go through the first and second report.

This post also outlines the work done throughout the duration of GSoC, describes the implications of the same and future improvements to come.

Current State

As of now TriforceAFL has been made available in the form of a pkgsrc package (wip/triforceafl). This package allows you to essentially fuzz anything in QEMU’s full system emulation mode using AFL. TriforceNetBSDSyscallFuzzer is built on top of TriforceAFL, specifically to fuzz the NetBSD kernel syscalls. It has also now been made available as wip/triforcenetbsdsyscallfuzzer.
Several minor issues found in the above two packages have now been resolved, and the project restructured.

Issues found include:

• The input generators would also test the the generated inputs, causing several syscalls to be executed which messed up permissions of several other directories and lead to other unwanted consequences. Testing of syscalls has now been disabled as the working ones are specified.
• Directory specified for the BIOS, VGA BIOS and keymaps for QEMU in the runFuzz script was incorrect. This was because wip/triforceafl did not install the specified directory. The package has now been patched to install the pc-bios directory.
• The project was structured in a manner where the host was Linux and the target was NetBSD. Since that is no longer the case and the fuzzer is meant to be run on a NetBSD machine, the project was restructured so that everything is now in one directory and there is no longer a need to move files around.

The packages should now work as intended by following the instructions outlined in the README.

The fuzzer was able to detect a few bugs in the last coding period, details can be found in the last report. During this coding period, the Fuzzer was able to detect 79 unique crashes in a period of 2 weeks running on a single machine. The kernel was built with DEBUG + LOCKDEBUG + DIAGNOSTIC. Work is underway to analyse, report and fix the new bugs and to make the process faster.

With an initial analysis on the outputs on the basis of the syscall that lead to the crash, 6 of the above crashes were unique bugs, the rest were duplicates or slight variants, of which 3 have been previously reported.
Here are the backtraces of the new bugs found (The reproducers were run with kUBSan Enabled):

BUG1:

[ 110.4035826] panic: cv_enter,172: uninitialized lock (lock=0xffffe3c1b9
fc0c50, from=ffffffff81a436e9)
[ 110.4035826] cpu0: Begin traceback...
[ 110.4035826] vpanic() at netbsd:vpanic+0x1fd
[ 110.4035826] snprintf() at netbsd:snprintf
[ 110.4035826] lockdebug_locked() at netbsd:lockdebug_locked+0x45e
[ 110.4035826] cv_timedwait_sig() at netbsd:cv_timedwait_sig+0xe7
[ 110.4035826] lfs_segwait() at netbsd:lfs_segwait+0x6e
[ 110.4035826] sys___lfs_segwait50() at netbsd:sys___lfs_segwait50+0xe2
[ 110.4035826] sys___syscall() at netbsd:sys___syscall+0x121
[ 110.4035826] syscall() at netbsd:syscall+0x1a5
[ 110.4035826] --- syscall (number 198) ---
[ 110.4035826] 40261a:
[ 110.4035826] cpu0: End traceback...
[ 110.4035826] fatal breakpoint trap in supervisor mode
[ 110.4035826] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x282 cr2
 0x73f454b70000 ilevel 0x8 rsp 0xffff8a0068390d70
[ 110.4035826] curlwp 0xffffe3c1efc556a0 pid 709.1 lowest kstack 0xffff8a006838d
2c0
Stopped in pid 709.1 (driver) at        netbsd:breakpoint+0x5:  leave
db{0}> bt
breakpoint() at netbsd:breakpoint+0x5
vpanic() at netbsd:vpanic+0x1fd
snprintf() at netbsd:snprintf
lockdebug_locked() at netbsd:lockdebug_locked+0x45e
cv_timedwait_sig() at netbsd:cv_timedwait_sig+0xe7
lfs_segwait() at netbsd:lfs_segwait+0x6e
sys___lfs_segwait50() at netbsd:sys___lfs_segwait50+0xe2
sys___syscall() at netbsd:sys___syscall+0x121
syscall() at netbsd:syscall+0x1a5
--- syscall (number 198) ---
40261a:

BUG2:

[ 161.4877660] panic: LOCKDEBUG: Mutex error: rw_vector_enter,296: spin lock hel
d
[ 161.4877660] cpu0: Begin traceback...
[ 161.4877660] vpanic() at netbsd:vpanic+0x1fd
[ 161.4877660] snprintf() at netbsd:snprintf
[ 161.4877660] lockdebug_abort1() at netbsd:lockdebug_abort1+0x115
[ 161.4877660] rw_enter() at netbsd:rw_enter+0x645
[ 161.4877660] uvm_fault_internal() at netbsd:uvm_fault_internal+0x1c5
[ 161.4877660] trap() at netbsd:trap+0xa71
[ 161.4877660] --- trap (number 6) ---
[ 161.4877660] config_devalloc() at netbsd:config_devalloc+0x644
[ 161.4877660] config_attach_pseudo() at netbsd:config_attach_pseudo+0x1c
[ 161.4877660] vndopen() at netbsd:vndopen+0x1f3
[ 161.4877660] cdev_open() at netbsd:cdev_open+0x12d
[ 161.4877660] spec_open() at netbsd:spec_open+0x2d0
[ 161.4877660] VOP_OPEN() at netbsd:VOP_OPEN+0xba
[ 161.4877660] vn_open() at netbsd:vn_open+0x434
[ 161.4877660] sys_ktrace() at netbsd:sys_ktrace+0x1ec
[ 161.4877660] sys___syscall() at netbsd:sys___syscall+0x121
[ 161.4877660] syscall() at netbsd:syscall+0x1a5
[ 161.4877660] --- syscall (number 198) ---
[ 161.4877660] 40261a:
[ 161.4877660] cpu0: End traceback...
[ 161.4877660] fatal breakpoint trap in supervisor mode
[ 161.4877660] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x286 cr2
 0xfffffffffffff800 ilevel 0x8 rsp 0xffff9c80683cd4f0
[ 161.4877660] curlwp 0xfffffcbcda7d36a0 pid 41.1 lowest kstack 0xffff9c80683ca2
c0
db{0}> bt
breakpoint() at netbsd:breakpoint+0x5
vpanic() at netbsd:vpanic+0x1fd
snprintf() at netbsd:snprintf
lockdebug_abort1() at netbsd:lockdebug_abort1+0x115
rw_enter() at netbsd:rw_enter+0x645
uvm_fault_internal() at netbsd:uvm_fault_internal+0x1c5
trap() at netbsd:trap+0xa71
--- trap (number 6) ---
config_devalloc() at netbsd:config_devalloc+0x644
config_attach_pseudo() at netbsd:config_attach_pseudo+0x1c
vndopen() at netbsd:vndopen+0x1f3
cdev_open() at netbsd:cdev_open+0x12d
spec_open() at netbsd:spec_open+0x2d0
VOP_OPEN() at netbsd:VOP_OPEN+0xba
vn_open() at netbsd:vn_open+0x434
sys_ktrace() at netbsd:sys_ktrace+0x1ec
sys___syscall() at netbsd:sys___syscall+0x121
syscall() at netbsd:syscall+0x1a5
--- syscall (number 198) ---
40261a:

BUG3:

[ 350.9942146] UBSan: Undefined Behavior in /home/ubuntu/triforce/kernel/
src/sys/kern/kern_ktrace.c:1398:2, member access within misaligned address 0x2b0
000002a for type 'struct ktr_desc' which requires 8 byte alignment
[ 351.0025346] uvm_fault(0xffffffff85b73100, 0x2b00000000, 1) -> e
[ 351.0025346] fatal page fault in supervisor mode
[ 351.0025346] trap type 6 code 0 rip 0xffffffff81b9dbf9 cs 0x8 rflags 0x286 cr2
 0x2b00000032 ilevel 0 rsp 0xffff8780684d7fb0
[ 351.0025346] curlwp 0xffffa992128116e0 pid 0.54 lowest kstack 0xffff8780684d42
c0
kernel: page fault trap, code=0
Stopped in pid 0.54 (system) at netbsd:ktrace_thread+0x1fd:     cmpq    %rbx,8(%
r12)
db{0}> bt
ktrace_thread() at netbsd:ktrace_thread+0x1fd

Reproducing Crashes

Right now the best way to reproduce a bug detected is to use the Fuzzer’s driver program itself:

./driver -tv < crash_file

The crash_file can be found in the outputs directory and is a custom file format made for the driver.
Memory allocation and Socket Creation remain to be added to the reproducer generator(genRepro) highlighted in the previous post and will be prioritised in the future.

Implications

Considering that we have a working fuzzer now, it is a good time to analyse how effective TriforceAFL is compared to other fuzzers.

Recently Syzkaller has been really effective in finding bugs in NetBSD. As shown in the below diagrams, both TriforceAFL and Syzkaller create multiple instances of the system to be fuzzed, gather coverage data, mutate input accordingly and continue fuzzing, but there are several differences in the way they work.

TriforceAFL



Syzkaller

Key differences between the two include:

• KCOV
  Syzkaller relies on the KCOV module for coverage data in NetBSD whereas TriforceAFL gets coverage information from its modified version of QEMU.
• VMs
  Syzkaller creates multiple VM’s and manages them with syz-manager. Whereas TriforceAFL simply forks the VM for each testcase.
• Communication
  Syzkaller uses RPC and ssh to communicate with the VMs whereas TriforceAFL uses a custom hypercall.
• Hardware Acceleration
  Syzkaller can use hardware acceleration to run the VMs at native speeds, whereas TriforceAFL can only utilize QEMU’s full system emulation mode, as it relies on it for coverage data.

These differences lead to very different results. To get a perspective, here are some stats from syzkaller, which can be found on the syzbot dashboard.

Comparatively in 1st Weekend of Fuzzing:

• Compared to syzkaller, the number of bugs found by TriforceAFL in the first few days were significantly less, but nevertheless TriforceAFL was able to find variants of bugs found by syzkaller and 1 which was different with simpler reproducers.
• Going by the stats provided by Syzbot and AFL, Syzkaller does ~80 execs / min whereas TriforceAFL can do ~1500 execs / min on average, Although this statistic is without any of the sanitizers enabled for TriforceAFL and kASan enabled for Syzkaller.
• TriforceAFL has an advantage when it comes to the fact that it does not rely on KCOV for coverage data. This means it can get coverage data for fuzzing other interfaces easily too. This will be beneficial when we move onto Network and USB Fuzzing. Issues have been found with KCOV where in certain cases the fuzzer lost track of the kernel trace, especially in networking where after enqueuing a networking packet the fuzzer lost track as the packet was then handled by some other thread. Coverage data gathered using TriforceAFL will not be sensitive to this, considering that noise from the kernel is handled in some way to an extent.
• Efforts were made in the original design of TriforceAFL to make the traces as deterministic as possible at a basic block level. To quote from this article: Sometimes QEMU starts executing a basic block and then gets interrupted. It may then re-execute the block from the start or translate a portion of the block that wasn't yet executed and execute that. This introduced non-determinism. To reduce the non-determinism, cpu-exec.c was altered to disable QEMU's "chaining" feature, and moved AFL's tracing feature to cpu_tb_exec to trace a basic block only after it has been executed to completion. Although nothing has been specifically done to reduce noise from the kernel, such as disabling coverage during interrupts.
• TriforceAFL runs 1 fuzzing process per instance, whereas Syzkaller can run upto 32 fuzzing processes. But multiple fuzzing instance can be created with TriforceAFL as detailed in this document to take advantage of parallelization.
• Maxime Villard was also able to rework TriforceNetBSDSyscallFuzzer to now also support compat_netbsd32 kernel. This work will be integrated as soon as possible.
• On the other hand, Syzkaller has syzbot which can be thought of as a 24/7 fuzzing service. TriforceAFL does not have a such a service or a dedicated server for fuzzing.
  A service like this will surely be advantageous in the future, but it is still a bit too early to set up one. To truly efficiently utilize such a service, it would be better to improve TriforceAFL in all ways possible first.

Targets to be met before a 24/7 fuzzing service is setup, include but are not limited to:

• Automatic initial Analysis & Report Generation
  Right now, There is nothing that can automatically perform an initial analysis to detect truly unique crashes and prepare reports with backtraces, this will be required and very helpful.
• Better Reproducers
  As mentioned before, the generated reproducers do not take care of Memory Allocation and Socket Creation, etc. These need to be included, if a good C reproducer is expected.
• Parallel Fuzzing and Management
  There is no central interface that summarises the statistics from all fuzzing instances, and manages such instances in case of anomalies/errors. Something like this would be needed for a service that will be run for longer periods of time without supervision.
• Updated AFL and QEMU versions
  Updated AFL and QEMU might significantly increase executions per second and lead to better mutation of inputs and also decrease the memory requirements.
• Fuzzing more Interfaces
  Right now we are only fuzzing syscalls, Network and USB Layers are great future prospects, at least prototype versions can be added in the near future.

Future Work

Although GSoC will be officially ending, I am looking forward to continuing the development of TriforceAFL, adding features and making it more effective.
Some improvements that can be expected include:

• Fuzzing different interfaces - Network Fuzzing, USB Fuzzing
• Updating AFL and QEMU versions
• Better Reproducers
• Syzbot like service

A new repo has been created at https://github.com/NetBSD/triforce4netbsd. Collaborative work in the future will be updated here.

Conclusion

TriforceAFL has been successfully adapted for NetBSD and all of the original goals of the GSoC proposal have been met, but the work is far from complete. Work done so far shows great potential, and incremental updates will surely make TriforceAFL a great fuzzer. I am looking forward to continuing the work and making sure that this is the case.
GSoC has been an amazing learning experience! I would like to thank Maxime Villard for taking an active interest in TriforceAFL and for assisting in testing and development. I would like to thank my mentor Kamil Rytarowski for being the guiding force throughout the project and for assisting me whenever I needed help. And finally I would like to thank Google for giving me this wonderful opportunity.

This is the third report summarising the work done in the third coding period for the GSoC project of Adapting TriforceAFL for NetBSD kernel syscall fuzzing.
Please also go through the first and second report.

This post also outlines the work done throughout the duration of GSoC, describes the implications of the same and future improvements to come.

Current State

As of now TriforceAFL has been made available in the form of a pkgsrc package (wip/triforceafl). This package allows you to essentially fuzz anything in QEMU’s full system emulation mode using AFL. TriforceNetBSDSyscallFuzzer is built on top of TriforceAFL, specifically to fuzz the NetBSD kernel syscalls. It has also now been made available as wip/triforcenetbsdsyscallfuzzer.
Several minor issues found in the above two packages have now been resolved, and the project restructured.

Issues found include:

• The input generators would also test the the generated inputs, causing several syscalls to be executed which messed up permissions of several other directories and lead to other unwanted consequences. Testing of syscalls has now been disabled as the working ones are specified.
• Directory specified for the BIOS, VGA BIOS and keymaps for QEMU in the runFuzz script was incorrect. This was because wip/triforceafl did not install the specified directory. The package has now been patched to install the pc-bios directory.
• The project was structured in a manner where the host was Linux and the target was NetBSD. Since that is no longer the case and the fuzzer is meant to be run on a NetBSD machine, the project was restructured so that everything is now in one directory and there is no longer a need to move files around.

The packages should now work as intended by following the instructions outlined in the README.

The fuzzer was able to detect a few bugs in the last coding period, details can be found in the last report. During this coding period, the Fuzzer was able to detect 79 unique crashes in a period of 2 weeks running on a single machine. The kernel was built with DEBUG + LOCKDEBUG + DIAGNOSTIC. Work is underway to analyse, report and fix the new bugs and to make the process faster.

With an initial analysis on the outputs on the basis of the syscall that lead to the crash, 6 of the above crashes were unique bugs, the rest were duplicates or slight variants, of which 3 have been previously reported.
Here are the backtraces of the new bugs found (The reproducers were run with kUBSan Enabled):

BUG1:

[ 110.4035826] panic: cv_enter,172: uninitialized lock (lock=0xffffe3c1b9
fc0c50, from=ffffffff81a436e9)
[ 110.4035826] cpu0: Begin traceback...
[ 110.4035826] vpanic() at netbsd:vpanic+0x1fd
[ 110.4035826] snprintf() at netbsd:snprintf
[ 110.4035826] lockdebug_locked() at netbsd:lockdebug_locked+0x45e
[ 110.4035826] cv_timedwait_sig() at netbsd:cv_timedwait_sig+0xe7
[ 110.4035826] lfs_segwait() at netbsd:lfs_segwait+0x6e
[ 110.4035826] sys___lfs_segwait50() at netbsd:sys___lfs_segwait50+0xe2
[ 110.4035826] sys___syscall() at netbsd:sys___syscall+0x121
[ 110.4035826] syscall() at netbsd:syscall+0x1a5
[ 110.4035826] --- syscall (number 198) ---
[ 110.4035826] 40261a:
[ 110.4035826] cpu0: End traceback...
[ 110.4035826] fatal breakpoint trap in supervisor mode
[ 110.4035826] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x282 cr2
 0x73f454b70000 ilevel 0x8 rsp 0xffff8a0068390d70
[ 110.4035826] curlwp 0xffffe3c1efc556a0 pid 709.1 lowest kstack 0xffff8a006838d
2c0
Stopped in pid 709.1 (driver) at        netbsd:breakpoint+0x5:  leave
db{0}> bt
breakpoint() at netbsd:breakpoint+0x5
vpanic() at netbsd:vpanic+0x1fd
snprintf() at netbsd:snprintf
lockdebug_locked() at netbsd:lockdebug_locked+0x45e
cv_timedwait_sig() at netbsd:cv_timedwait_sig+0xe7
lfs_segwait() at netbsd:lfs_segwait+0x6e
sys___lfs_segwait50() at netbsd:sys___lfs_segwait50+0xe2
sys___syscall() at netbsd:sys___syscall+0x121
syscall() at netbsd:syscall+0x1a5
--- syscall (number 198) ---
40261a:

BUG2:

[ 161.4877660] panic: LOCKDEBUG: Mutex error: rw_vector_enter,296: spin lock hel
d
[ 161.4877660] cpu0: Begin traceback...
[ 161.4877660] vpanic() at netbsd:vpanic+0x1fd
[ 161.4877660] snprintf() at netbsd:snprintf
[ 161.4877660] lockdebug_abort1() at netbsd:lockdebug_abort1+0x115
[ 161.4877660] rw_enter() at netbsd:rw_enter+0x645
[ 161.4877660] uvm_fault_internal() at netbsd:uvm_fault_internal+0x1c5
[ 161.4877660] trap() at netbsd:trap+0xa71
[ 161.4877660] --- trap (number 6) ---
[ 161.4877660] config_devalloc() at netbsd:config_devalloc+0x644
[ 161.4877660] config_attach_pseudo() at netbsd:config_attach_pseudo+0x1c
[ 161.4877660] vndopen() at netbsd:vndopen+0x1f3
[ 161.4877660] cdev_open() at netbsd:cdev_open+0x12d
[ 161.4877660] spec_open() at netbsd:spec_open+0x2d0
[ 161.4877660] VOP_OPEN() at netbsd:VOP_OPEN+0xba
[ 161.4877660] vn_open() at netbsd:vn_open+0x434
[ 161.4877660] sys_ktrace() at netbsd:sys_ktrace+0x1ec
[ 161.4877660] sys___syscall() at netbsd:sys___syscall+0x121
[ 161.4877660] syscall() at netbsd:syscall+0x1a5
[ 161.4877660] --- syscall (number 198) ---
[ 161.4877660] 40261a:
[ 161.4877660] cpu0: End traceback...
[ 161.4877660] fatal breakpoint trap in supervisor mode
[ 161.4877660] trap type 1 code 0 rip 0xffffffff8021ddf5 cs 0x8 rflags 0x286 cr2
 0xfffffffffffff800 ilevel 0x8 rsp 0xffff9c80683cd4f0
[ 161.4877660] curlwp 0xfffffcbcda7d36a0 pid 41.1 lowest kstack 0xffff9c80683ca2
c0
db{0}> bt
breakpoint() at netbsd:breakpoint+0x5
vpanic() at netbsd:vpanic+0x1fd
snprintf() at netbsd:snprintf
lockdebug_abort1() at netbsd:lockdebug_abort1+0x115
rw_enter() at netbsd:rw_enter+0x645
uvm_fault_internal() at netbsd:uvm_fault_internal+0x1c5
trap() at netbsd:trap+0xa71
--- trap (number 6) ---
config_devalloc() at netbsd:config_devalloc+0x644
config_attach_pseudo() at netbsd:config_attach_pseudo+0x1c
vndopen() at netbsd:vndopen+0x1f3
cdev_open() at netbsd:cdev_open+0x12d
spec_open() at netbsd:spec_open+0x2d0
VOP_OPEN() at netbsd:VOP_OPEN+0xba
vn_open() at netbsd:vn_open+0x434
sys_ktrace() at netbsd:sys_ktrace+0x1ec
sys___syscall() at netbsd:sys___syscall+0x121
syscall() at netbsd:syscall+0x1a5
--- syscall (number 198) ---
40261a:

BUG3:

[ 350.9942146] UBSan: Undefined Behavior in /home/ubuntu/triforce/kernel/
src/sys/kern/kern_ktrace.c:1398:2, member access within misaligned address 0x2b0
000002a for type 'struct ktr_desc' which requires 8 byte alignment
[ 351.0025346] uvm_fault(0xffffffff85b73100, 0x2b00000000, 1) -> e
[ 351.0025346] fatal page fault in supervisor mode
[ 351.0025346] trap type 6 code 0 rip 0xffffffff81b9dbf9 cs 0x8 rflags 0x286 cr2
 0x2b00000032 ilevel 0 rsp 0xffff8780684d7fb0
[ 351.0025346] curlwp 0xffffa992128116e0 pid 0.54 lowest kstack 0xffff8780684d42
c0
kernel: page fault trap, code=0
Stopped in pid 0.54 (system) at netbsd:ktrace_thread+0x1fd:     cmpq    %rbx,8(%
r12)
db{0}> bt
ktrace_thread() at netbsd:ktrace_thread+0x1fd

Reproducing Crashes

Right now the best way to reproduce a bug detected is to use the Fuzzer’s driver program itself:

./driver -tv < crash_file

The crash_file can be found in the outputs directory and is a custom file format made for the driver.
Memory allocation and Socket Creation remain to be added to the reproducer generator(genRepro) highlighted in the previous post and will be prioritised in the future.

Implications

Considering that we have a working fuzzer now, it is a good time to analyse how effective TriforceAFL is compared to other fuzzers.

Recently Syzkaller has been really effective in finding bugs in NetBSD. As shown in the below diagrams, both TriforceAFL and Syzkaller create multiple instances of the system to be fuzzed, gather coverage data, mutate input accordingly and continue fuzzing, but there are several differences in the way they work.

TriforceAFL



Syzkaller

Key differences between the two include:

• KCOV
  Syzkaller relies on the KCOV module for coverage data in NetBSD whereas TriforceAFL gets coverage information from its modified version of QEMU.
• VMs
  Syzkaller creates multiple VM’s and manages them with syz-manager. Whereas TriforceAFL simply forks the VM for each testcase.
• Communication
  Syzkaller uses RPC and ssh to communicate with the VMs whereas TriforceAFL uses a custom hypercall.
• Hardware Acceleration
  Syzkaller can use hardware acceleration to run the VMs at native speeds, whereas TriforceAFL can only utilize QEMU’s full system emulation mode, as it relies on it for coverage data.

These differences lead to very different results. To get a perspective, here are some stats from syzkaller, which can be found on the syzbot dashboard.

Comparatively in 1st Weekend of Fuzzing:

• Compared to syzkaller, the number of bugs found by TriforceAFL in the first few days were significantly less, but nevertheless TriforceAFL was able to find variants of bugs found by syzkaller and 1 which was different with simpler reproducers.
• Going by the stats provided by Syzbot and AFL, Syzkaller does ~80 execs / min whereas TriforceAFL can do ~1500 execs / min on average, Although this statistic is without any of the sanitizers enabled for TriforceAFL and kASan enabled for Syzkaller.
• TriforceAFL has an advantage when it comes to the fact that it does not rely on KCOV for coverage data. This means it can get coverage data for fuzzing other interfaces easily too. This will be beneficial when we move onto Network and USB Fuzzing. Issues have been found with KCOV where in certain cases the fuzzer lost track of the kernel trace, especially in networking where after enqueuing a networking packet the fuzzer lost track as the packet was then handled by some other thread. Coverage data gathered using TriforceAFL will not be sensitive to this, considering that noise from the kernel is handled in some way to an extent.
• Efforts were made in the original design of TriforceAFL to make the traces as deterministic as possible at a basic block level. To quote from this article: Sometimes QEMU starts executing a basic block and then gets interrupted. It may then re-execute the block from the start or translate a portion of the block that wasn't yet executed and execute that. This introduced non-determinism. To reduce the non-determinism, cpu-exec.c was altered to disable QEMU's "chaining" feature, and moved AFL's tracing feature to cpu_tb_exec to trace a basic block only after it has been executed to completion. Although nothing has been specifically done to reduce noise from the kernel, such as disabling coverage during interrupts.
• TriforceAFL runs 1 fuzzing process per instance, whereas Syzkaller can run upto 32 fuzzing processes. But multiple fuzzing instance can be created with TriforceAFL as detailed in this document to take advantage of parallelization.
• Maxime Villard was also able to rework TriforceNetBSDSyscallFuzzer to now also support compat_netbsd32 kernel. This work will be integrated as soon as possible.
• On the other hand, Syzkaller has syzbot which can be thought of as a 24/7 fuzzing service. TriforceAFL does not have a such a service or a dedicated server for fuzzing.
  A service like this will surely be advantageous in the future, but it is still a bit too early to set up one. To truly efficiently utilize such a service, it would be better to improve TriforceAFL in all ways possible first.

Targets to be met before a 24/7 fuzzing service is setup, include but are not limited to:

• Automatic initial Analysis & Report Generation
  Right now, There is nothing that can automatically perform an initial analysis to detect truly unique crashes and prepare reports with backtraces, this will be required and very helpful.
• Better Reproducers
  As mentioned before, the generated reproducers do not take care of Memory Allocation and Socket Creation, etc. These need to be included, if a good C reproducer is expected.
• Parallel Fuzzing and Management
  There is no central interface that summarises the statistics from all fuzzing instances, and manages such instances in case of anomalies/errors. Something like this would be needed for a service that will be run for longer periods of time without supervision.
• Updated AFL and QEMU versions
  Updated AFL and QEMU might significantly increase executions per second and lead to better mutation of inputs and also decrease the memory requirements.
• Fuzzing more Interfaces
  Right now we are only fuzzing syscalls, Network and USB Layers are great future prospects, at least prototype versions can be added in the near future.

Future Work

Although GSoC will be officially ending, I am looking forward to continuing the development of TriforceAFL, adding features and making it more effective.
Some improvements that can be expected include:

• Fuzzing different interfaces - Network Fuzzing, USB Fuzzing
• Updating AFL and QEMU versions
• Better Reproducers
• Syzbot like service

A new repo has been created at https://github.com/NetBSD/triforce4netbsd. Collaborative work in the future will be updated here.

Conclusion

TriforceAFL has been successfully adapted for NetBSD and all of the original goals of the GSoC proposal have been met, but the work is far from complete. Work done so far shows great potential, and incremental updates will surely make TriforceAFL a great fuzzer. I am looking forward to continuing the work and making sure that this is the case.
GSoC has been an amazing learning experience! I would like to thank Maxime Villard for taking an active interest in TriforceAFL and for assisting in testing and development. I would like to thank my mentor Kamil Rytarowski for being the guiding force throughout the project and for assisting me whenever I needed help. And finally I would like to thank Google for giving me this wonderful opportunity.

Prepared by Siddharth Muralee(@R3x) as a part of Google Summer of Code’19

As a part of Google Summer of Code’19, I am working on improving the support for Syzkaller kernel fuzzer. Syzkaller is an unsupervised coverage-guided kernel fuzzer, that supports a variety of operating systems including NetBSD.

You can take a look through the first report to see the initial changes that we made and you can look at the second report to read about the initial support we added for fuzzing the network stack.

This report details the work done during the final coding period where the target was to improve the support for fuzzing the filesystem stack.

Filesystem fuzzing is a relatively less explored area. Syzkaller itself only has filesystem fuzzing support for Linux.

Analysis of the existing Linux setup

Filesystems are more complex fuzzing target than standalone system calls. To fuzz Filesystems we do have a standard operation like mount which comes with system call vector and an additional binary image of the filesystem itself. While normal syscalls generally have a size of a few bytes, sizes of real world Filesystem images is in order of Gigabytes or larger, however for fuzzing minimal size can be used which is in order of KB-MB. Since syzkaller uses a technique called as mutational fuzzing - where it mutates random parts of the input (according to specified guidelines), having a large input size causes delay due to higher I/O time.

Syzkaller deals with large images by disassembling them to non-zero chunks of the filesystem image. Syzkaller extracts the non-zero chunks and their offsets and stores it as separate segments and just before execution it writes all the chunks into the corresponding offsets - generating back the new/modified image.

Porting it to NetBSD

As an initial step towards filesystem fuzzing we decided to port the existing Linux approach of creating random segments to NetBSD. There are a few differences between the mounting process in both the operating systems - the most significant of them being the difference in the arguments to mount(2).

Linux:

int mount(const char *source, const char *target, const char *filesystemtype, unsigned long mountflags, const void *data);

The data argument is interpreted by the different filesystems. Typically it is a string of comma-separated options understood by this filesystem. mount(8) - shows possible arguments for each of the filesystem.

possible options for xfs filesystem in linux :

    wsync, noalign, swalloc, nouuid, mtpt, grpid, nogrpid, bsdgroups, 
    sysvgroups,norecovery, inode64, inode32, ikeep, noikeep,
    largeio, nolargeio, attr2, noattr2, filestreams, quota,
    noquota, lazytime, nolazytime, usrquota, grpquota, prjquota,
    uquota, gquota, pquota, uqnoenforce, gqnoenforce, pqnoenforce,
    qnoenforce, discard, nodiscard, dax, barrier, nobarrier, logbufs,
    biosize, sunit, swidth, logbsize, allocsize, logdev, rtdev

NetBSD:

Int mount(const char *type, const char *dir, int flags, void *data, size_t data_len);

The argument data describes the file system object to be mounted, and is data_len bytes long. data is a pointer to a structure that contains the type specific arguments to mount.

For FFS (one of the most common filesystems for NetBSD) - the arguments look like :

struct ufs_args {
        char      *fspec;   /* block special file to mount */
};

Currently, we have a pseudo syscall syz_mount_image which does the job of writing the mutated chunks of the filesystem into a file based on their offsets and later configuring the loop device using vndconfig(8) and mounting the filesystem image using mount(8).

Analysis of the current approach

One way to create mountable filesystems is to convert an existing filesystem image into a syzkaller pseudo grammar representation and then add it to the corpus so that syzkaller uses it for mutation and we have a proper image.

Some of the noted issues with syzkaller approach (as noted in "Fuzzing File Systems via Two-Dimensional Input Space Exploration) :

Steps Forward

We also spent some time researching possible options to solve the existing issues and developing an approach that would give us better results.

Image mutator approach

One possible way forward is to actually use a seed image (a working filesystem image) and write a mutator which would be aware of all the metadata in the image. The mutator should be also be able to recreate metadata components such as the checksum so that the image is mountable.

An existing implementation of such a mutator is JANUS which is a filesystem mutator written for Linux with inspiration from fsck.

Grammar based approach

Syzkaller uses a pseudo-formal grammar for representing arguments to syscalls. This grammar can also be modified to actually be able to properly generate filesystem images.

Writing grammar to represent a filesystem image is quite a daunting task and we are not yet sure if it is possible but it is the approach that we have planned to take up as of now.

Proper documentation detailing the structure of a filesystem image is rather scarce which has led me to actually go through filesystem code to figure out the type, uses and limits of a certain filesystem image. This data then has to be converted to syzkaller representation to be used for fuzzing.

One advantage of writing a grammar that would be able to generate mountable images is that we would be able to get more coverage than fuzzing with a seed image, since we are also creating new images instead of just mutating the same image.

I am currently working on learning the internals of FFS and trying to write a grammar definition which can properly generate filesystem images.

Miscellaneous Work

Meanwhile, I have also been working in parallel on improving the existing state of Syzkaller.

Add kernel compiled with KUBSAN for fuzzing

So far we only used a kernel compiled with KCOV and KASAN for fuzzing with syzkaller. We also decided to add support for syzkaller building a kernel with KUBSAN and KCOV. This would help us have an another dimension in the fuzzing process.

This required some changes in the build config. We had to remove the hardcoded kernel config and add support for building a kernel with a config passed to the fuzzer. This move would also help us to easily add support for upcoming sanitizers such as KMSAN.

Improve syscall descriptions

Improving system call descriptions is a constant ongoing work - I recently added support for fuzzing syscalls such as mount, fork and posix_spawn.

We are also planning to add support for fuzzing device drivers soon.

Relevant Links

Summary

We have managed to meet most of the goals that we had planned for the GSoC project. Overall, I have had a wonderful summer with the NetBSD foundation and I look forward to working with them to complete the project.

Last but not least, I want to thank my mentors, @kamil and @cryo for their useful suggestions and guidance. I also thank Maciej for his insight and guidance which was very fundamental during the course of the project. I would also like to thank Dmitry Vyukov, Google for helping with any issues faced with regard to Syzkaller. Finally, thanks to Google to give me a good chance to work with NetBSD community.

Prepared by Siddharth Muralee(@R3x) as a part of Google Summer of Code’19

As a part of Google Summer of Code’19, I am working on improving the support for Syzkaller kernel fuzzer. Syzkaller is an unsupervised coverage-guided kernel fuzzer, that supports a variety of operating systems including NetBSD.

You can take a look through the first report to see the initial changes that we made and you can look at the second report to read about the initial support we added for fuzzing the network stack.

This report details the work done during the final coding period where the target was to improve the support for fuzzing the filesystem stack.

Filesystem fuzzing is a relatively less explored area. Syzkaller itself only has filesystem fuzzing support for Linux.

Analysis of the existing Linux setup

Filesystems are more complex fuzzing target than standalone system calls. To fuzz Filesystems we do have a standard operation like mount which comes with system call vector and an additional binary image of the filesystem itself. While normal syscalls generally have a size of a few bytes, sizes of real world Filesystem images is in order of Gigabytes or larger, however for fuzzing minimal size can be used which is in order of KB-MB. Since syzkaller uses a technique called as mutational fuzzing - where it mutates random parts of the input (according to specified guidelines), having a large input size causes delay due to higher I/O time.

Syzkaller deals with large images by disassembling them to non-zero chunks of the filesystem image. Syzkaller extracts the non-zero chunks and their offsets and stores it as separate segments and just before execution it writes all the chunks into the corresponding offsets - generating back the new/modified image.

Porting it to NetBSD

As an initial step towards filesystem fuzzing we decided to port the existing Linux approach of creating random segments to NetBSD. There are a few differences between the mounting process in both the operating systems - the most significant of them being the difference in the arguments to mount(2).

Linux:

int mount(const char *source, const char *target, const char *filesystemtype, unsigned long mountflags, const void *data);

The data argument is interpreted by the different filesystems. Typically it is a string of comma-separated options understood by this filesystem. mount(8) - shows possible arguments for each of the filesystem.

possible options for xfs filesystem in linux :

    wsync, noalign, swalloc, nouuid, mtpt, grpid, nogrpid, bsdgroups, 
    sysvgroups,norecovery, inode64, inode32, ikeep, noikeep,
    largeio, nolargeio, attr2, noattr2, filestreams, quota,
    noquota, lazytime, nolazytime, usrquota, grpquota, prjquota,
    uquota, gquota, pquota, uqnoenforce, gqnoenforce, pqnoenforce,
    qnoenforce, discard, nodiscard, dax, barrier, nobarrier, logbufs,
    biosize, sunit, swidth, logbsize, allocsize, logdev, rtdev

NetBSD:

Int mount(const char *type, const char *dir, int flags, void *data, size_t data_len);

The argument data describes the file system object to be mounted, and is data_len bytes long. data is a pointer to a structure that contains the type specific arguments to mount.

For FFS (one of the most common filesystems for NetBSD) - the arguments look like :

struct ufs_args {
        char      *fspec;   /* block special file to mount */
};

Currently, we have a pseudo syscall syz_mount_image which does the job of writing the mutated chunks of the filesystem into a file based on their offsets and later configuring the loop device using vndconfig(8) and mounting the filesystem image using mount(8).

Analysis of the current approach

One way to create mountable filesystems is to convert an existing filesystem image into a syzkaller pseudo grammar representation and then add it to the corpus so that syzkaller uses it for mutation and we have a proper image.

Some of the noted issues with syzkaller approach (as noted in "Fuzzing File Systems via Two-Dimensional Input Space Exploration) :

Steps Forward

We also spent some time researching possible options to solve the existing issues and developing an approach that would give us better results.

Image mutator approach

One possible way forward is to actually use a seed image (a working filesystem image) and write a mutator which would be aware of all the metadata in the image. The mutator should be also be able to recreate metadata components such as the checksum so that the image is mountable.

An existing implementation of such a mutator is JANUS which is a filesystem mutator written for Linux with inspiration from fsck.

Grammar based approach

Syzkaller uses a pseudo-formal grammar for representing arguments to syscalls. This grammar can also be modified to actually be able to properly generate filesystem images.

Writing grammar to represent a filesystem image is quite a daunting task and we are not yet sure if it is possible but it is the approach that we have planned to take up as of now.

Proper documentation detailing the structure of a filesystem image is rather scarce which has led me to actually go through filesystem code to figure out the type, uses and limits of a certain filesystem image. This data then has to be converted to syzkaller representation to be used for fuzzing.

One advantage of writing a grammar that would be able to generate mountable images is that we would be able to get more coverage than fuzzing with a seed image, since we are also creating new images instead of just mutating the same image.

I am currently working on learning the internals of FFS and trying to write a grammar definition which can properly generate filesystem images.

Miscellaneous Work

Meanwhile, I have also been working in parallel on improving the existing state of Syzkaller.

Add kernel compiled with KUBSAN for fuzzing

So far we only used a kernel compiled with KCOV and KASAN for fuzzing with syzkaller. We also decided to add support for syzkaller building a kernel with KUBSAN and KCOV. This would help us have an another dimension in the fuzzing process.

This required some changes in the build config. We had to remove the hardcoded kernel config and add support for building a kernel with a config passed to the fuzzer. This move would also help us to easily add support for upcoming sanitizers such as KMSAN.

Improve syscall descriptions

Improving system call descriptions is a constant ongoing work - I recently added support for fuzzing syscalls such as mount, fork and posix_spawn.

We are also planning to add support for fuzzing device drivers soon.

Relevant Links

Summary

We have managed to meet most of the goals that we had planned for the GSoC project. Overall, I have had a wonderful summer with the NetBSD foundation and I look forward to working with them to complete the project.

Last but not least, I want to thank my mentors, @kamil and @cryo for their useful suggestions and guidance. I also thank Maciej for his insight and guidance which was very fundamental during the course of the project. I would also like to thank Dmitry Vyukov, Google for helping with any issues faced with regard to Syzkaller. Finally, thanks to Google to give me a good chance to work with NetBSD community.