It has been a great opportunity to contribute to NetBSD as a part of Google Summer Of Code '20. The aim of the project I am working on is to setup a proper environment to fuzz the rumpkernel syscalls. This is the first report on the progress made so far.
Rumpkernels provide all the necessary components to run applications on baremetal without the necessity of an operating system. Simply put it is way to run kernel code in user space.
The main goal of rumpkernels in netbsd is to run,debug,examine and develop kernel drivers as easy as possible in the user space without having to run the entire kernel but run the exact same kernel code in userspace. This makes most of the components(drivers) easily portable to different environments.
Rump Kernels are constructed out of components, So the drivers are built as libraries and these libraries are linked to an interface(some application) that makes use of the libraries(drivers). So we need not build the entire monolithic kernel just the required parts of the kernel.
Why Honggfuzz?
I considered Honggfuzz the best place to start with for fuzzing the syscall layer as suggested by my mentors. LibFuzzer style library fuzzing method helped me in exploring how syscalls are implemented in the rumpkernel. With LibFuzzer we have the flexibility of modifying a few in-kernel functions as per the requirements to best suit the fuzzing target.
Fuzzing target
Taking a close look at src/sys/rump/librump/rumpkern/rump_syscalls.c
we observe that this is where the rump syscalls are defined. These functions are responsible for creating the arguments structure (like wrappers) and passing it to syscalls which is define
rsys_syscall(num, data, dlen, retval)
which is defined from
rump_syscall(num, data, dlen, retval)
This function is the one that invokes the execution of the syscalls. So this should be the target for fuzzing syscalls.
Fuzzing Using Honggfuzz
Initially we used the classic LibFuzzer style.
int
LLVMFuzzerTestOneInput(const uint8_t *Data, size_t Size)
{
if(Size != 2 + 8+sizeof(uint8_t))
return 0;
ExecuteSyscallusingtheData();
return 0;
}
However this approach into issues when we had to overload copyin(), copyout(), copyinstr(), copyoutstr() functions as the pointers that is used in these functions are from the Data buffer that the fuzzer provides for each fuzz iteration.
int
copyin(const void *uaddr, void *kaddr, size_t len)
{
..
..
..
..
if (RUMP_LOCALPROC_P(curproc)) {
memcpy(kaddr, uaddr, len); <- slow
} else if (len) {
error = rump_sysproxy_copyin(RUMP_SPVM2CTL(curproc->p_vmspace),
uaddr, kaddr, len);
}
return error;
}
Honggfuzz provides us HF_ITER interface which will be useful to actively fetch inputs from the fuzzer for example.
extern void HF_ITER(uint8_t **buf, size_t *len);
int
main()
{
for (;;) {
uint8_t *buf;
size_t len;
HF_ITER(&buf, &len);
DoSomethingWithInput(buf, len);
}
return 0;
}
So we switched to a faster method of using HF_ITER() in honggfuzz to fetch the data from the fuzzer as it is relatively flexible to use.
EXTERN int
rumpns_copyin(const void *uaddr, void *kaddr, size_t len)
{
int error = 0;
if (len == 0)
return 0;
//HF_MEMGET() is a wrapper around HF_ITER()
HF_MEMGET(kaddr, len);
return error;
}
Similar overloading is done for copyout(), copyinstr(), copyoutstr().
The current efforts to fuzz the rump syscalls using honggfuzz can be found here.
This gave quite a speed bump to the "dumb" fuzzer from few tens iterations to couple of hundreds as we replaced the memcpys with a wrapper around HF_ITER().
Further work by Kamil Rytarowski
Kamil has detected that overloading copyin()/copyout() functions have a shortcoming that we are mangling internal functionality of the rump that uses this copying mechanism, especially in rump_init(), but also in other rump wrappers, e.g. opening a file with rump_sys_open().
The decision has been made to alter the fuzzing mechanism from pumping random honggfuzz assisted data into the rump-kernel APIs and intercept copyin()/copyount() family of functions to add prior knowledge of the arguments that are valid. The initial target set by Kamil was to reproduce the following rump kernel crash (first detected with syzkaller in the real kernel):
#include <sys/types.h>
#include <sys/ioctl.h>
#include <rump/rump.h>
#include <rump/rump_syscalls.h>
int
main(int argc, char **argv)
{
int filedes[2];
rump_init();
rump_sys_pipe2(filedes, 0);
rump_sys_dup2(filedes[1], filedes[0]);
rump_sys_ioctl(filedes[1], FIONWRITE);
}
https://www.netbsd.org/~kamil/panic/rump_panic.c
We can compare that panicking the real kernel was analogous to the rump calls:
#include <sys/types.h>
#include <sys/ioctl.h>
#include <unistd.h>
#include <stdio.h>
#include <stdlib.h>
#include <err.h>
#include <fcntl.h>
int
main(int argc, char **argv)
{
int filedes[2];
pipe2(filedes, 0);
dup2(filedes[1], filedes[0]);
ioctl(filedes[1], FIONWRITE);
return 0;
}
https://www.netbsd.org/~kamil/panic/panic.c
Thus, Kamil wrote a new fuzzing program that uses pip2, dup2 and ioctl APIs exclusively and has prior knowledge about valid arguments to these functions (as the reproducer is using perfectly valid NetBSD syscalls and arguments):
https://github.com/adityavardhanpadala/rumpsyscallfuzz/blob/master/honggfuzz/ioctl/ioctl_fuzz2.c
The code instead of checking out impossible (non-existent) kernel code paths, it ensures that e.g. ioctl(2) operations are always existing ioctl(2) operations.
static unsigned long
get_ioctl_request(void)
{
unsigned long u;
// ioctlprint -l -f '\t\t%x /* %n */,\n'
static const unsigned long vals[] = {
0x8014447e /* DRVRESUMEDEV */,
0xc0784802 /* HPCFBIO_GDSPCONF */,
/* A lot of skipped lines here... */
0xc0186b01 /* KFILTER_BYNAME */,
0x80047476 /* TIOCSPGRP */,
};
u = get_ulong() % __arraycount(vals);
return vals[u];
}
Kamil also rewrote the HF_MEMGET() function, so instead of recharging the buffer from the fuzzer, whenever the buffer expired the fuzzed program terminates resetting its state and checks another honggfuzz input. This intends to make the fuzzing process more predictable in terms of getting good reproducers. So far we were unable to generate good reproducers and we are still working on it.
Unfortunately (or fortunately) the code in ioctl_fuzz2.c triggers another (rump)kernel bug, unrelated to the reproducer in rump_panic.c. Furthermore, the crash looks like a race condition that breaks randomly, sometimes and honggfuzz doesn't generate good reproducers for it.
The obvious solution to this is to run the fuzzing process with TSan involved and catch such bugs quickly, unfortunately reaching MKSANITIZER for TSan + rumpkernel is still unfinished and beyond the opportunities during this GSoC.
Obstacles
- The fuzzer is still dumb meaning that we are still using just random data as arguments to the fuzzer so the coverage did not show that much improvement b/w 13 mins and 13 hours of fuzzing.
- Crash reproducers get sliced due to the way we are fetching input from fuzzer using HF_ITER() and as functions like
copyin()andcopyout()requesting quite large buffers from the fuzzers for some non-trivial functions like rump_init().
To-Do
- Improving the crash reproduction mechanism.
- Making the fuzzer smart by using grammar so that the arguments to syscalls are syntactically valid.
- Finding an optimal way to fetch data from fuzzer so that the reproducers are not sliced.
If the above improvements are done we get more coverage and if we are lucky enough lot more crashes.
Finally I'd like to thank my mentors Siddharth Muralee, Maciej Grochowski, Christos Zoulas for their guidance and Kamil Rytarowski for his constant support throughout the coding period.
This is my first report for the Google Summer of Code project I am working on for NetBSD.
Prior work: In GSoC 2012 Charles Zhang added the posix_spawn syscall which according to its SF repository at the time (maybe even now, I have not looked very much into comparing all other systems and libcs + kernels) is an in-kernel implementation of posix_spawn which provides performance benefits compared to FreeBSD and other systems which had a userspace implementation (in 2012).
After 1 week of reading POSIX and writing code, 2 weeks of coding and another 1.5 weeks of bugfixes I have successfully implemented posix_spawn in usage in system(3) and popen(3) internally.
The biggest challenge for me was to understand POSIX, to read the standard. I am used to reading more formal books, but I can't remember working with the posix standard directly before.
The next part of my Google Summer of Code project will focus on similar rewrites of NetBSD's sh(1).
system(3)
The prototype
int system(const char *command);
remains the same. Below I'm just commenting on the differences, not the whole function, and therefore only include code block where the versions differ the most. The full work can be found at gsoc2020 as well as src and will be submitted for inclusion later in the project.
Previously we'd use vfork, sigaction, and execve in this stdlib function.
The biggest difference to the 2015 version of our system version is the usage of posix_spawnattr_ where we'd use sigaction before, and posix_spawn where execve executes the command in a vfork'd child:
posix_spawnattr_init(&attr); posix_spawnattr_setsigmask(&attr, &omask); posix_spawnattr_setflags(&attr, POSIX_SPAWN_SETSIGDEF|POSIX_SPAWN_SETSIGMASK); (void)__readlockenv(); status = posix_spawn(&pid, _PATH_BSHELL, NULL, &attr, __UNCONST(argp), environ); (void)__unlockenv(); posix_spawnattr_destroy(&attr);
The full version can be found here.
The prototype of posix_spawn is:
int posix_spawn(pid_t *restrict pid, const char *restrict path, const posix_spawn_file_actions_t *file_actions, const posix_spawnattr_t *restrict attrp, char *const argv[restrict], char *const envp[restrict]);
We first initialize a spawn attributes object with the default value for all of its attributes set. A spawn attributes object is used to modify the behavior of posix_spawn().
The previous fork-exec switch included calls to sigaction to set the behavior associated with SIGINT and SIGQUIT as defined by POSIX:
The system() function shall ignore the SIGINT and SIGQUIT signals, and shall block the SIGCHLD signal, while waiting for the command to terminate. If this might cause the application to miss a signal that would have killed it, then the application should examine the return value from system() and take whatever action is appropriate to the application if the command terminated due to receipt of a signal. source: https://pubs.opengroup.org/onlinepubs/9699919799/functions/system.html
This has been achieved with a combination of posix_spawnattr_setsigmask() and posix_spawnattr_setflags() in the initialized attributes object referenced by attr.
As before we call __readlockenv() and then call posix_spawn() which returns the process ID of the child process in the variable pointed to by 'pid', and returns zero as the function return value.
The old code:
(void)__readlockenv(); switch(pid = vfork()) { case -1: /* error */ (void)__unlockenv(); sigaction(SIGINT, &intsa, NULL); sigaction(SIGQUIT, &quitsa, NULL); (void)sigprocmask(SIG_SETMASK, &omask, NULL); return -1; case 0: /* child */ sigaction(SIGINT, &intsa, NULL); sigaction(SIGQUIT, &quitsa, NULL); (void)sigprocmask(SIG_SETMASK, &omask, NULL); execve(_PATH_BSHELL, __UNCONST(argp), environ); _exit(127); } (void)__unlockenv();
popen(3), popenve(3)
As with system, the prototype of both functions remains the same:
FILE * popenve(const char *cmd, char *const *argv, char *const *envp, const char *type); FILE * popen(const char *cmd, const char *type);
Since the popen and popenve implementation in NetBSD's libc use a couple of shared helper functions, I was able to change both functions while keeping the majority of the changes focused on (some of ) the helper functions (pdes_child).
pdes_child, an internal function in popen.c, now takes one more argument (const char *cmd) for the command to pass to posix_spawn which is called in pdes_child.
pdes_child previously looked like this:
static void
pdes_child(int *pdes, const char *type)
{
struct pid *old;
/* POSIX.2 B.3.2.2 "popen() shall ensure that any streams
from previous popen() calls that remain open in the
parent process are closed in the new child process. */
for (old = pidlist; old; old = old->next)
#ifdef _REENTRANT
(void)close(old->fd); /* don't allow a flush */
#else
(void)close(fileno(old->fp)); /* don't allow a flush */
#endif
if (type[0] == 'r') {
(void)close(pdes[0]);
if (pdes[1] != STDOUT_FILENO) {
(void)dup2(pdes[1], STDOUT_FILENO);
(void)close(pdes[1]);
}
if (type[1] == '+')
(void)dup2(STDOUT_FILENO, STDIN_FILENO);
} else {
(void)close(pdes[1]);
if (pdes[0] != STDIN_FILENO) {
(void)dup2(pdes[0], STDIN_FILENO);
(void)close(pdes[0]);
}
}
}This is the new version (the whole file is here):
static int
pdes_child(int *pdes, const char *type, const char *cmd)
{
struct pid *old;
posix_spawn_file_actions_t file_action_obj;
pid_t pid;
const char *argp[] = {"sh", "-c", NULL, NULL};
argp[2] = cmd;
int error;
error = posix_spawn_file_actions_init(&file_action_obj);
if (error) {
goto fail;
}
/* POSIX.2 B.3.2.2 "popen() shall ensure that any streams
from previous popen() calls that remain open in the
parent process are closed in the new child process. */
for (old = pidlist; old; old = old->next)
#ifdef _REENTRANT
error = posix_spawn_file_actions_addclose(&file_action_obj, old->fd); /* don't allow a flush */
if (error) {
goto fail;
}
#else
error = posix_spawn_file_actions_addclose(&file_action_obj, fileno(old->fp)); /* don't allow a flush */
if (error) {
goto fail;
}
#endif
if (type[0] == 'r') {
error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[0]);
if (error) {
goto fail;
}
if (pdes[1] != STDOUT_FILENO) {
error = posix_spawn_file_actions_adddup2(&file_action_obj, pdes[1], STDOUT_FILENO);
if (error) {
goto fail;
}
error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[1]);
if (error) {
goto fail;
}
}
if (type[1] == '+') {
error = posix_spawn_file_actions_adddup2(&file_action_obj, STDOUT_FILENO, STDIN_FILENO);
if (error) {
goto fail;
}
}
} else {
error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[1]);
if (error) {
goto fail;
}
if (pdes[0] != STDIN_FILENO) {
error = posix_spawn_file_actions_adddup2(&file_action_obj, pdes[0], STDIN_FILENO);
if (error) {
goto fail;
}
error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[0]);
if (error) {
goto fail;
}
}
}
(void)__readlockenv();
error = posix_spawn(&pid, _PATH_BSHELL, &file_action_obj, 0, __UNCONST(argp), environ);
if (error) {
(void)__unlockenv();
goto fail;
}
(void)__unlockenv();
error = posix_spawn_file_actions_destroy(&file_action_obj);
/*
* TODO: if _destroy() fails we have to go on, otherwise we
* leak the pid.
*/
if (error) {
errno = error;
return -1;
}
return pid;
fail:
errno = error;
posix_spawn_file_actions_destroy(&file_action_obj);
return -1;
}On a high level what happens in pdes_child() and popen is that we first lock the pidlist_mutex. Then we create a file file action list for all concurrent popen() / popenve() instances and the side of the pipe not necessary, and the move to stdin/stdout. We unlock the pidlist_mutex. Finally we return the list and destroy.
In the new version of this helper function which now handles the majority of what popen/popenve did, we have to initialize a file_actions object which by default contains no file actions for posix_spawn() to perform. Since we have to have error handling and a common return value for the functions calling pdes_child() and deconstruction, we make use of goto in some parts of this function.
The close() and dup2() actions now get replaced by corresponding file_actions syscalls, they are used to specify a series of actions to be performed by a posix_spawn operation.
After this series of actions, we call _readlockenv(), and call posix_spawn with the file_action object and the other arguments to be executed. If it succeeds, we return the pid of the child to popen, otherwise we return -1, in both cases we destroy the file_action object before we proceed.
In popen and popenve our code has been reduced to just the 'pid == -1' branch, everything else happens in pdes_child() now.
After readlockenv we call pdes_child and pass it the command to execute in the posix_spawn'd child process; if pdes_child returns -1 we run the old error handling code. Likewise for popenve.
popen, old:
FILE *
popen(const char *cmd, const char *type)
{
struct pid *cur;
int pdes[2], serrno;
pid_t pid;
_DIAGASSERT(cmd != NULL);
_DIAGASSERT(type != NULL);
if ((cur = pdes_get(pdes, &type)) == NULL)
return NULL;
MUTEX_LOCK();
(void)__readlockenv();
switch (pid = vfork()) {
case -1: /* Error. */
serrno = errno;
(void)__unlockenv();
MUTEX_UNLOCK();
pdes_error(pdes, cur);
errno = serrno;
return NULL;
/* NOTREACHED */
case 0: /* Child. */
pdes_child(pdes, type);
execl(_PATH_BSHELL, "sh", "-c", cmd, NULL);
_exit(127);
/* NOTREACHED */
}
(void)__unlockenv();
pdes_parent(pdes, cur, pid, type);
MUTEX_UNLOCK();
return cur->fp;
}popen, new:
FILE *
popen(const char *cmd, const char *type)
{
struct pid *cur;
int pdes[2], serrno;
pid_t pid;
_DIAGASSERT(cmd != NULL);
_DIAGASSERT(type != NULL);
if ((cur = pdes_get(pdes, &type)) == NULL)
return NULL;
MUTEX_LOCK();
(void)__readlockenv();
pid = pdes_child(pdes, type, cmd);
if (pid == -1) {
/* Error. */
serrno = errno;
(void)__unlockenv();
MUTEX_UNLOCK();
pdes_error(pdes, cur);
errno = serrno;
return NULL;
/* NOTREACHED */
}
(void)__unlockenv();
pdes_parent(pdes, cur, pid, type);
MUTEX_UNLOCK();
return cur->fp;
}popenve, old:
FILE *
popenve(const char *cmd, char *const *argv, char *const *envp, const char *type)
{
struct pid *cur;
int pdes[2], serrno;
pid_t pid;
_DIAGASSERT(cmd != NULL);
_DIAGASSERT(type != NULL);
if ((cur = pdes_get(pdes, &type)) == NULL)
return NULL;
MUTEX_LOCK();
switch (pid = vfork()) {
case -1: /* Error. */
serrno = errno;
MUTEX_UNLOCK();
pdes_error(pdes, cur);
errno = serrno;
return NULL;
/* NOTREACHED */
case 0: /* Child. */
pdes_child(pdes, type);
execve(cmd, argv, envp);
_exit(127);
/* NOTREACHED */
}
pdes_parent(pdes, cur, pid, type);
MUTEX_UNLOCK();
return cur->fp;
}popenve, new:
FILE *
popenve(const char *cmd, char *const *argv, char *const *envp, const char *type)
{
struct pid *cur;
int pdes[2], serrno;
pid_t pid;
_DIAGASSERT(cmd != NULL);
_DIAGASSERT(type != NULL);
if ((cur = pdes_get(pdes, &type)) == NULL)
return NULL;
MUTEX_LOCK();
pid = pdes_child(pdes, type, cmd);
if (pid == -1) {
/* Error. */
serrno = errno;
MUTEX_UNLOCK();
pdes_error(pdes, cur);
errno = serrno;
return NULL;
/* NOTREACHED */
}
pdes_parent(pdes, cur, pid, type);
MUTEX_UNLOCK();
return cur->fp;
}This is my first report for the Google Summer of Code project I am working on for NetBSD.
Prior work: In GSoC 2012 Charles Zhang added the posix_spawn syscall which according to its SF repository at the time (maybe even now, I have not looked very much into comparing all other systems and libcs + kernels) is an in-kernel implementation of posix_spawn which provides performance benefits compared to FreeBSD and other systems which had a userspace implementation (in 2012).
After 1 week of reading POSIX and writing code, 2 weeks of coding and another 1.5 weeks of bugfixes I have successfully implemented posix_spawn in usage in system(3) and popen(3) internally.
The biggest challenge for me was to understand POSIX, to read the standard. I am used to reading more formal books, but I can't remember working with the posix standard directly before.
The next part of my Google Summer of Code project will focus on similar rewrites of NetBSD's sh(1).
system(3)
The prototype
int system(const char *command);
remains the same. Below I'm just commenting on the differences, not the whole function, and therefore only include code block where the versions differ the most. The full work can be found at gsoc2020 as well as src and will be submitted for inclusion later in the project.
Previously we'd use vfork, sigaction, and execve in this stdlib function.
The biggest difference to the 2015 version of our system version is the usage of posix_spawnattr_ where we'd use sigaction before, and posix_spawn where execve executes the command in a vfork'd child:
posix_spawnattr_init(&attr); posix_spawnattr_setsigmask(&attr, &omask); posix_spawnattr_setflags(&attr, POSIX_SPAWN_SETSIGDEF|POSIX_SPAWN_SETSIGMASK); (void)__readlockenv(); status = posix_spawn(&pid, _PATH_BSHELL, NULL, &attr, __UNCONST(argp), environ); (void)__unlockenv(); posix_spawnattr_destroy(&attr);
The full version can be found here.
The prototype of posix_spawn is:
int posix_spawn(pid_t *restrict pid, const char *restrict path, const posix_spawn_file_actions_t *file_actions, const posix_spawnattr_t *restrict attrp, char *const argv[restrict], char *const envp[restrict]);
We first initialize a spawn attributes object with the default value for all of its attributes set. A spawn attributes object is used to modify the behavior of posix_spawn().
The previous fork-exec switch included calls to sigaction to set the behavior associated with SIGINT and SIGQUIT as defined by POSIX:
The system() function shall ignore the SIGINT and SIGQUIT signals, and shall block the SIGCHLD signal, while waiting for the command to terminate. If this might cause the application to miss a signal that would have killed it, then the application should examine the return value from system() and take whatever action is appropriate to the application if the command terminated due to receipt of a signal. source: https://pubs.opengroup.org/onlinepubs/9699919799/functions/system.html
This has been achieved with a combination of posix_spawnattr_setsigmask() and posix_spawnattr_setflags() in the initialized attributes object referenced by attr.
As before we call __readlockenv() and then call posix_spawn() which returns the process ID of the child process in the variable pointed to by 'pid', and returns zero as the function return value.
The old code:
(void)__readlockenv(); switch(pid = vfork()) { case -1: /* error */ (void)__unlockenv(); sigaction(SIGINT, &intsa, NULL); sigaction(SIGQUIT, &quitsa, NULL); (void)sigprocmask(SIG_SETMASK, &omask, NULL); return -1; case 0: /* child */ sigaction(SIGINT, &intsa, NULL); sigaction(SIGQUIT, &quitsa, NULL); (void)sigprocmask(SIG_SETMASK, &omask, NULL); execve(_PATH_BSHELL, __UNCONST(argp), environ); _exit(127); } (void)__unlockenv();
popen(3), popenve(3)
As with system, the prototype of both functions remains the same:
FILE * popenve(const char *cmd, char *const *argv, char *const *envp, const char *type); FILE * popen(const char *cmd, const char *type);
Since the popen and popenve implementation in NetBSD's libc use a couple of shared helper functions, I was able to change both functions while keeping the majority of the changes focused on (some of ) the helper functions (pdes_child).
pdes_child, an internal function in popen.c, now takes one more argument (const char *cmd) for the command to pass to posix_spawn which is called in pdes_child.
pdes_child previously looked like this:
static void
pdes_child(int *pdes, const char *type)
{
struct pid *old;
/* POSIX.2 B.3.2.2 "popen() shall ensure that any streams
from previous popen() calls that remain open in the
parent process are closed in the new child process. */
for (old = pidlist; old; old = old->next)
#ifdef _REENTRANT
(void)close(old->fd); /* don't allow a flush */
#else
(void)close(fileno(old->fp)); /* don't allow a flush */
#endif
if (type[0] == 'r') {
(void)close(pdes[0]);
if (pdes[1] != STDOUT_FILENO) {
(void)dup2(pdes[1], STDOUT_FILENO);
(void)close(pdes[1]);
}
if (type[1] == '+')
(void)dup2(STDOUT_FILENO, STDIN_FILENO);
} else {
(void)close(pdes[1]);
if (pdes[0] != STDIN_FILENO) {
(void)dup2(pdes[0], STDIN_FILENO);
(void)close(pdes[0]);
}
}
}This is the new version (the whole file is here):
static int
pdes_child(int *pdes, const char *type, const char *cmd)
{
struct pid *old;
posix_spawn_file_actions_t file_action_obj;
pid_t pid;
const char *argp[] = {"sh", "-c", NULL, NULL};
argp[2] = cmd;
int error;
error = posix_spawn_file_actions_init(&file_action_obj);
if (error) {
goto fail;
}
/* POSIX.2 B.3.2.2 "popen() shall ensure that any streams
from previous popen() calls that remain open in the
parent process are closed in the new child process. */
for (old = pidlist; old; old = old->next)
#ifdef _REENTRANT
error = posix_spawn_file_actions_addclose(&file_action_obj, old->fd); /* don't allow a flush */
if (error) {
goto fail;
}
#else
error = posix_spawn_file_actions_addclose(&file_action_obj, fileno(old->fp)); /* don't allow a flush */
if (error) {
goto fail;
}
#endif
if (type[0] == 'r') {
error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[0]);
if (error) {
goto fail;
}
if (pdes[1] != STDOUT_FILENO) {
error = posix_spawn_file_actions_adddup2(&file_action_obj, pdes[1], STDOUT_FILENO);
if (error) {
goto fail;
}
error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[1]);
if (error) {
goto fail;
}
}
if (type[1] == '+') {
error = posix_spawn_file_actions_adddup2(&file_action_obj, STDOUT_FILENO, STDIN_FILENO);
if (error) {
goto fail;
}
}
} else {
error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[1]);
if (error) {
goto fail;
}
if (pdes[0] != STDIN_FILENO) {
error = posix_spawn_file_actions_adddup2(&file_action_obj, pdes[0], STDIN_FILENO);
if (error) {
goto fail;
}
error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[0]);
if (error) {
goto fail;
}
}
}
(void)__readlockenv();
error = posix_spawn(&pid, _PATH_BSHELL, &file_action_obj, 0, __UNCONST(argp), environ);
if (error) {
(void)__unlockenv();
goto fail;
}
(void)__unlockenv();
error = posix_spawn_file_actions_destroy(&file_action_obj);
/*
* TODO: if _destroy() fails we have to go on, otherwise we
* leak the pid.
*/
if (error) {
errno = error;
return -1;
}
return pid;
fail:
errno = error;
posix_spawn_file_actions_destroy(&file_action_obj);
return -1;
}On a high level what happens in pdes_child() and popen is that we first lock the pidlist_mutex. Then we create a file file action list for all concurrent popen() / popenve() instances and the side of the pipe not necessary, and the move to stdin/stdout. We unlock the pidlist_mutex. Finally we return the list and destroy.
In the new version of this helper function which now handles the majority of what popen/popenve did, we have to initialize a file_actions object which by default contains no file actions for posix_spawn() to perform. Since we have to have error handling and a common return value for the functions calling pdes_child() and deconstruction, we make use of goto in some parts of this function.
The close() and dup2() actions now get replaced by corresponding file_actions syscalls, they are used to specify a series of actions to be performed by a posix_spawn operation.
After this series of actions, we call _readlockenv(), and call posix_spawn with the file_action object and the other arguments to be executed. If it succeeds, we return the pid of the child to popen, otherwise we return -1, in both cases we destroy the file_action object before we proceed.
In popen and popenve our code has been reduced to just the 'pid == -1' branch, everything else happens in pdes_child() now.
After readlockenv we call pdes_child and pass it the command to execute in the posix_spawn'd child process; if pdes_child returns -1 we run the old error handling code. Likewise for popenve.
popen, old:
FILE *
popen(const char *cmd, const char *type)
{
struct pid *cur;
int pdes[2], serrno;
pid_t pid;
_DIAGASSERT(cmd != NULL);
_DIAGASSERT(type != NULL);
if ((cur = pdes_get(pdes, &type)) == NULL)
return NULL;
MUTEX_LOCK();
(void)__readlockenv();
switch (pid = vfork()) {
case -1: /* Error. */
serrno = errno;
(void)__unlockenv();
MUTEX_UNLOCK();
pdes_error(pdes, cur);
errno = serrno;
return NULL;
/* NOTREACHED */
case 0: /* Child. */
pdes_child(pdes, type);
execl(_PATH_BSHELL, "sh", "-c", cmd, NULL);
_exit(127);
/* NOTREACHED */
}
(void)__unlockenv();
pdes_parent(pdes, cur, pid, type);
MUTEX_UNLOCK();
return cur->fp;
}popen, new:
FILE *
popen(const char *cmd, const char *type)
{
struct pid *cur;
int pdes[2], serrno;
pid_t pid;
_DIAGASSERT(cmd != NULL);
_DIAGASSERT(type != NULL);
if ((cur = pdes_get(pdes, &type)) == NULL)
return NULL;
MUTEX_LOCK();
(void)__readlockenv();
pid = pdes_child(pdes, type, cmd);
if (pid == -1) {
/* Error. */
serrno = errno;
(void)__unlockenv();
MUTEX_UNLOCK();
pdes_error(pdes, cur);
errno = serrno;
return NULL;
/* NOTREACHED */
}
(void)__unlockenv();
pdes_parent(pdes, cur, pid, type);
MUTEX_UNLOCK();
return cur->fp;
}popenve, old:
FILE *
popenve(const char *cmd, char *const *argv, char *const *envp, const char *type)
{
struct pid *cur;
int pdes[2], serrno;
pid_t pid;
_DIAGASSERT(cmd != NULL);
_DIAGASSERT(type != NULL);
if ((cur = pdes_get(pdes, &type)) == NULL)
return NULL;
MUTEX_LOCK();
switch (pid = vfork()) {
case -1: /* Error. */
serrno = errno;
MUTEX_UNLOCK();
pdes_error(pdes, cur);
errno = serrno;
return NULL;
/* NOTREACHED */
case 0: /* Child. */
pdes_child(pdes, type);
execve(cmd, argv, envp);
_exit(127);
/* NOTREACHED */
}
pdes_parent(pdes, cur, pid, type);
MUTEX_UNLOCK();
return cur->fp;
}popenve, new:
FILE *
popenve(const char *cmd, char *const *argv, char *const *envp, const char *type)
{
struct pid *cur;
int pdes[2], serrno;
pid_t pid;
_DIAGASSERT(cmd != NULL);
_DIAGASSERT(type != NULL);
if ((cur = pdes_get(pdes, &type)) == NULL)
return NULL;
MUTEX_LOCK();
pid = pdes_child(pdes, type, cmd);
if (pid == -1) {
/* Error. */
serrno = errno;
MUTEX_UNLOCK();
pdes_error(pdes, cur);
errno = serrno;
return NULL;
/* NOTREACHED */
}
pdes_parent(pdes, cur, pid, type);
MUTEX_UNLOCK();
return cur->fp;
}This is my first report for the Google Summer of Code project I am working on for NetBSD.
Prior work: In GSoC 2012 Charles Zhang added the posix_spawn syscall which according to its SF repository at the time (maybe even now, I have not looked very much into comparing all other systems and libcs + kernels) is an in-kernel implementation of posix_spawn which provides performance benefits compared to FreeBSD and other systems which had a userspace implementation (in 2012).
After 1 week of reading POSIX and writing code, 2 weeks of coding and another 1.5 weeks of bugfixes I have successfully implemented posix_spawn in usage in system(3) and popen(3) internally.
The biggest challenge for me was to understand POSIX, to read the standard. I am used to reading more formal books, but I can't remember working with the posix standard directly before.
The next part of my Google Summer of Code project will focus on similar rewrites of NetBSD's sh(1).
system(3)
The prototype
int system(const char *command);
remains the same. Below I'm just commenting on the differences, not the whole function, and therefore only include code block where the versions differ the most. The full work can be found at gsoc2020 as well as src and will be submitted for inclusion later in the project.
Previously we'd use vfork, sigaction, and execve in this stdlib function.
The biggest difference to the 2015 version of our system version is the usage of posix_spawnattr_ where we'd use sigaction before, and posix_spawn where execve executes the command in a vfork'd child:
posix_spawnattr_init(&attr); posix_spawnattr_setsigmask(&attr, &omask); posix_spawnattr_setflags(&attr, POSIX_SPAWN_SETSIGDEF|POSIX_SPAWN_SETSIGMASK); (void)__readlockenv(); status = posix_spawn(&pid, _PATH_BSHELL, NULL, &attr, __UNCONST(argp), environ); (void)__unlockenv(); posix_spawnattr_destroy(&attr);
The full version can be found here.
The prototype of posix_spawn is:
int posix_spawn(pid_t *restrict pid, const char *restrict path, const posix_spawn_file_actions_t *file_actions, const posix_spawnattr_t *restrict attrp, char *const argv[restrict], char *const envp[restrict]);
We first initialize a spawn attributes object with the default value for all of its attributes set. A spawn attributes object is used to modify the behavior of posix_spawn().
The previous fork-exec switch included calls to sigaction to set the behavior associated with SIGINT and SIGQUIT as defined by POSIX:
The system() function shall ignore the SIGINT and SIGQUIT signals, and shall block the SIGCHLD signal, while waiting for the command to terminate. If this might cause the application to miss a signal that would have killed it, then the application should examine the return value from system() and take whatever action is appropriate to the application if the command terminated due to receipt of a signal. source: https://pubs.opengroup.org/onlinepubs/9699919799/functions/system.html
This has been achieved with a combination of posix_spawnattr_setsigmask() and posix_spawnattr_setflags() in the initialized attributes object referenced by attr.
As before we call __readlockenv() and then call posix_spawn() which returns the process ID of the child process in the variable pointed to by 'pid', and returns zero as the function return value.
The old code:
(void)__readlockenv(); switch(pid = vfork()) { case -1: /* error */ (void)__unlockenv(); sigaction(SIGINT, &intsa, NULL); sigaction(SIGQUIT, &quitsa, NULL); (void)sigprocmask(SIG_SETMASK, &omask, NULL); return -1; case 0: /* child */ sigaction(SIGINT, &intsa, NULL); sigaction(SIGQUIT, &quitsa, NULL); (void)sigprocmask(SIG_SETMASK, &omask, NULL); execve(_PATH_BSHELL, __UNCONST(argp), environ); _exit(127); } (void)__unlockenv();
popen(3), popenve(3)
As with system, the prototype of both functions remains the same:
FILE * popenve(const char *cmd, char *const *argv, char *const *envp, const char *type); FILE * popen(const char *cmd, const char *type);
Since the popen and popenve implementation in NetBSD's libc use a couple of shared helper functions, I was able to change both functions while keeping the majority of the changes focused on (some of ) the helper functions (pdes_child).
pdes_child, an internal function in popen.c, now takes one more argument (const char *cmd) for the command to pass to posix_spawn which is called in pdes_child.
pdes_child previously looked like this:
static void
pdes_child(int *pdes, const char *type)
{
struct pid *old;
/* POSIX.2 B.3.2.2 "popen() shall ensure that any streams
from previous popen() calls that remain open in the
parent process are closed in the new child process. */
for (old = pidlist; old; old = old->next)
#ifdef _REENTRANT
(void)close(old->fd); /* don't allow a flush */
#else
(void)close(fileno(old->fp)); /* don't allow a flush */
#endif
if (type[0] == 'r') {
(void)close(pdes[0]);
if (pdes[1] != STDOUT_FILENO) {
(void)dup2(pdes[1], STDOUT_FILENO);
(void)close(pdes[1]);
}
if (type[1] == '+')
(void)dup2(STDOUT_FILENO, STDIN_FILENO);
} else {
(void)close(pdes[1]);
if (pdes[0] != STDIN_FILENO) {
(void)dup2(pdes[0], STDIN_FILENO);
(void)close(pdes[0]);
}
}
}This is the new version (the whole file is here):
static int
pdes_child(int *pdes, const char *type, const char *cmd)
{
struct pid *old;
posix_spawn_file_actions_t file_action_obj;
pid_t pid;
const char *argp[] = {"sh", "-c", NULL, NULL};
argp[2] = cmd;
int error;
error = posix_spawn_file_actions_init(&file_action_obj);
if (error) {
goto fail;
}
/* POSIX.2 B.3.2.2 "popen() shall ensure that any streams
from previous popen() calls that remain open in the
parent process are closed in the new child process. */
for (old = pidlist; old; old = old->next)
#ifdef _REENTRANT
error = posix_spawn_file_actions_addclose(&file_action_obj, old->fd); /* don't allow a flush */
if (error) {
goto fail;
}
#else
error = posix_spawn_file_actions_addclose(&file_action_obj, fileno(old->fp)); /* don't allow a flush */
if (error) {
goto fail;
}
#endif
if (type[0] == 'r') {
error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[0]);
if (error) {
goto fail;
}
if (pdes[1] != STDOUT_FILENO) {
error = posix_spawn_file_actions_adddup2(&file_action_obj, pdes[1], STDOUT_FILENO);
if (error) {
goto fail;
}
error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[1]);
if (error) {
goto fail;
}
}
if (type[1] == '+') {
error = posix_spawn_file_actions_adddup2(&file_action_obj, STDOUT_FILENO, STDIN_FILENO);
if (error) {
goto fail;
}
}
} else {
error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[1]);
if (error) {
goto fail;
}
if (pdes[0] != STDIN_FILENO) {
error = posix_spawn_file_actions_adddup2(&file_action_obj, pdes[0], STDIN_FILENO);
if (error) {
goto fail;
}
error = posix_spawn_file_actions_addclose(&file_action_obj, pdes[0]);
if (error) {
goto fail;
}
}
}
(void)__readlockenv();
error = posix_spawn(&pid, _PATH_BSHELL, &file_action_obj, 0, __UNCONST(argp), environ);
if (error) {
(void)__unlockenv();
goto fail;
}
(void)__unlockenv();
error = posix_spawn_file_actions_destroy(&file_action_obj);
/*
* TODO: if _destroy() fails we have to go on, otherwise we
* leak the pid.
*/
if (error) {
errno = error;
return -1;
}
return pid;
fail:
errno = error;
posix_spawn_file_actions_destroy(&file_action_obj);
return -1;
}On a high level what happens in pdes_child() and popen is that we first lock the pidlist_mutex. Then we create a file file action list for all concurrent popen() / popenve() instances and the side of the pipe not necessary, and the move to stdin/stdout. We unlock the pidlist_mutex. Finally we return the list and destroy.
In the new version of this helper function which now handles the majority of what popen/popenve did, we have to initialize a file_actions object which by default contains no file actions for posix_spawn() to perform. Since we have to have error handling and a common return value for the functions calling pdes_child() and deconstruction, we make use of goto in some parts of this function.
The close() and dup2() actions now get replaced by corresponding file_actions syscalls, they are used to specify a series of actions to be performed by a posix_spawn operation.
After this series of actions, we call _readlockenv(), and call posix_spawn with the file_action object and the other arguments to be executed. If it succeeds, we return the pid of the child to popen, otherwise we return -1, in both cases we destroy the file_action object before we proceed.
In popen and popenve our code has been reduced to just the 'pid == -1' branch, everything else happens in pdes_child() now.
After readlockenv we call pdes_child and pass it the command to execute in the posix_spawn'd child process; if pdes_child returns -1 we run the old error handling code. Likewise for popenve.
popen, old:
FILE *
popen(const char *cmd, const char *type)
{
struct pid *cur;
int pdes[2], serrno;
pid_t pid;
_DIAGASSERT(cmd != NULL);
_DIAGASSERT(type != NULL);
if ((cur = pdes_get(pdes, &type)) == NULL)
return NULL;
MUTEX_LOCK();
(void)__readlockenv();
switch (pid = vfork()) {
case -1: /* Error. */
serrno = errno;
(void)__unlockenv();
MUTEX_UNLOCK();
pdes_error(pdes, cur);
errno = serrno;
return NULL;
/* NOTREACHED */
case 0: /* Child. */
pdes_child(pdes, type);
execl(_PATH_BSHELL, "sh", "-c", cmd, NULL);
_exit(127);
/* NOTREACHED */
}
(void)__unlockenv();
pdes_parent(pdes, cur, pid, type);
MUTEX_UNLOCK();
return cur->fp;
}popen, new:
FILE *
popen(const char *cmd, const char *type)
{
struct pid *cur;
int pdes[2], serrno;
pid_t pid;
_DIAGASSERT(cmd != NULL);
_DIAGASSERT(type != NULL);
if ((cur = pdes_get(pdes, &type)) == NULL)
return NULL;
MUTEX_LOCK();
(void)__readlockenv();
pid = pdes_child(pdes, type, cmd);
if (pid == -1) {
/* Error. */
serrno = errno;
(void)__unlockenv();
MUTEX_UNLOCK();
pdes_error(pdes, cur);
errno = serrno;
return NULL;
/* NOTREACHED */
}
(void)__unlockenv();
pdes_parent(pdes, cur, pid, type);
MUTEX_UNLOCK();
return cur->fp;
}popenve, old:
FILE *
popenve(const char *cmd, char *const *argv, char *const *envp, const char *type)
{
struct pid *cur;
int pdes[2], serrno;
pid_t pid;
_DIAGASSERT(cmd != NULL);
_DIAGASSERT(type != NULL);
if ((cur = pdes_get(pdes, &type)) == NULL)
return NULL;
MUTEX_LOCK();
switch (pid = vfork()) {
case -1: /* Error. */
serrno = errno;
MUTEX_UNLOCK();
pdes_error(pdes, cur);
errno = serrno;
return NULL;
/* NOTREACHED */
case 0: /* Child. */
pdes_child(pdes, type);
execve(cmd, argv, envp);
_exit(127);
/* NOTREACHED */
}
pdes_parent(pdes, cur, pid, type);
MUTEX_UNLOCK();
return cur->fp;
}popenve, new:
FILE *
popenve(const char *cmd, char *const *argv, char *const *envp, const char *type)
{
struct pid *cur;
int pdes[2], serrno;
pid_t pid;
_DIAGASSERT(cmd != NULL);
_DIAGASSERT(type != NULL);
if ((cur = pdes_get(pdes, &type)) == NULL)
return NULL;
MUTEX_LOCK();
pid = pdes_child(pdes, type, cmd);
if (pid == -1) {
/* Error. */
serrno = errno;
MUTEX_UNLOCK();
pdes_error(pdes, cur);
errno = serrno;
return NULL;
/* NOTREACHED */
}
pdes_parent(pdes, cur, pid, type);
MUTEX_UNLOCK();
return cur->fp;
}Introduction:
The objective of this project is to fuzz the various protocols and layers of the network stack of NetBSD using rumpkernel. This project is being carried out as a part of GSoC 2020. This blog post is regarding the project, the concepts and tools involved, the objectives and the current progress and next steps.
Fuzzing:
Fuzzing or fuzz testing is an automated software testing technique in which a program is tested by passing unusual, unexpected or semi-random input generated data to the input of the program and repeatedly doing so, trying to crash the program and detect potential bugs or undealt corner cases.
There are several tools available today that enable this which are known as fuzzers. An effective fuzzer generates semi-valid inputs that are "valid enough" in that they are not directly rejected by the parser, but do create unexpected behaviors deeper in the program and are "invalid enough" to expose corner cases that have not been properly dealt with.
Fuzzers can be of various types like dumb vs smart, generation-based vs mutation-based and so on. A dumb fuzzer generates random input without looking at the input format or model but it can follow some sophisticated algorithms like in AFL, though considered a dumb fuzzer as it just flips bits and replaces bytes, still uses a genetic algorithm to create new test cases, where as a smart fuzzer will follow an input model to generate semi-random data that can penetrate well in the code and trigger more edge cases. Mutation and generation fuzzers handle test case generation differently. Mutation fuzzers mutate a supplied seed input object, while generation fuzzers generate new test cases from a supplied model.
Some examples of popular fuzzers are: AFL(American Fuzzy Lop), Syzkaller, Honggfuzz.
RumpKernel
Kernels can have several different architectures like monolithic, microkernel, exokernel etc. An interesting architecture is the “anykernel” architecture, according to wikipedia, “The "anykernel" concept refers to an architecture-agnostic approach to drivers where drivers can either be compiled into the monolithic kernel or be run as a userspace process, microkernel-style, without code changes.” Rumpkernel is an implementation of this anykernel architecture. In case of the NetBSD rumpkernel, all the NetBSD subsystems like file system, network stack, drivers etc are compiled into standalone libraries that can be linked into any application process to utilize the functionalities of the kernel, like the file system or the drivers. This allows us to run and test various components of NetBSD kernel as a library linked to programs running in the user space.
This idea of rumpkernel is really helpful in fuzzing the components of the kernel. We can fuzz separate subsystems and allow it to crash without having to manage the crash of a running Operating System. Also the fact that context switching has an overhead in case syscalls are made to the kernel, Rumpkernel running in the userspace can eliminate this and save time.(Also since the spectre-meltdown vulnerabilities, system calls have become more costly due to the security reasons)
HonggFuzz + Rumpkernel Network Stack:
In this project we will use the outlined Rumpkernel’s network stack and a fuzzer called the honggfuzz. Honggfuzz is a security oriented, feedback-driven, evolutionary, easy-to-use fuzzer. It is maintained by google.(https://github.com/google/honggfuzz)
The project is hosted on github at: https://github.com/NJnisarg/fuzznetrump/ .The Readme can help in setting it up. We first require NetBSD installed either on a physical machine or on a virtual machine like qemu. Then we will build the NetBSD distribution by grabbing the latest sources(https://github.com/NetBSD/src). We will enable fuzzer coverage by using MKSANITIZER and MKLIBCSANITIZER flags and using the ASan(Address) and UBSan(Undefined Behavior) sanitizers. These sanitizers will help the fuzzer in catching bugs related to undefined behavior and address and memory leaks. After that we will grab one of the fuzzer programs(example: src/hfuzz_ip_output_fuzz.c) and chroot into the newly built distribution from the host NetBSD OS. Then we will compile it using hfuzz-clang by linking the required rumpkernel libraries (for example in our case: rump, rumpnet, rumpnet_netinet and so on). This is where we use the rumpkernel as libraries linked to our program. The program will access the network stack of the linked rumpkernel and the fuzzer will fuzz those components of the kernel. The compiled binary will be run with honggfuzz. Detailed steps are outlined in the Readme of the linked repository.
Our Approach for network stack fuzzing:
We have planned to fuzz various protocols at different layers of the TCP/IP stack. We have started with mostly widely used yet simple protocols like IP(v4), UDP etc. Along the progress of the project, we will be adding support for more L3(and above) protocols like ICMP, IP(v6), TCP as well as L2 protocols like Ethernet as a bit later phase.
The network stack has 2 paths:
- Input/ingress path
- Output/egress path
A packet is sent down the network stack via a socket from an application from the output path, whereas a packet is said to be received on a network interface into the network stack via the input path. Each network protocol has major input and output APIs for the packet processing. Example IP protocol has an ip_input() function to process an incoming packet and an ip_output() function to process an outgoing packet. We are planning to fuzz each protocol’s output and input APIs by sending the packet via the output path and receiving the packet via input path respectively.
In order to fuzz the output and input path, the network stack setup and configuration we have is as follows:
- We have a TUN device to which we can read and write a packet.
- We have a socket that is bound to the TUN device, which can send and receive packets
- In order to fuzz the input path, we “write” a packet to the TUN interface, which emulates a received packet on the network stack.
- In order to fuzz the output path, we send a packet via the socket to the TUN interface to fuzz the output path.
For carrying out all the above setup, we have separated out the common function for creating and configuring the TUN device and socket into a file called “net_config.c”
Also in order to reduce the rejection of packets carrying random data for trivial cases like checksum or header len, we have created functions that create/forge semi-random packets using the input data from honggfuzz. We manipulate certain fields in the packet to ensure that it does not get rejected trivially by the network stack and hence can reach and trigger deeper parts of the code. These functions for packet creations are located in the “pkt_create.c” file. For each protocol we fuzz, we add these functions to forge the headers and the packet. Currently we have support from UDP and IP(v4).
With these building blocks we have written programs like hfuzz_ip_output_fuzz.c, hfuzz_ip_input_fuzz.c etc, which setup the TUN device and socket using net_config.c and after taking the random data from honggfuzz, use it to forge a packet and send it down or up the stack. We compile these programs using hfuzz-clang as mentioned above and run it under honggfuzz to fuzz the network stack’s particular APIs.
Current Progress:
Following things were worked upon in the first phase:
- Getting honggfuzz functional for NetBSD(thanks to Kamil for those patches)
- Coming up with the strategy for network configuration and packet creation. Writing utilities for the same.
- Adding fuzzing code for protocols like IP(v4) and UDP.
- Carrying out fuzzing for those protocols.
Next Steps:
As next steps following things are planned for upcoming phase:
- Making changes and improvements by taking suggestions from the mentors.
- Adding support for ICMP, IP(v6), TCP and later on for Ethernet.
- Analyze and come up with effective ways to improve the fuzzing by focusing on the packet creation part.
- Standardize the code to be extensible for adding future protocols.
Introduction:
The objective of this project is to fuzz the various protocols and layers of the network stack of NetBSD using rumpkernel. This project is being carried out as a part of GSoC 2020. This blog post is regarding the project, the concepts and tools involved, the objectives and the current progress and next steps.
Fuzzing:
Fuzzing or fuzz testing is an automated software testing technique in which a program is tested by passing unusual, unexpected or semi-random input generated data to the input of the program and repeatedly doing so, trying to crash the program and detect potential bugs or undealt corner cases.
There are several tools available today that enable this which are known as fuzzers. An effective fuzzer generates semi-valid inputs that are "valid enough" in that they are not directly rejected by the parser, but do create unexpected behaviors deeper in the program and are "invalid enough" to expose corner cases that have not been properly dealt with.
Fuzzers can be of various types like dumb vs smart, generation-based vs mutation-based and so on. A dumb fuzzer generates random input without looking at the input format or model but it can follow some sophisticated algorithms like in AFL, though considered a dumb fuzzer as it just flips bits and replaces bytes, still uses a genetic algorithm to create new test cases, where as a smart fuzzer will follow an input model to generate semi-random data that can penetrate well in the code and trigger more edge cases. Mutation and generation fuzzers handle test case generation differently. Mutation fuzzers mutate a supplied seed input object, while generation fuzzers generate new test cases from a supplied model.
Some examples of popular fuzzers are: AFL(American Fuzzy Lop), Syzkaller, Honggfuzz.
RumpKernel
Kernels can have several different architectures like monolithic, microkernel, exokernel etc. An interesting architecture is the “anykernel” architecture, according to wikipedia, “The "anykernel" concept refers to an architecture-agnostic approach to drivers where drivers can either be compiled into the monolithic kernel or be run as a userspace process, microkernel-style, without code changes.” Rumpkernel is an implementation of this anykernel architecture. In case of the NetBSD rumpkernel, all the NetBSD subsystems like file system, network stack, drivers etc are compiled into standalone libraries that can be linked into any application process to utilize the functionalities of the kernel, like the file system or the drivers. This allows us to run and test various components of NetBSD kernel as a library linked to programs running in the user space.
This idea of rumpkernel is really helpful in fuzzing the components of the kernel. We can fuzz separate subsystems and allow it to crash without having to manage the crash of a running Operating System. Also the fact that context switching has an overhead in case syscalls are made to the kernel, Rumpkernel running in the userspace can eliminate this and save time.(Also since the spectre-meltdown vulnerabilities, system calls have become more costly due to the security reasons)
HonggFuzz + Rumpkernel Network Stack:
In this project we will use the outlined Rumpkernel’s network stack and a fuzzer called the honggfuzz. Honggfuzz is a security oriented, feedback-driven, evolutionary, easy-to-use fuzzer. It is maintained by google.(https://github.com/google/honggfuzz)
The project is hosted on github at: https://github.com/NJnisarg/fuzznetrump/ .The Readme can help in setting it up. We first require NetBSD installed either on a physical machine or on a virtual machine like qemu. Then we will build the NetBSD distribution by grabbing the latest sources(https://github.com/NetBSD/src). We will enable fuzzer coverage by using MKSANITIZER and MKLIBCSANITIZER flags and using the ASan(Address) and UBSan(Undefined Behavior) sanitizers. These sanitizers will help the fuzzer in catching bugs related to undefined behavior and address and memory leaks. After that we will grab one of the fuzzer programs(example: src/hfuzz_ip_output_fuzz.c) and chroot into the newly built distribution from the host NetBSD OS. Then we will compile it using hfuzz-clang by linking the required rumpkernel libraries (for example in our case: rump, rumpnet, rumpnet_netinet and so on). This is where we use the rumpkernel as libraries linked to our program. The program will access the network stack of the linked rumpkernel and the fuzzer will fuzz those components of the kernel. The compiled binary will be run with honggfuzz. Detailed steps are outlined in the Readme of the linked repository.
Our Approach for network stack fuzzing:
We have planned to fuzz various protocols at different layers of the TCP/IP stack. We have started with mostly widely used yet simple protocols like IP(v4), UDP etc. Along the progress of the project, we will be adding support for more L3(and above) protocols like ICMP, IP(v6), TCP as well as L2 protocols like Ethernet as a bit later phase.
The network stack has 2 paths:
- Input/ingress path
- Output/egress path
A packet is sent down the network stack via a socket from an application from the output path, whereas a packet is said to be received on a network interface into the network stack via the input path. Each network protocol has major input and output APIs for the packet processing. Example IP protocol has an ip_input() function to process an incoming packet and an ip_output() function to process an outgoing packet. We are planning to fuzz each protocol’s output and input APIs by sending the packet via the output path and receiving the packet via input path respectively.
In order to fuzz the output and input path, the network stack setup and configuration we have is as follows:
- We have a TUN device to which we can read and write a packet.
- We have a socket that is bound to the TUN device, which can send and receive packets
- In order to fuzz the input path, we “write” a packet to the TUN interface, which emulates a received packet on the network stack.
- In order to fuzz the output path, we send a packet via the socket to the TUN interface to fuzz the output path.
For carrying out all the above setup, we have separated out the common function for creating and configuring the TUN device and socket into a file called “net_config.c”
Also in order to reduce the rejection of packets carrying random data for trivial cases like checksum or header len, we have created functions that create/forge semi-random packets using the input data from honggfuzz. We manipulate certain fields in the packet to ensure that it does not get rejected trivially by the network stack and hence can reach and trigger deeper parts of the code. These functions for packet creations are located in the “pkt_create.c” file. For each protocol we fuzz, we add these functions to forge the headers and the packet. Currently we have support from UDP and IP(v4).
With these building blocks we have written programs like hfuzz_ip_output_fuzz.c, hfuzz_ip_input_fuzz.c etc, which setup the TUN device and socket using net_config.c and after taking the random data from honggfuzz, use it to forge a packet and send it down or up the stack. We compile these programs using hfuzz-clang as mentioned above and run it under honggfuzz to fuzz the network stack’s particular APIs.
Current Progress:
Following things were worked upon in the first phase:
- Getting honggfuzz functional for NetBSD(thanks to Kamil for those patches)
- Coming up with the strategy for network configuration and packet creation. Writing utilities for the same.
- Adding fuzzing code for protocols like IP(v4) and UDP.
- Carrying out fuzzing for those protocols.
Next Steps:
As next steps following things are planned for upcoming phase:
- Making changes and improvements by taking suggestions from the mentors.
- Adding support for ICMP, IP(v6), TCP and later on for Ethernet.
- Analyze and come up with effective ways to improve the fuzzing by focusing on the packet creation part.
- Standardize the code to be extensible for adding future protocols.
Introduction
My GSoC project under NetBSD involves the development of test framework of curses library. Automated Test Framework (ATF) was introduced in 2007 but ATF cannot be used directly for curses testing for several reasons most important of them being curses has functions which do timed reads/writes which is hard to do with just piping characters to test applications. Also, stdin is not a tty device and behaves differently and may affect the results. A lot of work regarding this has been done and we have a separate test framework in place for testing curses.
The aim of project is to build a robust test suite for the library and complete the SUSv2 specification. This includes writing tests for the remaining functions and enhancing the existing ones. Meanwhile, the support for complex character function has to be completed along with fixing some bugs, adding features and improving the test framework.
Why did I chose this project?
I am a final year undergraduate at Indian Institute of Technology, Kanpur. I have my majors in Computer Science and Engineering, and I am specifically interested in algorithms and computer systems. I had worked on building and testing a library on Distributed Tracing at an internship and understand the usefulness of having a test suite in place. Libcurses being very special in itself for the testing purpose interested me. Knowing some of the concepts of compiler design made my interest a bit more profound.
Test Framwork
The testframework consists of 2 programs, director and slave. The framework provides its own simple language for writing tests. The slave is a curses application capable of running any curses function, while the director acts as a coordinator and interprets the test file and drives the slave program. The director can also capture slave's output which can be used for comparison with desired output.
The director forks a process operating in pty and executes a slave program on that fork. The master side of pty is used for getting the data stream that the curses function call produces which can be futher used to check the correctness of behaviour. Director and slave communicate via pipes; command pipe and slave pipe. The command pipe carries the function name and arguments, while slave pipe carries return codes and values from function calls.
Let's walk through a sample test to understand how this works. Consider a sample program:
include start
call win newwin 2 5 2 5
check win NON_NULL
call OK waddstr $win "Hello World!"
call OK wrefresh $win
compare waddstr_refresh.chk
This is a basic program which initialises the screen, creates new window, checks if the window creation was successful, adds as string "Hello World!" on the window, refreshes the window, and compares it with desired output stored in check file. The details of the language can be accessed at libcurses testframe.
The test file is interpreted by the language parser and the correponding actions are taken. Let's look how line #2 is processed. This command creates a window using newwin(). The line is ultimately parsed as call: CALL result fn_name args eol grammar rule and executes the function do_funtion_call()). Now, this function sends function name and arguments using command pipe to the slave. The slave, who is waiting to get command from the director, reads from the pipe and executes the command. This executes the correponding curses function from the command table and the pointer to new window is returned via the slave pipe (here) after passing wrappers of functions. The director recieves them, and returned value is assigned to a variable(win in line#2) or compared (OK in line#4). This is the typical life cycle of a certain function call made in tests.
Along with these, the test framework provides capability to include other test (line#1), check the variable content (line#3), compare the data stream due to function call in pty with desired stream (line#6). Tester can also provide inputs to functions via input directive, perform delay via delay directive, assign values to variables via assign directive, and create a wide or complex charater via wchar and cchar directives respectively. The framework supports 3 kind of strings; null terminated string, byte string, and string of type chtype, based on the quotes enclosing it.
Progress till the first evaluation
Improvements in the framework:
- Automated the checkfile generation that has to be done manually earlier.
- Completed the support for complex chacter tests in director and slave.
- Added features like variable-variable comparison.
- Fixed non-critical bugs in the framework.
- Refactored the code.
Testing and bug reports
- Wrote tests for wide character routines.
- Reported bugs (and possible fixes if I know):
Project Proposal and References:
- Proposal: https://github.com/NamanJain8/curses/blob/master/reports/proposal.pdf
- Project Repo: https://github.com/NamanJain8/curses
- Test Language Details: https://github.com/NetBSD/src/blob/trunk/tests/lib/libcurses/testframe.txt
- Wonderful report by Brett Lymn: https://github.com/NamanJain8/curses/blob/master/reports/curses-testframe.pdf
Introduction
My GSoC project under NetBSD involves the development of test framework of curses library. Automated Test Framework (ATF) was introduced in 2007 but ATF cannot be used directly for curses testing for several reasons most important of them being curses has functions which do timed reads/writes which is hard to do with just piping characters to test applications. Also, stdin is not a tty device and behaves differently and may affect the results. A lot of work regarding this has been done and we have a separate test framework in place for testing curses.
The aim of project is to build a robust test suite for the library and complete the SUSv2 specification. This includes writing tests for the remaining functions and enhancing the existing ones. Meanwhile, the support for complex character function has to be completed along with fixing some bugs, adding features and improving the test framework.
Why did I chose this project?
I am a final year undergraduate at Indian Institute of Technology, Kanpur. I have my majors in Computer Science and Engineering, and I am specifically interested in algorithms and computer systems. I had worked on building and testing a library on Distributed Tracing at an internship and understand the usefulness of having a test suite in place. Libcurses being very special in itself for the testing purpose interested me. Knowing some of the concepts of compiler design made my interest a bit more profound.
Test Framwork
The testframework consists of 2 programs, director and slave. The framework provides its own simple language for writing tests. The slave is a curses application capable of running any curses function, while the director acts as a coordinator and interprets the test file and drives the slave program. The director can also capture slave's output which can be used for comparison with desired output.
The director forks a process operating in pty and executes a slave program on that fork. The master side of pty is used for getting the data stream that the curses function call produces which can be futher used to check the correctness of behaviour. Director and slave communicate via pipes; command pipe and slave pipe. The command pipe carries the function name and arguments, while slave pipe carries return codes and values from function calls.
Let's walk through a sample test to understand how this works. Consider a sample program:
include start
call win newwin 2 5 2 5
check win NON_NULL
call OK waddstr $win "Hello World!"
call OK wrefresh $win
compare waddstr_refresh.chk
This is a basic program which initialises the screen, creates new window, checks if the window creation was successful, adds as string "Hello World!" on the window, refreshes the window, and compares it with desired output stored in check file. The details of the language can be accessed at libcurses testframe.
The test file is interpreted by the language parser and the correponding actions are taken. Let's look how line #2 is processed. This command creates a window using newwin(). The line is ultimately parsed as call: CALL result fn_name args eol grammar rule and executes the function do_funtion_call()). Now, this function sends function name and arguments using command pipe to the slave. The slave, who is waiting to get command from the director, reads from the pipe and executes the command. This executes the correponding curses function from the command table and the pointer to new window is returned via the slave pipe (here) after passing wrappers of functions. The director recieves them, and returned value is assigned to a variable(win in line#2) or compared (OK in line#4). This is the typical life cycle of a certain function call made in tests.
Along with these, the test framework provides capability to include other test (line#1), check the variable content (line#3), compare the data stream due to function call in pty with desired stream (line#6). Tester can also provide inputs to functions via input directive, perform delay via delay directive, assign values to variables via assign directive, and create a wide or complex charater via wchar and cchar directives respectively. The framework supports 3 kind of strings; null terminated string, byte string, and string of type chtype, based on the quotes enclosing it.
Progress till the first evaluation
Improvements in the framework:
- Automated the checkfile generation that has to be done manually earlier.
- Completed the support for complex chacter tests in director and slave.
- Added features like variable-variable comparison.
- Fixed non-critical bugs in the framework.
- Refactored the code.
Testing and bug reports
- Wrote tests for wide character routines.
- Reported bugs (and possible fixes if I know):
Project Proposal and References:
- Proposal: https://github.com/NamanJain8/curses/blob/master/reports/proposal.pdf
- Project Repo: https://github.com/NamanJain8/curses
- Test Language Details: https://github.com/NetBSD/src/blob/trunk/tests/lib/libcurses/testframe.txt
- Wonderful report by Brett Lymn: https://github.com/NamanJain8/curses/blob/master/reports/curses-testframe.pdf
NetPGP is a library and suite of tools implementing OpenPGP under a BSD license. As part of Google Summer of Code 2020, we are working to extend its functionality and work towards greater parity with similar tools. During the first phase, we have made the following contributions
- Added the Blowfish block cipher
- ECDSA key creation
- ECDSA signature and verification
- Symmetric file encryption/decryption
- S2K Iterated+Salt for symmetric encryption
ECDSA key generation is done using the '--ecdsa' flag with netpgpkeys or the 'ecdsa' property if using libnetpgp
[jhigh@gsoc2020nb gsoc]$ netpgpkeys --generate-key --ecdsa --homedir=/tmp
signature secp521r1/ECDSA a0cdb04e3e8c5e34 2020-06-25
fingerprint d9e0 2ae5 1d2f a9ae eb96 ebd4 a0cd b04e 3e8c 5e34
uid jhigh@localhost
Enter passphrase for a0cdb04e3e8c5e34:
Repeat passphrase for a0cdb04e3e8c5e34:
generated keys in directory /tmp/a0cdb04e3e8c5e34
[jhigh@gsoc2020nb gsoc]$
[jhigh@gsoc2020nb gsoc]$ ls -l /tmp/a0cdb04e3e8c5e34
total 16
-rw------- 1 jhigh wheel 331 Jun 25 16:03 pubring.gpg
-rw------- 1 jhigh wheel 440 Jun 25 16:03 secring.gpg
[jhigh@gsoc2020nb gsoc]$
Signing with ECDSA does not require any changes
[jhigh@gsoc2020nb gsoc]$ netpgp --sign --homedir=/tmp/a0cdb04e3e8c5e34 --detach --armor testfile.txt
signature secp521r1/ECDSA a0cdb04e3e8c5e34 2020-06-25
fingerprint d9e0 2ae5 1d2f a9ae eb96 ebd4 a0cd b04e 3e8c 5e34
uid jhigh@localhost
netpgp passphrase:
[jhigh@gsoc2020nb gsoc]$
[jhigh@gsoc2020nb gsoc]$ cat testfile.txt.asc
-----BEGIN PGP MESSAGE-----
wqcEABMCABYFAl71EYwFAwAAAAAJEKDNsE4+jF40AAAVPgIJASyzuZgyS13FHHF/9qk6E3pYra2H
tDdkqxYzNIqKnWHaB+a4J+/R7FkZItbC/EyXH5YA68AC1dJ7tRN/tJNIWfYjAgUb75SvM2mLHk13
qmBo48S0Ai8C82G4nT7/16VF2OOUn7F/3XICghoQOyS1nxJilj8u2uphLOoy9VayL1ErORIZVw==
=p30e
-----END PGP MESSAGE-----
[jhigh@gsoc2020nb gsoc]$
Verification remains the same, as well.
[jhigh@gsoc2020nb gsoc]$ netpgp --homedir=/tmp/a0cdb04e3e8c5e34 --verify testfile.txt.asc
netpgp: assuming signed data in "testfile.txt"
Good signature for testfile.txt.asc made Thu Jun 25 16:05:16 2020
using ECDSA key a0cdb04e3e8c5e34
signature secp521r1/ECDSA a0cdb04e3e8c5e34 2020-06-25
fingerprint d9e0 2ae5 1d2f a9ae eb96 ebd4 a0cd b04e 3e8c 5e34
uid jhigh@localhost
[jhigh@gsoc2020nb gsoc]$
Symmetric encryption is now possible using the '--symmetric' flag with netpgp or the 'symmetric' property in libnetpgp
[jhigh@gsoc2020nb gsoc]$ netpgp --encrypt --symmetric --armor testfile.txt
Enter passphrase:
Repeat passphrase:
[jhigh@gsoc2020nb gsoc]$
[jhigh@gsoc2020nb gsoc]$ cat testfile.txt.asc
-----BEGIN PGP MESSAGE-----
wwwEAwEIc39k1V6xVi3SPwEl2ww75Ufjhw7UA0gO/niahHWK50DFHSD1lB10nUyCTgRLe6iS9QZl
av5Nji9BuQFcrqo03I1jG/L9s/4hww==
=x41O
-----END PGP MESSAGE-----
[jhigh@gsoc2020nb gsoc]$
Decryption of symmetric packets requires no changes
[jhigh@gsoc2020nb gsoc]$ netpgp --decrypt testfile.txt.asc
netpgp passphrase:
[jhigh@gsoc2020nb gsoc]$
We added two new flags to support s2k mode 3: '--s2k-mode' and '--s2k-count'. See RFC4880 for details.
[jhigh@gsoc2020nb gsoc]$ netpgp --encrypt --symmetric --s2k-mode=3 --s2k-count=96 testfile.txt
Enter passphrase:
Repeat passphrase:
[jhigh@gsoc2020nb gsoc]$
[jhigh@gsoc2020nb gsoc]$ gpg -d testfile.txt.gpg
gpg: CAST5 encrypted data
gpg: encrypted with 1 passphrase
this
is
a
test
[jhigh@gsoc2020nb gsoc]$
NetPGP is a library and suite of tools implementing OpenPGP under a BSD license. As part of Google Summer of Code 2020, we are working to extend its functionality and work towards greater parity with similar tools. During the first phase, we have made the following contributions
- Added the Blowfish block cipher
- ECDSA key creation
- ECDSA signature and verification
- Symmetric file encryption/decryption
- S2K Iterated+Salt for symmetric encryption
ECDSA key generation is done using the '--ecdsa' flag with netpgpkeys or the 'ecdsa' property if using libnetpgp
[jhigh@gsoc2020nb gsoc]$ netpgpkeys --generate-key --ecdsa --homedir=/tmp
signature secp521r1/ECDSA a0cdb04e3e8c5e34 2020-06-25
fingerprint d9e0 2ae5 1d2f a9ae eb96 ebd4 a0cd b04e 3e8c 5e34
uid jhigh@localhost
Enter passphrase for a0cdb04e3e8c5e34:
Repeat passphrase for a0cdb04e3e8c5e34:
generated keys in directory /tmp/a0cdb04e3e8c5e34
[jhigh@gsoc2020nb gsoc]$
[jhigh@gsoc2020nb gsoc]$ ls -l /tmp/a0cdb04e3e8c5e34
total 16
-rw------- 1 jhigh wheel 331 Jun 25 16:03 pubring.gpg
-rw------- 1 jhigh wheel 440 Jun 25 16:03 secring.gpg
[jhigh@gsoc2020nb gsoc]$
Signing with ECDSA does not require any changes
[jhigh@gsoc2020nb gsoc]$ netpgp --sign --homedir=/tmp/a0cdb04e3e8c5e34 --detach --armor testfile.txt
signature secp521r1/ECDSA a0cdb04e3e8c5e34 2020-06-25
fingerprint d9e0 2ae5 1d2f a9ae eb96 ebd4 a0cd b04e 3e8c 5e34
uid jhigh@localhost
netpgp passphrase:
[jhigh@gsoc2020nb gsoc]$
[jhigh@gsoc2020nb gsoc]$ cat testfile.txt.asc
-----BEGIN PGP MESSAGE-----
wqcEABMCABYFAl71EYwFAwAAAAAJEKDNsE4+jF40AAAVPgIJASyzuZgyS13FHHF/9qk6E3pYra2H
tDdkqxYzNIqKnWHaB+a4J+/R7FkZItbC/EyXH5YA68AC1dJ7tRN/tJNIWfYjAgUb75SvM2mLHk13
qmBo48S0Ai8C82G4nT7/16VF2OOUn7F/3XICghoQOyS1nxJilj8u2uphLOoy9VayL1ErORIZVw==
=p30e
-----END PGP MESSAGE-----
[jhigh@gsoc2020nb gsoc]$
Verification remains the same, as well.
[jhigh@gsoc2020nb gsoc]$ netpgp --homedir=/tmp/a0cdb04e3e8c5e34 --verify testfile.txt.asc
netpgp: assuming signed data in "testfile.txt"
Good signature for testfile.txt.asc made Thu Jun 25 16:05:16 2020
using ECDSA key a0cdb04e3e8c5e34
signature secp521r1/ECDSA a0cdb04e3e8c5e34 2020-06-25
fingerprint d9e0 2ae5 1d2f a9ae eb96 ebd4 a0cd b04e 3e8c 5e34
uid jhigh@localhost
[jhigh@gsoc2020nb gsoc]$
Symmetric encryption is now possible using the '--symmetric' flag with netpgp or the 'symmetric' property in libnetpgp
[jhigh@gsoc2020nb gsoc]$ netpgp --encrypt --symmetric --armor testfile.txt
Enter passphrase:
Repeat passphrase:
[jhigh@gsoc2020nb gsoc]$
[jhigh@gsoc2020nb gsoc]$ cat testfile.txt.asc
-----BEGIN PGP MESSAGE-----
wwwEAwEIc39k1V6xVi3SPwEl2ww75Ufjhw7UA0gO/niahHWK50DFHSD1lB10nUyCTgRLe6iS9QZl
av5Nji9BuQFcrqo03I1jG/L9s/4hww==
=x41O
-----END PGP MESSAGE-----
[jhigh@gsoc2020nb gsoc]$
Decryption of symmetric packets requires no changes
[jhigh@gsoc2020nb gsoc]$ netpgp --decrypt testfile.txt.asc
netpgp passphrase:
[jhigh@gsoc2020nb gsoc]$
We added two new flags to support s2k mode 3: '--s2k-mode' and '--s2k-count'. See RFC4880 for details.
[jhigh@gsoc2020nb gsoc]$ netpgp --encrypt --symmetric --s2k-mode=3 --s2k-count=96 testfile.txt
Enter passphrase:
Repeat passphrase:
[jhigh@gsoc2020nb gsoc]$
[jhigh@gsoc2020nb gsoc]$ gpg -d testfile.txt.gpg
gpg: CAST5 encrypted data
gpg: encrypted with 1 passphrase
this
is
a
test
[jhigh@gsoc2020nb gsoc]$
I have been working on the project - Enhance the Syzkaller support for NetBSD, as a part of GSoc’20. Past two months have given me quite an enriching experience, pushing me to comprehend more knowledge on fuzzers. This post would give a peek into the work which has been done so far.
Syzkaller
Syzkaller is a coverage guided fuzzer developed by Google, to fuzz the system calls mainly keeping linux in mind. However, the support has been extended to fuzz the system calls of other Operating Systems as well for eg. Akaros, FreeBSD, NetBSD, OpenBSD, Windows etc.
An automated system Syzbot continuously runs the syzkaller fuzzer on NetBSD kernel and reports the crashes
Increasing syscall support
Initially, the syscall support for Linux as well as FreeBSD was analysed by an automated script. Also coverage of NetBSD syscalls was looked over. This helped us to easily port a few syscalls descriptions for NetBSD. The necessary tweaks were made according to the documentation which describes rules for writing syscall descriptions.
Major groups of syscalls which have been added:
- statfs
- __getlogin
- getsid
- mknod
- utimes
- wait4
- seek
- setitimer
- setpriority
- getrusage
- clock_settime
- nanosleep
- getdents
- acct
- dup
Bugs Found
There were a few bugs reported as a result of adding the descriptions for syscalls of the mentioned syscall families. Few of them are yet to be fixed.Stats
Syscall coverage percent for NetBSD has now increased from nearly 30 to 44.96. Addition of compat syscalls resulted in getting a few new bugs.
Percentage of syscalls covered in few of the other Operating Systems:
- Linux: 82
- FreeBSD: 37
- OpenBSD: 61
Conclusion
In the next phase I would be working on generating the syscall descriptions using automation and adding ioctl device drivers with it’s help.
Atlast, I would like to thank my mentors Cryo, Siddharth and Santhosh for their constant support and guidance.I am also thankful to NetBSD community for being kind to give me this opportunity of having an amazing summer this year.
I have been working on the project - Enhance the Syzkaller support for NetBSD, as a part of GSoc’20. Past two months have given me quite an enriching experience, pushing me to comprehend more knowledge on fuzzers. This post would give a peek into the work which has been done so far.
Syzkaller
Syzkaller is a coverage guided fuzzer developed by Google, to fuzz the system calls mainly keeping linux in mind. However, the support has been extended to fuzz the system calls of other Operating Systems as well for eg. Akaros, FreeBSD, NetBSD, OpenBSD, Windows etc.
An automated system Syzbot continuously runs the syzkaller fuzzer on NetBSD kernel and reports the crashes
Increasing syscall support
Initially, the syscall support for Linux as well as FreeBSD was analysed by an automated script. Also coverage of NetBSD syscalls was looked over. This helped us to easily port a few syscalls descriptions for NetBSD. The necessary tweaks were made according to the documentation which describes rules for writing syscall descriptions.
Major groups of syscalls which have been added:
- statfs
- __getlogin
- getsid
- mknod
- utimes
- wait4
- seek
- setitimer
- setpriority
- getrusage
- clock_settime
- nanosleep
- getdents
- acct
- dup
Bugs Found
There were a few bugs reported as a result of adding the descriptions for syscalls of the mentioned syscall families. Few of them are yet to be fixed.Stats
Syscall coverage percent for NetBSD has now increased from nearly 30 to 44.96. Addition of compat syscalls resulted in getting a few new bugs.
Percentage of syscalls covered in few of the other Operating Systems:
- Linux: 82
- FreeBSD: 37
- OpenBSD: 61
Conclusion
In the next phase I would be working on generating the syscall descriptions using automation and adding ioctl device drivers with it’s help.
Atlast, I would like to thank my mentors Cryo, Siddharth and Santhosh for their constant support and guidance.I am also thankful to NetBSD community for being kind to give me this opportunity of having an amazing summer this year.
I have been working on the project - Enhance the Syzkaller support for NetBSD, as a part of GSoc’20. Past two months have given me quite an enriching experience, pushing me to comprehend more knowledge on fuzzers. This post would give a peek into the work which has been done so far.
Syzkaller
Syzkaller is a coverage guided fuzzer developed by Google, to fuzz the system calls mainly keeping linux in mind. However, the support has been extended to fuzz the system calls of other Operating Systems as well for eg. Akaros, FreeBSD, NetBSD, OpenBSD, Windows etc.
An automated system Syzbot continuously runs the syzkaller fuzzer on NetBSD kernel and reports the crashes
Increasing syscall support
Initially, the syscall support for Linux as well as FreeBSD was analysed by an automated script. Also coverage of NetBSD syscalls was looked over. This helped us to easily port a few syscalls descriptions for NetBSD. The necessary tweaks were made according to the documentation which describes rules for writing syscall descriptions.
Major groups of syscalls which have been added:
- statfs
- __getlogin
- getsid
- mknod
- utimes
- wait4
- seek
- setitimer
- setpriority
- getrusage
- clock_settime
- nanosleep
- getdents
- acct
- dup
Bugs Found
There were a few bugs reported as a result of adding the descriptions for syscalls of the mentioned syscall families. Few of them are yet to be fixed.Stats
Syscall coverage percent for NetBSD has now increased from nearly 30 to 44.96. Addition of compat syscalls resulted in getting a few new bugs.
Percentage of syscalls covered in few of the other Operating Systems:
- Linux: 82
- FreeBSD: 37
- OpenBSD: 61
Conclusion
In the next phase I would be working on generating the syscall descriptions using automation and adding ioctl device drivers with it’s help.
Atlast, I would like to thank my mentors Cryo, Siddharth and Santhosh for their constant support and guidance.I am also thankful to NetBSD community for being kind to give me this opportunity of having an amazing summer this year.
This report was written by Apurva Nandan as part of Google Summer of Code 2020.
My GSoC project under NetBSD involves developing an automated regression and performance test framework for NetBSD that offers reproducible benchmarking results with detailed history and logs across various hardware & architectures.
To achieve this performance testing framework, I am using the Phoronix Test Suite (PTS) which is an open-source, cross-platform automated testing/benchmarking software for Linux, Windows and BSD environments. It allows the creation of new tests using simple XML files and shell scripts and integrates with revision control systems for per-commit regression testing.
About PTS
PTS core
PTS core is the engine of the Phoronix Test Suite and handles the following jobs:
- Regularly updating the test profiles, test suites, and repository index.
- Downloading, compilation, installation and evaluation of test packages.
- Test result management and uploading results and user-defined test-profiles/suites to OpenBenchmarking.org
Test-profiles/Suites & OpenBenchmarking
Test-profiles are light-weight XMLs and shell scripts that allow easy installation of the benchmarking packages and their evaluation. Test-profiles generally contains the following files:
downloads.xml: Stores the links of the benchmarking packages, required additional packages, patches to be applied, etc. with their hash sums.install.sh: Actual compilation, installation, and test evaluation shell script. Also handles the task of applying patches.results-definition.xml: Meta-data to define the information for test result data (e.g. result data units, LIB or HIB, etc.)test-definition.xml: Meta-data to specify test-profile details, such as compatible OS, external dependencies, test arguments, etc.
A simple test-profile C-Ray-1.2.0 can be seen to get an idea of the XMLs and shell scripts.
Test-suites are bundles of related test-profiles or more suites, focusing on a subsystem or certain category of tests e.g. Disk Test Suite, Kernel, CPU suite, etc.
OpenBenchmarking.org serves as a repository for storing test profiles, test suites, hence allowing new/updated tests to be seamlessly obtained via PTS. OpenBenchmarking.org also provides a platform to store the test result data openly and do a comparison between any test results stored in the OpenBenchmarking.org cloud.
Usage
Test installation
The command:
$ phoronix-test-suite install test-profile-name
Fetches the sources of the tests related to the the test-profile in
~/.phoronix-test-suite/installed-tests, applies patches and
carries out compilation of test sources for generating the executables
to run the tests.
Test execution
The command:
$ phoronix-test-suite run test-profile-name
Performs installation of the test (similar to $ phoronix-test-suite
install test-profile-name) followed by exectution the binary from
the compiled sources of the test in
~/.phoronix-test-suite/installed-tests and finally reports the
tests results/outcome to the user and provides an option to upload
results to OpenBenchmarking.org.
Progress in the first phase of GSoC
pkgsrc-wip
The first task performed by me was upgrading the Phoronix Test Suite
from version 8.8 to the latest stable version 9.6.1 in pkgsrc-wip
and is available as
wip/phoronix-test-suite.
You can have a look at the
PTS Changelog
to know about the improvements between these two versions.
Major Commits:
- wip/phoronix-test-suite: update phoronix-test-suite to 9.6.1
- Added required PHP dependencies & removed TODO file
Currently, the PTS upgrade to 9.6.1 is subject to more modifications
and fixes before it gets finally merged to the pkgsrc upstream.
To test the Phoronix Test Suite 9.6.1 on NetBSD till then, you can
setup pkgsrc-wip on your system via:
$ cd /usr/pkgsrc $ git clone git://wip.pkgsrc.org/pkgsrc-wip.git wip
To learn more about pkgsrc-wip please see The pkgsrc-wip project homepage.
After setting up pkgsrc-wip, use the following command for
installation of PTS 9.6.1:
$ cd /usr/pkgsrc/wip/phoronix-test-suite $ make install
If any new build/installation errors are encountered, please document them in wip/phoronix-test-suite/TODO file and/or contact me.
Testing & Debugging
As we now know, having a look over OpenBenchmarking.org’s available
test-profiles section or using
$ phoronix-test-suite list-available-tests,
there are 309 test-profiles available on PTS for Linux, and only 166 of
309 tests have been ported to NetBSD (can be differentiated by a
thunder symbol on the test profile on the website). These 166
test-profiles can be fetch, build and executed on NetBSD using just a
single command $ phoronix-test-suite run test-profile-name. I am
ignoring the graphics ones in both Linux & NetBSD as GPU benchmarking
wasn’t required in the project.
Many of the test profiles available on NetBSD have installation, build,
or runtime errors i.e. they don’t work out of the box using the command
$ phoronix-test-suite run test-profile-name. I ran 90 of these
test profiles on a NetBSD-current x86_64 QEMU VM (took a lot of time
due to the heavy tests) and have reported the status in the sheet:
NetBSD GSoC: Test Porting Progress Report
You can have a look at how a PTS test-profile under action looks like!
Aircrack-NG test-profile demonstration
Aircrack-ng is a tool for assessing WiFi/WLAN network security.
The PTS test-profile is available at: https://openbenchmarking.org/test/pts/aircrack-ng
For further information, complete results can be seen at https://openbenchmarking.org/result/2007116-NI-AIRCRACKN51
AOBench test-profile demonstration
AOBench is a lightweight ambient occlusion renderer, written in C.
The PTS test-profile is available at: https://openbenchmarking.org/test/pts/aircrack-ng
For further information, complete results can be seen at: https://openbenchmarking.org/result/2007148-NI-AOBENCHTE94
Debugging and fixing non-working test-profiles
After testing these test-profiles, I debugged the following build errors in the following test-profiles:
- pts/t-test1-1.0.1:
memalign()not available on NetBSD - pts/coremark-1.0.0: calling bmake instead of gmake
- pts/apache-siege-1.0.4: Missing
signal.hheader - pts/mbw-1.0.0:
mempcpy()not available on NetBSD
Fixes to the above bugs can be found in the sheet or in the GitHub repositories shared in the next paragraph.
The modified test-profiles have been pushed to
pts-test-profiles-dev
and the fix patches to
pts-test-profiles-patches.
These patches are automatically downloaded by the debugged
test-profiles and automatically applied before building of tests. The
steps to install the debugged test-profiles have been added in the
README.md of
pts-test-profiles-dev.
These patches have been added in the downloads.xml of the
modified test-profiles, and hence they get fetched during the test
installation. The install.sh script in these modified
test-profiles applies them on the benchmarking packages if the
operating system is detected as NetBSD, thereby removing the build
errors.
coremark test-profile demonstration
You can have a look at the patched coremark-1.0.0 test-profile under execution:
The patched PTS test-profile is available at: https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/coremark-1.0.0
This is a test of EEMBC CoreMark processor benchmark.
For further information, complete results can be seen at: https://openbenchmarking.org/result/2007148-NI-LOCALCORE64
Porting more test-profiles to NetBSD
Attempts were made to port new test-profiles from Linux to NetBSD, but the major incompatibility issue was missing external dependencies in NetBSD. Hence, this may reduce the number of test-profiles we can actually port, but more sincere efforts will be made in this direction in the next phase.
Future Plans
My next steps would involve porting the non-available tests to NetBSD i.e. the non-graphics ones available on Linux but unavailable on NetBSD, and fix the remaining test-profiles for NetBSD.
After gaining an availability of a decent number of test-profiles on NetBSD, I will use Phoromatic, a remote tests management system for the PTS (allows the automatic scheduling of tests, remote installation of new tests, and the management of multiple test systems) to host the live results of the automated benchmarking framework on benchmark.NetBSD.org, thereby delivering a functional performance and regression testing framework.
This report was written by Apurva Nandan as part of Google Summer of Code 2020.
My GSoC project under NetBSD involves developing an automated regression and performance test framework for NetBSD that offers reproducible benchmarking results with detailed history and logs across various hardware & architectures.
To achieve this performance testing framework, I am using the Phoronix Test Suite (PTS) which is an open-source, cross-platform automated testing/benchmarking software for Linux, Windows and BSD environments. It allows the creation of new tests using simple XML files and shell scripts and integrates with revision control systems for per-commit regression testing.
About PTS
PTS core
PTS core is the engine of the Phoronix Test Suite and handles the following jobs:
- Regularly updating the test profiles, test suites, and repository index.
- Downloading, compilation, installation and evaluation of test packages.
- Test result management and uploading results and user-defined test-profiles/suites to OpenBenchmarking.org
Test-profiles/Suites & OpenBenchmarking
Test-profiles are light-weight XMLs and shell scripts that allow easy installation of the benchmarking packages and their evaluation. Test-profiles generally contains the following files:
downloads.xml: Stores the links of the benchmarking packages, required additional packages, patches to be applied, etc. with their hash sums.install.sh: Actual compilation, installation, and test evaluation shell script. Also handles the task of applying patches.results-definition.xml: Meta-data to define the information for test result data (e.g. result data units, LIB or HIB, etc.)test-definition.xml: Meta-data to specify test-profile details, such as compatible OS, external dependencies, test arguments, etc.
A simple test-profile C-Ray-1.2.0 can be seen to get an idea of the XMLs and shell scripts.
Test-suites are bundles of related test-profiles or more suites, focusing on a subsystem or certain category of tests e.g. Disk Test Suite, Kernel, CPU suite, etc.
OpenBenchmarking.org serves as a repository for storing test profiles, test suites, hence allowing new/updated tests to be seamlessly obtained via PTS. OpenBenchmarking.org also provides a platform to store the test result data openly and do a comparison between any test results stored in the OpenBenchmarking.org cloud.
Usage
Test installation
The command:
$ phoronix-test-suite install test-profile-name
Fetches the sources of the tests related to the the test-profile in
~/.phoronix-test-suite/installed-tests, applies patches and
carries out compilation of test sources for generating the executables
to run the tests.
Test execution
The command:
$ phoronix-test-suite run test-profile-name
Performs installation of the test (similar to $ phoronix-test-suite
install test-profile-name) followed by exectution the binary from
the compiled sources of the test in
~/.phoronix-test-suite/installed-tests and finally reports the
tests results/outcome to the user and provides an option to upload
results to OpenBenchmarking.org.
Progress in the first phase of GSoC
pkgsrc-wip
The first task performed by me was upgrading the Phoronix Test Suite
from version 8.8 to the latest stable version 9.6.1 in pkgsrc-wip
and is available as
wip/phoronix-test-suite.
You can have a look at the
PTS Changelog
to know about the improvements between these two versions.
Major Commits:
- wip/phoronix-test-suite: update phoronix-test-suite to 9.6.1
- Added required PHP dependencies & removed TODO file
Currently, the PTS upgrade to 9.6.1 is subject to more modifications
and fixes before it gets finally merged to the pkgsrc upstream.
To test the Phoronix Test Suite 9.6.1 on NetBSD till then, you can
setup pkgsrc-wip on your system via:
$ cd /usr/pkgsrc $ git clone git://wip.pkgsrc.org/pkgsrc-wip.git wip
To learn more about pkgsrc-wip please see The pkgsrc-wip project homepage.
After setting up pkgsrc-wip, use the following command for
installation of PTS 9.6.1:
$ cd /usr/pkgsrc/wip/phoronix-test-suite $ make install
If any new build/installation errors are encountered, please document them in wip/phoronix-test-suite/TODO file and/or contact me.
Testing & Debugging
As we now know, having a look over OpenBenchmarking.org’s available
test-profiles section or using
$ phoronix-test-suite list-available-tests,
there are 309 test-profiles available on PTS for Linux, and only 166 of
309 tests have been ported to NetBSD (can be differentiated by a
thunder symbol on the test profile on the website). These 166
test-profiles can be fetch, build and executed on NetBSD using just a
single command $ phoronix-test-suite run test-profile-name. I am
ignoring the graphics ones in both Linux & NetBSD as GPU benchmarking
wasn’t required in the project.
Many of the test profiles available on NetBSD have installation, build,
or runtime errors i.e. they don’t work out of the box using the command
$ phoronix-test-suite run test-profile-name. I ran 90 of these
test profiles on a NetBSD-current x86_64 QEMU VM (took a lot of time
due to the heavy tests) and have reported the status in the sheet:
NetBSD GSoC: Test Porting Progress Report
You can have a look at how a PTS test-profile under action looks like!
Aircrack-NG test-profile demonstration
Aircrack-ng is a tool for assessing WiFi/WLAN network security.
The PTS test-profile is available at: https://openbenchmarking.org/test/pts/aircrack-ng
For further information, complete results can be seen at https://openbenchmarking.org/result/2007116-NI-AIRCRACKN51
AOBench test-profile demonstration
AOBench is a lightweight ambient occlusion renderer, written in C.
The PTS test-profile is available at: https://openbenchmarking.org/test/pts/aircrack-ng
For further information, complete results can be seen at: https://openbenchmarking.org/result/2007148-NI-AOBENCHTE94
Debugging and fixing non-working test-profiles
After testing these test-profiles, I debugged the following build errors in the following test-profiles:
- pts/t-test1-1.0.1:
memalign()not available on NetBSD - pts/coremark-1.0.0: calling bmake instead of gmake
- pts/apache-siege-1.0.4: Missing
signal.hheader - pts/mbw-1.0.0:
mempcpy()not available on NetBSD
Fixes to the above bugs can be found in the sheet or in the GitHub repositories shared in the next paragraph.
The modified test-profiles have been pushed to
pts-test-profiles-dev
and the fix patches to
pts-test-profiles-patches.
These patches are automatically downloaded by the debugged
test-profiles and automatically applied before building of tests. The
steps to install the debugged test-profiles have been added in the
README.md of
pts-test-profiles-dev.
These patches have been added in the downloads.xml of the
modified test-profiles, and hence they get fetched during the test
installation. The install.sh script in these modified
test-profiles applies them on the benchmarking packages if the
operating system is detected as NetBSD, thereby removing the build
errors.
coremark test-profile demonstration
You can have a look at the patched coremark-1.0.0 test-profile under execution:
The patched PTS test-profile is available at: https://github.com/apurvanandan1997/pts-test-profiles-dev/tree/master/coremark-1.0.0
This is a test of EEMBC CoreMark processor benchmark.
For further information, complete results can be seen at: https://openbenchmarking.org/result/2007148-NI-LOCALCORE64
Porting more test-profiles to NetBSD
Attempts were made to port new test-profiles from Linux to NetBSD, but the major incompatibility issue was missing external dependencies in NetBSD. Hence, this may reduce the number of test-profiles we can actually port, but more sincere efforts will be made in this direction in the next phase.
Future Plans
My next steps would involve porting the non-available tests to NetBSD i.e. the non-graphics ones available on Linux but unavailable on NetBSD, and fix the remaining test-profiles for NetBSD.
After gaining an availability of a decent number of test-profiles on NetBSD, I will use Phoromatic, a remote tests management system for the PTS (allows the automatic scheduling of tests, remote installation of new tests, and the management of multiple test systems) to host the live results of the automated benchmarking framework on benchmark.NetBSD.org, thereby delivering a functional performance and regression testing framework.