During the past month I have been working on coverage of various corner cases in the signal subsystem in the kernel. I have also spent some time on improvements in the land of sanitizers. As a mentor I was able to, thanks to the full-time focus on NetBSD work, actively help three Google Summer of Code students. Not every question would be answered by myself without code reading but at least I am available for active collaboration, especially when it's to improve code that I have already authored, like sanitizers. At the end of the month we have managed to catch two uninitialized memory reads in the top(1) utility, using the Memory Sanitizer feature and rebuilt part of the basesystem (i.e. library dependencies: libterminfo, libkvm, libutil) with dedicated sanitization flags.

ptrace(2) and related distribution changes

I am actively working on handling of processes, forks/vforks, signals and threads that is reliable and fully functional under a debugger. This is a process and the situation is actively improving. For the end-user this means that we are achieving the state when a developer will be able to trace an application like Firefox using modern tools and save time detecting the issues quickly.

I am using the Test-Driven Development approach in my work. I keep extending the Automatic Test Framework with new tests, covering sets of scenarios handled by debuggers and related code. This is followed by kernel fixes. Thanks to the tests, I can more confidently introduce changes to critical routines inside the Operating System, test new changes quickly for regressions and keep covering new verifiable scenarios.

Titles of the merged commits with the main tree of NetBSD:

• Remove an element from struct emul: e_tracesig.

  e_tracesig used to be implemented for Darwin compat. Nowadays the Darwin compatib[i]lity layer is gone and there are no other users.

• Refactoring of can_we_set_dbregs() in ATF ptrace(2) tests. Push this auxiliary function to all ports.
• Add a new ptrace(2) ATF exploit for: CVE-2018-8897 (POP SS debug exception).
• Correct handling of: vfork(2) + PT_TRACE_ME + raise(2).
• Add a new ATF ptrace(2) test: traceme_vfork_breakpoint.
• Improve the description of traceme_vfork_raise in ATF ptrace(2) tests.
• Add a new ATF ptrace(2) test: traceme_vfork_exec.
• Improve the description of traceme_vfork_breakpoint (ATF ptrace(2) test).
• Add extra asserts in three ATF ptrace(2) tests.

  In traceme* tests after validate_status_stopped() include additional check the verify the received signal with PT_GET_SIGINFO.

• Correct assert in ATF t_zombie test.
• Add new ATF tests: t_fork and t_vfork.
• Stop masking SIGSTOP in a vfork(2)ed child.
• Stop masking raise(SIGSTOP) in a vfork(2)ed child that called PT_TRACE_ME.
• Add new auxiliary functions in t_ptrace_wait.h

  New functions:

  • FORKEE_ASSERT_NEQ()
  • await_stopped_child()
• Enable traceme_vfork_raise2 in ATF ptrace(2) tests.

  raise(SIGSTOP) is now handled correctly by the kernel, in a child that vfork(2)ed and called PT_TRACE_ME.

• Cover SIGTSTP, SIGTTIN and SIGTTOU in traceme_vfork_raise ATF tests.
• Note in vfork(2) that SIGTSTP is masked.
• Fix and enable traceme_signal_nohandler2 in ATF ptrace(2) tests.
• Make stopsigmask a non-static symbol now as it's used in ptrace(2) code.
• Refactor and enable the signal3 ATF ptrace(2) test

  Adapt the test to be independent from the software breakpoint trap behavior, whether the Program Counter is moved or not. Just kill the process after catching the expected signal, instead of pretending to resume it.

• Add new ATF test: t_trapsignal:trap_ignore.
• Minor update to signal(7)

  Note that SIGCHLD is not just a child exit signal. Note that SIGIOT is PDP-11 specific signal.

• Minor improvement in sigaction(2)

  Note that SIGCHLD covers process continued event.

• Extend ATF tests in t_trapsignal.sh to verify software breakpoint traps.
• Add new ATF ptrace(2) tests: traceme_sendsignal_{masked,ignored}[1-3].
• Define PTRACE_BREAKPOINT_ASM for i386 in the MD part of .
• Refactor the attach[1-8] and race1 ATF t_ptrace_wait* tests.
• Cherry-pick upstream patch for internal_mmap() in GCC sanitizers.
• Cherry-pick upstream patch for internal_mmap() in GCC(.old) sanitizers
• Add new auxiliary functions in ATF ptrace(2) tests

  Introduce:

  • trigger_trap()
  • trigger_segv()
  • trigger_ill()
  • trigger_fpe()
  • trigger_bus()
• Extend traceme_vfork_breakpoint in ATF ptrace(2) tests for more scenarios

  Added tests:

  • traceme_vfork_crash_trap
  • traceme_vfork_crash_segv (renamed from traceme_vfork_breakpoint)
  • traceme_vfork_crash_ill (disabled)
  • traceme_vfork_crash_fpe
  • traceme_vfork_crash_bus
• Merge the eventmask[1-6] ATF ptrace(2) tests into a shared function body.
• Introduce can_we_write_to_text() to ATF ptrace(2) tests

  The purpose of this function is to detect whether a tracer can write to the .text section of its tracee.

• Refactor the PT_WRITE*/PT_READ* and PIOD_* ATF ptrace(2) tests.
• Handle vm.maxaddress in compat_netbsd32(8).
• Port the CVE 2018-8897 mitigation to i386 ATF ptrace(2) tests.
• Fix sysctl(3):vm.minaddress in compat_netbsd32(8).
• Fix ATF ptrace(2) bytes_transfer_piod_read_auxv test.
• Handle FPE and BUS scenarios in the ATF t_trapsignal tests.
• Try to fool $CC harder in ATF ptrace(2) tests in trigger_fpe().
• Correct reporting SIGTRAP TRAP_EXEC when SIGTRAP is masked.
• Correct the t_ptrace_wait*:signal5 ATF test case.

This functionality now works.

• Add new ATF ptrace(2) tests verifying crash signal handling.
• Harden PT_ATTACH in ptrace(2).

  Don't allow to PT_ATTACH from a vfork(2)ed child (before exec(3)/_exit(3)) to its parent. Return error with EPERM errno.

  This scenario does not have a purpose and there is no clear picture how to route signals.

• Simplify comparison of two processes

  No need to check p_pid to compare whether two processes are the same.

LLVM compiler-rt features

I have helped the GSoC student to prepare for LLVM libfuzzer integration with the NetBSD base system. We have managed to get down to the following results for the test target in the upstream repository:

$ check-fuzzer-default

  Expected Passes    : 105
  Unsupported Tests  : 8
  Unexpected Failures: 2

$ check-fuzzer

  Expected Passes    : 105
  Unsupported Tests  : 8
  Unexpected Failures: 2

$ check-fuzzer-unit

  Expected Passes    : 35

The remaining two failures appear to be false positives and specific to the differences between the NetBSD setup difference and other supported Operating Systems (including Linux). I have decided not to investigate them and instead to move on to more urgent tasks.

While there, I have been working on restoring a good state to userland LLVM sanitizers in the upstream repository, in order ship them in the NetBSD distribution along with the libfuzzer utility.

A number of patches were merged upstream:

• LLVM: Register NetBSD/i386 in AddressSanitizer.cpp
• Clang: Permit -fxray-instrument for NetBSD/amd64
• Clang: Support XRay in the NetBSD driver
• compiler-rt: Remove dead sanitizer_procmaps_freebsd.cc
• compiler-rt: wrong usages of sem_open in the libFuzzer (patch by Yang Zheng, the GSoC student)
• compiler-rt: Register NetBSD/i386 in asan_mapping.h
• compiler-rt: Setup ORIGIN/NetBSD option in sanitizer tests
• compiler-rt: Enable SANITIZER_INTERCEPTOR_HOOKS for NetBSD

There is also at least a single pending upstream patch that is worth to note: Introduce CheckASLR() in sanitizers

At least the ASan, MSan, TSan sanitizers require disabled ASLR on a NetBSD.

Introduce a generic CheckASLR() routine, that implements a check for the
current process. This flag depends on the global or per-process settings.

There is no simple way to disable ASLR in the build process from the
level of a sanitizer or during the runtime execution.

With ASLR enabled sanitizers that operate over the process virtual address
space can misbehave usually breaking with cryptic messages.

This check is dummy for !NetBSD.

The current results for test targets in the compiler-rt features are as follows:

$ make check-builtins

  Expected Passes    : 343
  Expected Failures  : 4
  Unsupported Tests  : 36
  Unexpected Failures: 5

$ check-interception

-- Testing: 0 tests, 0 threads --

$ check-lsan

  Expected Passes    : 6
  Unsupported Tests  : 60
  Unexpected Failures: 106

$ check-ubsan

  Expected Passes    : 229
  Expected Failures  : 1
  Unsupported Tests  : 32
  Unexpected Failures: 2

$ check-cfi

  Unsupported Tests  : 232

$ check-cfi-and-supported

BaseException: Tests unsupported

$ make check-sanitizer

  Expected Passes    : 576
  Expected Failures  : 13
  Unsupported Tests  : 206
  Unexpected Failures: 31

$ check-asan

  Expected Passes    : 852
  Expected Failures  : 4
  Unsupported Tests  : 440
  Unexpected Failures: 16

$ check-asan-dynamic

  Expected Passes    : 394
  Expected Failures  : 3
  Unsupported Tests  : 440
  Unexpected Passes  : 1
  Unexpected Failures: 222

$ check-msan

  Expected Passes    : 102
  Expected Failures  : 1
  Unsupported Tests  : 30
  Unexpected Failures: 4

$ check-tsan

  Expected Passes    : 288
  Expected Failures  : 1
  Unsupported Tests  : 84
  Unexpected Failures: 8

$ check-safestack

  Expected Passes    : 7
  Unsupported Tests  : 1

$ check-scudo

  Expected Passes    : 14
  Unexpected Failures: 28

$ check-ubsan-minimal

  Expected Passes    : 6
  Unsupported Tests  : 2

$ check-profile

  Unsupported Tests  : 116

$ check-xray

  Expected Passes    : 21
  Unsupported Tests  : 1
  Unexpected Failures: 21

$ check-shadowcallstack

  Unsupported Tests  : 4

Sanitization of userland and the kernel

I am helping to setup the process for shipping a NetBSD userland that is prebuilt with a desired sanitizer. This involves consulting the Google Summer of Code student, fixing known issues, reviewing patches etc.

There were two new uninitialized memory read bugs detected in the top(1) program:

Fix unitialized signal mask passed to sigaction(2) in top(1)

Detected with Memory Sanitizer during the integration of sanitizers with
the NetBSD basesystem.

Reported by <Yang Zheng>

Fix read of uni[ni]tialized array elements in top(1)

The cp_old array is allocated with malloc(3) and its pointer is passed to
percentages64().

In this function there happens a calculation of total_change, which value
depends on the value inside the unitialized cp_old[] array.

==26662==WARNING: MemorySanitizer: use-of-uninitialized-value
#0 0x268a2c in percentages64 /usr/src/external/bsd/top/bin/../dist/machine/m_netbsd.c:1341:6
#1 0x26748b in get_system_info /usr/src/external/bsd/top/bin/../dist/machine/m_netbsd.c:478:6
#2 0x25518e in do_display /usr/src/external/bsd/top/bin/../dist/top.c:507:5
#3 0x253038 in main /usr/src/external/bsd/top/bin/../dist/top.c:975:2
#4 0x21cad1 in ___start (/usr/bin/top+0x1cad1)
SUMMARY: MemorySanitizer: use-of-uninitialized-value /usr/src/external/bsd/top/bin/../dist/machine/m_netbsd.c:1341:6 in percentages64
Exiting

Fix this issue by chang[]ing malloc(3) with calloc(3).

Detected with Memory Sanitizer during the integration of sanitizers with
the NetBSD basesystem.

Reported by <Yang Zheng>

As similar process happens with two kernel sanitizer GSoC tasks: kernel-ubsan and kernel-asan.

Thanks to the involvement to The NetBSD Foundation tasks, I can be reachable for students (although not always in all cases) for active feedback and collaboration.

Summary

The number of ATF ptrace(2) tests cases has been significantly incremented, however there is still a substantial amount of work to be done and a number of serious bugs to be resolved.

With fixes and addition of new test cases, as of today we are passing 1,206 (last month: 961) ptrace(2) tests and skipping 1 (out of 1,256 total; last month: 1,018 total). No counted here tests that appeared outside the ptrace(2) context.

Plan for the next milestone

Cover with regression tests remaining elementary scenarios of handling crash signals. Fix known bugs in the NetBSD kernel.

Follow up the process with the remaining fork(2) and vfork(2) scenarios.

This work was sponsored by The NetBSD Foundation.

The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL, and chip in what you can:

http://netbsd.org/donations/#how-to-donate

I like to use CLI email clients (mutt). This by itself is not unusual, but I happen to do this while speaking a language written right-to-left, Hebrew.
Decent bidi support in CLI tools is rare, so my impression is that very few people do this.

In the dark ages before Unicode, Hebrew used its own encodings which allowed typing both Latin and Hebrew letters: Windows-1255, ISO-8859-8.
I speculate that people initially expected input to be written in reverse order (aka "visual order"), assuming that everything will display text left to right.

When people wanted to use e-mail, they decided they'll write a line stating the charset encoding as others do, and use quoted-printable or base64 to avoid the content being mangled by clueless servers (8BITMIME wasn't around then).

But then they thought about bidi, and realized that writing in reverse isn't that great when you can have some bidi support. I've yet to write a bidi algorithm, but I suspect it makes line-wrapping illogical.

To avoid conflicts with existing emails, they decided on a separate encoding for the purpose of conveying that the information isn't in reverse: iso-8859-8-i: the content is in logical order, and Hebrew is assumed to be rtl.
iso-8859-8-e: the text direction is explicit using control codes.

The latter is a neat idea, but hasn't caught on. Now it's common to assume logical order, and even iso-8859-8 might be in that format.
While defining this, they've also done the same for Arabic (iso-8859-6).

This is a discussion that should've been part of the past - Unicode is now a thing, and I can send messages that contain Hebrew, Arabic, Chinese, English - without flipping back and forth in encoding (if that was ever even possible?), and out of the box! Never a need to enable support for specifying charset. Unicode has a detailed algorithm for handling bidi.
Unicode is love. Unicode is life. Use Unicode.
But I recently was looking for work, and HR's presumed Microsoft Outlook MUA did not use Unicode.

One of the emails I got was encoded as iso-8859-8-i.
It turns out, my MUA setup cannot handle this charset. It ended up looking like \344 things, and the subject as boxes.
mail is a plaintext format with extensions hacked into it, so you can view the raw content as a file. I used 'e' on mutt to open it:

Subject: =?iso-8859-8-i?B?base64stuff

(The magical 'encode this in a different way' for email subjects)

Content-Type: text/plain; charset="iso-8859-8-i"
Content-Transfer-Encoding: quoted-printable

So this is an iso-8859-8-i file.

OK, let's just read this file. I've got python.
I saved the file, which looked like this in its raw format:

=EE=E0=E9=E4

Or quoted-printable. Gotta turn that into raw data, then convert ISO-8859-8 to UTF-8.

import quopri
import sys
rawmsg = sys.stdin.read()
notutf8msg = quopri.decodestring(rawmsg)
utf8msg = notutf8msg.decode('iso-8859-8')
print(utf8msg)

Cool. I can read the message. I even discover 'fribidi' isn't just a library, but also provides a command I can pipe this into and see nicely-formatted Hebrew even without using weirdo terminal emulators.

But let's not leave bugs like that lurking around. It is my duty as an RTL warrior to fix it.

One of the perks to using pkgsrc/netbsd and open source is that I can immediately look at mutt's source code. I knew it could handle iso-8859-8, so that's what I looked for.

The amount of results (combined with experience) quickly suggested that the encoding is handled by the OS, netbsd in this case.
NetBSD didn't know about iso-8859-8-i.

Experience meant I knew to look in either src/lib/ (wasn't there) or src/share/ for 'data used by things'. I've looked for 'iso-8859-8' to see if it appears anywhere, and found it. It was good to see that NetBSD does appear to have a way to alias charsets as being equivalent, and I added iso-8859-8-i here, and did a full build because I didn't know how the files are used.

Testing locally, I could now read the email with mutt! But what about replying?
I have a weird email setup again. I had a hard time setting up a remote POP/IMAP thing, so I ssh to sdf.org and email from there. And I can't change their libc or install.
Hoping to just elide all the corrupted characters and reply with UTF-8 was too optimistic - mutt wanted to reply in the original encoding, and again could not handle it properly.

Well, I'll just put in my updated libc, and LD_PRELOAD it, then!
Except, after ktracing it (via 'ktruss -i mutt |grep esdb'), it turns out that it opens a file in /usr/share/i18n/ to figure out charset aliases.
I'll need to tell it to look elsewhere I can modify.
I've edited out paths.h, which is where the lookup path is stored, changed it to my home on sdf.org, and then built myself a fresh libc.
(It was during this I realized I could've just edited the email to say it's iso-8859-8, rather than iso-8859-8-i)

A few minor setbacks, and I could finally reply to the email, saying that yes, I will show up to the job interview.

I leave you with this tidbit from the RFC announcing these encodings and that finally, emails in Hebrew are possible:
"Within Israel there are in excess of 40 Listserv lists which will now start using Hebrew for part of their conversations."
Hurray!

Prepared by Yang Zheng (tomsun.0.7 AT Gmail DOT com) as part of GSoC 2018

During the Google Summer of Code 2018, I'm working on the project of integrating libFuzzer for the userland applications. The libFuzzer is a fuzzing engine based on the coverage information provided by the SanitizerCoverage in LLVM. It can repeatedly generate mutations of input data and test them until it finds the potential bugs. In this post, I'm going to share what I have done in the first month of this summer.

For the first month, I mainly tried to apply the sanitizers to the userland applications. Sanitizers (such as MemorySanitizer, AddressSanitizer, and etc.) are helpful to the fuzzing process because they can detect various types of run-time errors like uninitialized reads, out-of-bounds accesses, use-after-free and so on. I tried to apply MemorySanitizer as a start and there were three steps to finish this:

1. Import new version LLVM as an external toolchain
2. Add new interceptors for userland applications
3. Enable MemorySanitizer for userland applications and test them

Compile New Version LLVM Statically with EXTERNAL_TOOLCHAIN

Using a new version of LLVM toolchain is necessary because the LLVM in NetBSD trunk is old and there are some changes in the new version. However, updating the toolchain in the src will introduce extra work for this project, so we decided to use the EXTERNAL_TOOLCHAIN parameter provided by NetBSD to work with the new version.

During this period, I chose to use a pure-LLVM userland to avoid potential problems. This means that we should replace the libc++ instead of libstdc++ library for the userland programs. As a result, I used -DSANITIZER_CXX_ABI=libc++ and -DCLANG_DEFAULT_CXX_STDLIB=libc++ flags to eliminate some compilation errors while compiling the LLVM toolchain.

Another compiling issue is related to the sanitizers. Whenever there is failed check with sanitizers, the program will abort with backtrace information like this:

    ==15299==WARNING: MemorySanitizer: use-of-uninitialized-value
        #0 0x41c837 in main /home/zhengy/free.c:6:3
        #1 0x41c580 in ___start (//./a.out+0x41c580)

    SUMMARY: MemorySanitizer: use-of-uninitialized-value /home/zhengy/free.c:6:3 in main
    Exiting
  

The backtrace is generated with the support of llvm-symbolizer. However, if we compile some dynamic libraries, which are needed by llvm-symbolizer, with sanitizers (because some userland programs with sanitizers also need them), then it will not available for generating a readable backtrace anymore:

    ==1623==WARNING: MemorySanitizer: use-of-uninitialized-value
        #0 0x41c837  (//./a.out+0x41c837)
        #1 0x41c580  (//./a.out+0x41c580)

    SUMMARY: MemorySanitizer: use-of-uninitialized-value (//./a.out+0x41c837)
    Exiting
  

So, to remove the dependencies of the sanitized dynamic libraries for llvm-symbolizer and other LLVM tools, we chose to compile the whole LLVM toolchain statically. For this purpose, we found that the static building behavior of LLVM on NetBSD is not workable, so we need to do some subtle modification to the cmake file. But this modification still needs some correctness confirmation from the LLVM community.

After all of these preparations, I wrote a shell script to automatically do the jobs of preparing external LLVM toolchains, compiling the NetBSD from source and finally generate a chroot(8)-able environment to work with sanitizers and libFuzzer.

With this environment, I first tried to run the test cases from both the LLVM and the NetBSD. For the LLVM part, I found that some libFuzzer cases were not working. But finally, we found that this resulted from the improper usages of sem_open(3) interface in the libFuzzer and so I submitted a patch to fix this.

For the NetBSD part, it worked well with the existing ATF(7) test cases for the AddressSanitizer and UndefinedBehaviorSanitizer. To test the MemorySanitizer, ThreadSanitizer, and libFuzzer, I added some test cases for them.

Add New Interceptors

Some libraries (such as libc, libm, and libpthread) and syscalls cannot be applied properly with sanitizers. This will introduce some troubles because we will lack information with these unsanitized interfaces. Fortunately, sanitizers can provide wrappers, namely interceptors, for these interfaces to manually provide some information. However, the set of interceptors is quite incomplete and thus need some effort to add some unsanitized functions needed by userland applications. As a summary, I added interceptors for the following interfaces:

• strtonum(3) family: strtonum(3), strtoi(3), strtou(3)
• vis(3) family: vis(3), nvis(3), strvis(3) and etc.
• getmntinfo(3)
• puts(3), fputs(3)
• Hash interfaces: sha1(3), md2(3), md4(3), md5(3), rmd160(3) and sha2(3)
• getvfsstat(2)
• nl_langinfo(3)
• fparseln(3)
• unvis(3) family: unvis(3), strunvis(3) and etc.
• statvfs(2) family: statvfs(2), fstatvfs(2) and etc.
• mount(2) and unmount(2)
• fseek(3) family: fseek(3), ftell(3), rewind(3) and etc.
• cdbr(3) family: cdbr_open(3), cdbr_get(3), cdbr_find(3) and etc.
• setvbuf(3) family: setbuf(3), setbuffer(3), setlinebuf(3), setvbuf(3)
• mi_vector_hash(3)

Most of these interceptors are easy to add, we only need to leverage the interceptor interfaces provided by the compiler-rt and do the pre- and post- function call check. As an example, I choose the interceptor of strvis(3) to illustrate the implementation:

    INTERCEPTOR(int, strvis, char *dst, const char *src, int flag) {
      void *ctx;
      COMMON_INTERCEPTOR_ENTER(ctx, strvis, dst, src, flag);
      if (src)
        COMMON_INTERCEPTOR_READ_RANGE(ctx, src, REAL(strlen)(src) + 1);
      int len = REAL(strvis)(dst, src, flag);
      if (dst)
        COMMON_INTERCEPTOR_WRITE_RANGE(ctx, dst, len + 1);
      return len;
    }
  

The strvis(3) interface will transform the representation of string stored in src and then return it with dst. So, its interceptor wants to tell the sanitizers two messages:

1. strvis(3) will read the string in src (COMMON_INTERCEPTOR_READ_RANGE interface)
2. strvis(3) will write a string to dst (COMMON_INTERCEPTOR_WRITE_RANGE interface)

So, with interceptors, the sanitizers can obtain information of unsanitized interfaces. There are three unsolved issues with interceptors:

1. Interceptors with FILE type: the FILE type is implemented as a structure and contains some pointers inside. This means that we should check these pointers one by one in the interceptors. However, the FILE type is common among different OSs and their implementations vary a lot. So, for different OSs, we should write different conditions. What's worse, there are some interceptors (such as fopen) implemented by others skipping the checks for FILE. This will introduce some incompatible problems if we enforce the check with other interfaces (like fputs). For example, the fopen is the interface to initialize the FILE type, if we skip marking the returned FILE pointer as initialized (with COMMON_INTERCEPTOR_WRITE_RANGE), we will get an error in the interceptor of fputs after we enforce the check of this pointer (with COMMON_INTERCEPTOR_READ_RANGE).
2. mount(2) interface: The mount(2) interface requires data parameter for different file systems. This parameter can be different types, such as struct ufs_args, struct nfs_args and so on. These types usually contain pointers, so we need to check them one by one. However, there are around 34 different struct xxx_args types in NetBSD, so it will be quite hard to add and maintain them in compiler-rt repository.
3. getchar(3) and putchar(3) family interfaces: these interfaces will be defined by macros with some compiler conditions, so their implementation will be complicated.

Enable the Sanitizers for the Userland with MKSANITIZER

After adding interceptors, we can then enable the sanitizers for userland applications. To ship the sanitizers to the user, Christos Zoulas prepared the MKSANITIZER framework, dedicated for building the whole sanitizer userland with a dedicated sanitizer (including UndefinedBehaviorSanitizer, Control Flow Integrity, MemorySanitizer, ThreadSanitizer, SafeStack, LeakSanitizer and etc).

Based on this framework, Kamil Rytarowski used the NetBSD building parameters like MKSANITIZER=yes USE_SANITIZER=undefined HAVE_LLVM=yes and managed to enable the UndefinedBehaviorSanitizer option for the whole userland. There is the ongoing effort on upstreaming local patches, fixing detected bugs. It is planned to follow up this with the remaining sanitizer options.

I also tried to enable the MemorySanitizer for the userland programs and here is the result. If you have any insights or suggestions, please feel free to comment on it. Applying the MemorySanitizer option also helped to improve the interceptors and integrate MKSANITIZER. The MemorySanitizer is sensitive to the interceptor issues and so actually this job was twisted with the process of adding and improving the interceptors. With the MemorySanitizer, I also find out two bugs with top(1) program. You can refer to this post to learn about it.

There are also some unsolved issues with some applications. As shown in the sheet, I divide them into five categories:

1. DEADLYSIGNAL: mainly happening when sending CTRL-C to programs
2. IOCTL: ioctl(2)-related errors
3. GETC, PUTC, FFLUSH: stdio(3)-related errors
4. REALLOC: realloc(3)-related errors
5. Compilation errors: conflict symbols between programs and base libraries

The challenging of GETC, PUTC, FFLUSH category has been mentioned above, it mainly results from lacking the interceptors of these interfaces. The other categories are still remained to be investigated.

Summary

In the last month, I have a good start of working with LLVM and NetBSD and successfully build some userland programs with MemorySanitizer. All of these jobs mentioned above are based on the forked repositories instead of the official ones. If you have interests in them, please refer to these repositories: NetBSD source, pkgsrc-wip, LLVM, clang, and compiler-rt. Next, I will switch to the integration work of libFuzzer and try to run some programs as a trial.

Last but not least, I want to thank my mentors, Christos Zoulas and Kamil Rytarowski, they help me a lot with so many good suggestions and assistance. I also want to thank Matthew Green and Joerg Sonnenberger for their help with LLVM-related suggestions. Finally, thanks to Google to give me a good chance to work with NetBSD community.

Prepared by Siddharth Muralee (@Tr3x__) as part of GSoC 2018.

It's been a fun couple of weeks since I started working on the Kernel Address Sanitizer (KASan) project with NetBSD. I have learned a lot during this period. It's been pretty amazing. This is a report on the work I have done prior to the first evaluation period.

What is an Address Sanitizer?

The Address Sanitizer (ASan) is an open source tool that was developed by Google to detect memory corruption bugs such as buffer overflows or access to dangling pointers (use after free). Its a part of the toolset that Google has which includes an Undefined Behaviour Sanitizer (UBSan), a Thread Sanitizer (TSan) and a Leak Sanitizer (LSan).

On adding the feature to NetBSD it would be possible to add build the kernel with ASan and then use it to find memory corruption bugs.

Testing ASan in the User Space

My first step was to testing whether ASan had been implemented in the NetBSD userspace. I wrote a couple of ATF regression tests for checking whether ASan worked in the userspace for C and C++ compilers and also whether manual poisoning would work.

This allowed me to get familiar with the ATF testing framework that NetBSD.

Added a couple of Kernel Modules

I was asked to add a set of example kernel modules to the kernel. I added an example module to show how to make a /dev module multiprocessor safe and to add a node in the sysctl tree.

Reading about UVM

My next task was to get familiar with the UVM (virtual memory system of NetBSD). I read through a 1998 dissertation by Dr. Chuck Cranor. I published a blog article containing my scratch notes on reading the article.

Adding an option to compile the kernel with KASan

Finally, I had to build the kernel with the KASan stubs (Dummy functions so that the build would be working). I added a configuration file which can be used to build the kernel with KASAN. I also published a blog post regarding how to do the same.

Summary

In short, I am pretty excited to move forward with the project. The community has been supportive and helpful all the way.

I would like to thank my mentor, Kamil Rytarowski who was always ready to dive deep into code and help whenever required. I also want to thank Cherry Mathews for helping clear up doubts related to UVM.

Prepared by Harry Pantazis (IRC:luserx0, Mail:luserx0 AT gmail DOT com) as part of GSoC 2018.

For GSoC '18, I'm working on the Kernel Undefined Behavior Sanitizer (KUBSAN) project for the integration of Undefined Behavior regression testing on the amd64 kernel. This article summarizes what has been done up to this point (Phase 1 Evaluation), future goals and a brief introduction to Undefined Behavior.

So, first things first, let's get started. The mailing list project presentation

What is Undefined Behavior?

For Turing-complete languages we cannot reliably decide offline whether a program has the potential to execute an error; we just have to run it and see. DUH!

Undefined Behavior in C is basically what the ANSI standard leaves unexplained. Code containing Undefined Behavior is ANSI C compatible. It follows all the rules explained in the standard and causes real trouble. In programming terms, it involves all the possible functionalities C code can run. It's whatever the compiler doesn't moan about, but when run it causes run-time bugs, hard to locate.

The C FAQ defines "Undefined Behavior" like this:

Anything at all can happen; the Standard imposes no requirements. The program may fail to compile, or it may execute incorrectly (either crashing or silently generating incorrect results), or it may fortuitously do exactly what the programmer intended.

A brief explanation of what is classifed as UB and some real case scenarios

A great blog post explaining more than mere mortals might need

The important and scary thing to realize is that just about *any* optimization based on undefined behavior can start being triggered on buggy code at any time in the future. Inlining, loop unrolling, memory promotion and other optimizations will and a significant part of their reason for existing is to expose secondary optimizations like the ones above.

Solution: Make a UB Sanitizer

What we can do to find undefined behavior errors in our code, is creating a Sanitizer. Hopefully both CLang and GCC have taken care of such "dream" tools, covering the majority of undefined behavior cases in a very nice manner. They allow us to simply parse the -fsanitize=undefined option when we build our code and the compiler "spits out" simple warnings for us to see. The CLang supported flags (same as GCC's but they don't have such extensive explanation docs).

Adding ATF Tests for Userland UBSan

This was my first deliverable for the integration of KUBSan. The concept was to include tests causing simple C programs to portray Undefined Behavior, such as overflows, erroneous shifting and out of bounds accessing of arrays (VLAs actually). The ATF framework is not a real "sweetheart" to learn, so it took me more than expected to complete this preliminary step to the project. The good news was that I had enough time to understand Undefined Behavior to a suave depth and make my extensive research for ideas (and not only).

The initial commit of the tests cleaned up and submitted by my mentor Kamil Rytarowski.

Addition of Example Kernel Module Panic_String

Next on our roadmap was the understanding of NetBSD's loadable kernel modules. For this, I created a kernel module parsing a string from a device named /dev/panic and calling the kernel panic(9) with it as argument, after syncing the system. This took a long time, but in the process I had the priviledge of reading FreeBSD Device Drivers: A Guide for the Intrepid, which unfortunatelyfor our foundation is the only book in close resemblance to our kernel module infrastructure.

The panic_string module commit revised, corrected and uploaded by Kamil.

Compiling the kernel with -fsanitize=undefined

Compiled the kernel with the aforementioned option to catch UB bugs. We got one. Only one! Which was reported to the tech-kern mailing list in this Thread.

Adding the option to compile the Kernel with KUBSan

At last what was our last deliverable for GSoC's first evaluation, was getting the amd64 kernel to boot with the KUBSan option enabled. This was a trick. We only needed the appropriate dummy functions, so we could use them as symbols in the linking process of a kernel build. At first I created KUBSan as a loadable kernel module, but the chaotic structure of our codebase was to much for me. This means that I searched for 4 whole days a way to link the exported symbols to the kernel build and was unsuccessful :(. But everything happens for a reason, because that one failure ignited me to search for all the available UBSan implementations and I was able to locate the initial support of the KUBSan functionality for: Linux, Chromium and FreeBSD. Which in turn, made me realise that the module was not necessary, since I could include the KUBSan functiuonality to our /sys infrastructure. Which I did and which was successful and which allowed me to boot and run a fully KUBSan-ed kernel.

It hasn't been uploaded to upstream yet, but you can have a look at my local (and totally messy) fork.

Summary and Future Goals

This first month of GSoC has been a great experience. Last year I participated again with project trying to "revamp" support for Scheme R7RS in the Eclipse IDE (we later tried to create a Kawa-Scheme Language Server-LSP, but that's a sad story) and my overall experience was really bad (I had to quit mid-July). This year we are doing things in much friendlier, cooperative and result-producing manner. I'm really happy about that.

A brief summary is that: the Kernel booted with KUBSan and I'm in knowledge of all the tools needed to extent that functionality. That's of ye need to know up to this point.

Future goals include:
1. Making a full implementation of KUBSan, with an edge on surpassing other existing implementations,
2. Clear up any license issues,
3. Finish the amd64 implementation and switch focus to the i386,
4. Spead the NetBSD hype

At last I would like to deliver a huge thanks to my mentors Kamil and Christos for their advices and help with the project, but mostly for their incredible behavior towards the problems I went through this past month. Much love

Further Reading:
• Linux Kernel UBSan docs
• A nice in-depth article on UBSan for C/C++
• My Personal Blogsite