I have also transplanted basesystem GDB patched to my GDB repository and managed to run the GDB regression test-suite.
NetBSD distribution changes
I have enhanced and imported my local MKSANITIZER code that makes whole distribution sanitization possible. Few real bugs were fixed and a number of patches were newly written to reflect the current NetBSD sources state. I have also merged another chunk of the fruits of the GSoC-2018 project with fuzzing the userland (by plusun@).
The following changes were committed to the sources:
- ab7de18d0283 Cherry-pick upstream compiler-rt patches for LLVM sanitizers
- 966c62a34e30 Add LLVM sanitizers in the MKLLVM=yes build
- 8367b667adb9 telnetd: Stop defining the same variables concurrently in bss and data
- fe72740f64bf fsck: Stop defining the same variable concurrently in bss and data
- 40e89e890d66 Fix build of t_ubsan/t_ubsanxx under MKSANITIZER
- b71326fd7b67 Avoid symbol clashes in tests/usr.bin/id under MKSANITIZER
- c581f2e39fa5 Avoid symbol clashes in fs/nfs/nfsservice under MKSANITIZER
- 030a4686a3c6 Avoid symbol clashes in bin/df under MKSANITIZER
- fd9679f6e8b1 Avoid symbol clashes in usr.sbin/ypserv/ypserv under MKSANITIZER
- 5df2d7939ce3 Stop defining _rpcsvcdirty in bss and data
- 5fafbe8b8f64 Add missing extern declaration of ib_mach_emips in installboot
- d134584be69a Add SANITIZER_RENAME_CLASSES in bsd.prog.mk
- 2d00d9b08eae Adapt tests/kernel/t_subr_prf for MKSANITIZER
- ce54363fe452 Ship with sanitizer/lsan_interface.h for GCC 7
- 7bd5ee95e9a0 Ship with sanitizer/lsan_interface.h for LLVM 7
- d8671fba7a78 Set NODEBUG for LLVM sanitizers
- 242cd44890a2 Add PAXCTL_FLAG rules for MKSANITIZER
- 5e80ab99d9ce Avoid symbol clashes in test/rump/modautoload/t_modautoload with sanitizers
- e7ce7ecd9c2a sysctl: Add indirection of symbols to remove clash with sanitizers
- 231aea846aba traceroute: Add indirection of symbol to remove clash with sanitizers
- 8d85053f487c sockstat: Add indirection of symbols to remove clash with sanitizers
- 81b333ab151a netstat: Add indirection of symbols to remove clash with sanitizers
- a472baefefe8 Correct the memset(3)'s third argument in i386 biosdisk.c
- 7e4e92115bc3 Add ATF c and c++ tests for TSan, MSan, libFuzzer
- 921ddc9bc97c Set NOSANITIZER in i386 ramdisk image
- 64361771c78d Enhance MKSANITIZER support
- 3b5608f80a2b Define target_not_supported_body() in TSan, MSan and libFuzzer tests
- c27f4619d513 Avoids signedness bit shift in db_get_value()
- 680c5b3cc24f Fix LLVM sanitizer build by GCC (HAVE_LLVM=no)
- 4ecfbbba2f2a Rework the LLVM compiler_rt build rules
- 748813da5547 Correct the build rules of LLVM sanitizers
- 20e223156dee Enhance the support of LLVM sanitizers
- 0bb38eb2f20d Register syms.extra in LLVM sanitizer .syms files
Almost all of the mentioned commits were backported to NetBSD-9 and will land 9.0.
As a demo, I have crafted a writing on combining RUMPKERNEL, MKSANITIZER with the honggfuzz fuzzer: Rumpkernel assisted fuzzing of the NetBSD file system kernel code in userland.
GDB
I've merged NetBSD distribution downstream GDB patches into my local GDB tree and executed the regression tests (check-gdb):
[...]
Test run by kamil on Mon Sep 2 12:36:03 2019
Native configuration is x86_64-unknown-netbsd9.99
=== gdb tests ===
Schedule of variations:
unix
[...]
=== gdb Summary ===
# of expected passes 54591
# of unexpected failures 3267
# of expected failures 35
# of unknown successes 3
# of known failures 59
# of unresolved testcases 29
# of untested testcases 141
# of unsupported tests 399
Full log is here.
This means that there are a lot of more tests and known failures than in 2017-09-05:
$ uname -a
NetBSD chieftec 8.99.2 NetBSD 8.99.2 (GENERIC) #0: Sat Sep 2 22:55:29 CEST 2017 root@chieftec:/public/netbsd-root/sys/arch/amd64/compile/GENERIC amd64
Test run by kamil on Tue Sep 5 17:06:28 2017
Native configuration is x86_64--netbsd
=== gdb tests ===
Schedule of variations:
unix
[...]
=== gdb Summary ===
# of expected passes 16453
# of unexpected failures 483
# of expected failures 9
# of known failures 28
# of unresolved testcases 17
# of untested testcases 41
# of unsupported tests 25
There are actually some regressions and a set of tests that fails probably due to environment differences like lack of gfortran at hand.
Full log is here
GSoC Mentoring
The Google Summer of Code programme reached the end. My mentees wrote successfully their final reports:
I'm also mentoring the AFL+KCOV work by Maciej Grochowski. Maciej will visit EuroBSDCon-2019 and speak about his work.
Add methods for setting and getting the thread name
I've reached out to the people from standards bodies and I'm working on defining the standard approach for setting and getting the thread name. I have received a proper ID of my proposal and I'm now supposted to submit the text in either PDF or HTML format.
This change will allow to manage the thread name with an uniform interface on all comforming platforms.
Plan for the next milestone
Keep enhancing GDB support. Keep detecting ptrace(2) bugs and addressing them.
This work was sponsored by The NetBSD Foundation.
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL to chip in what you can:
I have also transplanted basesystem GDB patched to my GDB repository and managed to run the GDB regression test-suite.
NetBSD distribution changes
I have enhanced and imported my local MKSANITIZER code that makes whole distribution sanitization possible. Few real bugs were fixed and a number of patches were newly written to reflect the current NetBSD sources state. I have also merged another chunk of the fruits of the GSoC-2018 project with fuzzing the userland (by plusun@).
The following changes were committed to the sources:
- ab7de18d0283 Cherry-pick upstream compiler-rt patches for LLVM sanitizers
- 966c62a34e30 Add LLVM sanitizers in the MKLLVM=yes build
- 8367b667adb9 telnetd: Stop defining the same variables concurrently in bss and data
- fe72740f64bf fsck: Stop defining the same variable concurrently in bss and data
- 40e89e890d66 Fix build of t_ubsan/t_ubsanxx under MKSANITIZER
- b71326fd7b67 Avoid symbol clashes in tests/usr.bin/id under MKSANITIZER
- c581f2e39fa5 Avoid symbol clashes in fs/nfs/nfsservice under MKSANITIZER
- 030a4686a3c6 Avoid symbol clashes in bin/df under MKSANITIZER
- fd9679f6e8b1 Avoid symbol clashes in usr.sbin/ypserv/ypserv under MKSANITIZER
- 5df2d7939ce3 Stop defining _rpcsvcdirty in bss and data
- 5fafbe8b8f64 Add missing extern declaration of ib_mach_emips in installboot
- d134584be69a Add SANITIZER_RENAME_CLASSES in bsd.prog.mk
- 2d00d9b08eae Adapt tests/kernel/t_subr_prf for MKSANITIZER
- ce54363fe452 Ship with sanitizer/lsan_interface.h for GCC 7
- 7bd5ee95e9a0 Ship with sanitizer/lsan_interface.h for LLVM 7
- d8671fba7a78 Set NODEBUG for LLVM sanitizers
- 242cd44890a2 Add PAXCTL_FLAG rules for MKSANITIZER
- 5e80ab99d9ce Avoid symbol clashes in test/rump/modautoload/t_modautoload with sanitizers
- e7ce7ecd9c2a sysctl: Add indirection of symbols to remove clash with sanitizers
- 231aea846aba traceroute: Add indirection of symbol to remove clash with sanitizers
- 8d85053f487c sockstat: Add indirection of symbols to remove clash with sanitizers
- 81b333ab151a netstat: Add indirection of symbols to remove clash with sanitizers
- a472baefefe8 Correct the memset(3)'s third argument in i386 biosdisk.c
- 7e4e92115bc3 Add ATF c and c++ tests for TSan, MSan, libFuzzer
- 921ddc9bc97c Set NOSANITIZER in i386 ramdisk image
- 64361771c78d Enhance MKSANITIZER support
- 3b5608f80a2b Define target_not_supported_body() in TSan, MSan and libFuzzer tests
- c27f4619d513 Avoids signedness bit shift in db_get_value()
- 680c5b3cc24f Fix LLVM sanitizer build by GCC (HAVE_LLVM=no)
- 4ecfbbba2f2a Rework the LLVM compiler_rt build rules
- 748813da5547 Correct the build rules of LLVM sanitizers
- 20e223156dee Enhance the support of LLVM sanitizers
- 0bb38eb2f20d Register syms.extra in LLVM sanitizer .syms files
Almost all of the mentioned commits were backported to NetBSD-9 and will land 9.0.
As a demo, I have crafted a writing on combining RUMPKERNEL, MKSANITIZER with the honggfuzz fuzzer: Rumpkernel assisted fuzzing of the NetBSD file system kernel code in userland.
GDB
I've merged NetBSD distribution downstream GDB patches into my local GDB tree and executed the regression tests (check-gdb):
[...]
Test run by kamil on Mon Sep 2 12:36:03 2019
Native configuration is x86_64-unknown-netbsd9.99
=== gdb tests ===
Schedule of variations:
unix
[...]
=== gdb Summary ===
# of expected passes 54591
# of unexpected failures 3267
# of expected failures 35
# of unknown successes 3
# of known failures 59
# of unresolved testcases 29
# of untested testcases 141
# of unsupported tests 399
Full log is here.
This means that there are a lot of more tests and known failures than in 2017-09-05:
$ uname -a
NetBSD chieftec 8.99.2 NetBSD 8.99.2 (GENERIC) #0: Sat Sep 2 22:55:29 CEST 2017 root@chieftec:/public/netbsd-root/sys/arch/amd64/compile/GENERIC amd64
Test run by kamil on Tue Sep 5 17:06:28 2017
Native configuration is x86_64--netbsd
=== gdb tests ===
Schedule of variations:
unix
[...]
=== gdb Summary ===
# of expected passes 16453
# of unexpected failures 483
# of expected failures 9
# of known failures 28
# of unresolved testcases 17
# of untested testcases 41
# of unsupported tests 25
There are actually some regressions and a set of tests that fails probably due to environment differences like lack of gfortran at hand.
Full log is here
GSoC Mentoring
The Google Summer of Code programme reached the end. My mentees wrote successfully their final reports:
I'm also mentoring the AFL+KCOV work by Maciej Grochowski. Maciej will visit EuroBSDCon-2019 and speak about his work.
Add methods for setting and getting the thread name
I've reached out to the people from standards bodies and I'm working on defining the standard approach for setting and getting the thread name. I have received a proper ID of my proposal and I'm now supposted to submit the text in either PDF or HTML format.
This change will allow to manage the thread name with an uniform interface on all comforming platforms.
Plan for the next milestone
Keep enhancing GDB support. Keep detecting ptrace(2) bugs and addressing them.
This work was sponsored by The NetBSD Foundation.
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL to chip in what you can:
This year EuroBSDCon took place in Lillehammer Norway. I had the pleasure to attend as a speaker with my talk about fuzzing the NetBSD filesystems.
Venue
Lillehammer is a ski resort, nestled amid very beautiful scenery between mountains and lakes, just two hours from Oslo. The conference took place in the Scandic Lillehammer Hotel, a little bit away from the downtown of Lillehammer, close to the Olympic Ski Jumps.
Talks
Every year, EuroBSDCon has a lot of interesting talks. Unfortunately, it is hard to attend all the interesting seminars, as many of them take place at the same time, so I won't be able to highlight all of them; accordingly, I gratefully acknowledge several organizations for handling the live streaming from every session.
Keynote: Embedded Ethics
The conference started with an excellent Keynote from Patricia Aas (ex. Opera/Cisco/Vivaldi, cur Turtlesec), about the Ethics in the IT industry. As a person who is familiar with the issues with the privacy and many different threads of abusing user data by the company, I have to say that this talk started the avalanche of different thoughts and reflections in my mind. To my surprise, I was not the only one to have such thoughts. This topic arose quite often during the rest of the conference through many conversations between different people. For those of you who didn't see it yet, I highly recommend that you do. The key takeaway is that we, the people who are building today's digital world, need to think about the implications of our work and decisions upon the users of our services. This topic is getting more complicated even as we think about it. However, Patricia come here with the strategy "Annoying as a Service" that can be simply used in every situation to at least not makes things worse...
Conference Talks
During the first day, there were a couple of interesting talks about NetBSD: "Improving modularity of NetBSD compat code", and mine, on "Fuzzing NetBSD Filesystems" [+ Taking NetBSD kernel bug roast to the next level: Kernel Fuzzers (quick A.D. 2019 overview) by Kamil Rytarowski]. As it turns out, there was another interesting talk about foundations of kernel fuzzing by Andrew Turner, in which he presented the connection between sanitizers, tracing modes and fuzzers. After the break, I attended the excellent talk "7th Edition Unix at 40" by Warner Losh -- if you love the history of Unix, this is a must-see. The first day finished with the social mixer. The second day started with one of my favourites of the entire conference: "Kernel TLS and TLS hardware offload" via Drew Gallatin and Hans Petter Selasky. In another room was also a very interesting seminar on Rust for System Programmers. The next session via Netflix folks was about NUMA optimizations in the FreeBSD Network stack, another interesting talk about the usage of BSD as a high-speed CDN serving about 200Gbps Video content(!). After that, I attended the session on The Future of OpenZFS via Allan Jude, where he showed the progress done in the collaboration of different OSes on ZFS Filesystem. The last sessions I attended were the "23 years of software side-channel attacks" by Colin, and the last one before the closing notes: "Unbound & FreeBSD: A true love story", by Pablo Carboni.
Highlights
- Security: We can see clearly that the BSD community continues efforts for making BSDs more secure on various levels. This year we talked mostly about fuzzing, and in this area, it is impossible not to recognise NetBSD for great progress.
- CDN use-case: Netflix contributions to FreeBSD make it a great system for CDN, year after year innovating and increasing the performance. I hope we will see more companies using BSDs as core for their CDN infrastructure.
- ZFS: The filesystem has come a long way, despite being a project divided between different communities. Now thanks to the efforts of the developers, OpenZFS as a united community will be able to progress even faster and take advantage of projects that are using it. I believe the OpenZFS initiative is one of the most important steps taken by the community in many years.
Social Event
This year's social event took place in the Open Air Museum in Maihaugen, where we were able to see, preserved in excellent condition, parts of the Norwegian houses from the 19th century through the late 20th century. The fun part was that every house was open and you were able to go inside, some of them with people dressed up in the fashion of the same years, talking about the age. I very much enjoyed it, as it was a great opportunity to learn more about Norwegian culture and history.
Next Year!
The most important key point during closing notes is always: "where will the next EuroBSDCon take place?!" This year the guessing game was:
- Beer will be cheaper than in Norway
- [picture of Schnitzel]
- Photo of...
Vienna!
Hope to see you all next year in Vienna!
This year EuroBSDCon took place in Lillehammer Norway. I had the pleasure to attend as a speaker with my talk about fuzzing the NetBSD filesystems.
Venue
Lillehammer is a ski resort, nestled amid very beautiful scenery between mountains and lakes, just two hours from Oslo. The conference took place in the Scandic Lillehammer Hotel, a little bit away from the downtown of Lillehammer, close to the Olympic Ski Jumps.
Talks
Every year, EuroBSDCon has a lot of interesting talks. Unfortunately, it is hard to attend all the interesting seminars, as many of them take place at the same time, so I won't be able to highlight all of them; accordingly, I gratefully acknowledge several organizations for handling the live streaming from every session.
Keynote: Embedded Ethics
The conference started with an excellent Keynote from Patricia Aas (ex. Opera/Cisco/Vivaldi, cur Turtlesec), about the Ethics in the IT industry. As a person who is familiar with the issues with the privacy and many different threads of abusing user data by the company, I have to say that this talk started the avalanche of different thoughts and reflections in my mind. To my surprise, I was not the only one to have such thoughts. This topic arose quite often during the rest of the conference through many conversations between different people. For those of you who didn't see it yet, I highly recommend that you do. The key takeaway is that we, the people who are building today's digital world, need to think about the implications of our work and decisions upon the users of our services. This topic is getting more complicated even as we think about it. However, Patricia come here with the strategy "Annoying as a Service" that can be simply used in every situation to at least not makes things worse...
Conference Talks
During the first day, there were a couple of interesting talks about NetBSD: "Improving modularity of NetBSD compat code", and mine, on "Fuzzing NetBSD Filesystems" [+ Taking NetBSD kernel bug roast to the next level: Kernel Fuzzers (quick A.D. 2019 overview) by Kamil Rytarowski]. As it turns out, there was another interesting talk about foundations of kernel fuzzing by Andrew Turner, in which he presented the connection between sanitizers, tracing modes and fuzzers. After the break, I attended the excellent talk "7th Edition Unix at 40" by Warner Losh -- if you love the history of Unix, this is a must-see. The first day finished with the social mixer. The second day started with one of my favourites of the entire conference: "Kernel TLS and TLS hardware offload" via Drew Gallatin and Hans Petter Selasky. In another room was also a very interesting seminar on Rust for System Programmers. The next session via Netflix folks was about NUMA optimizations in the FreeBSD Network stack, another interesting talk about the usage of BSD as a high-speed CDN serving about 200Gbps Video content(!). After that, I attended the session on The Future of OpenZFS via Allan Jude, where he showed the progress done in the collaboration of different OSes on ZFS Filesystem. The last sessions I attended were the "23 years of software side-channel attacks" by Colin, and the last one before the closing notes: "Unbound & FreeBSD: A true love story", by Pablo Carboni.
Highlights
- Security: We can see clearly that the BSD community continues efforts for making BSDs more secure on various levels. This year we talked mostly about fuzzing, and in this area, it is impossible not to recognise NetBSD for great progress.
- CDN use-case: Netflix contributions to FreeBSD make it a great system for CDN, year after year innovating and increasing the performance. I hope we will see more companies using BSDs as core for their CDN infrastructure.
- ZFS: The filesystem has come a long way, despite being a project divided between different communities. Now thanks to the efforts of the developers, OpenZFS as a united community will be able to progress even faster and take advantage of projects that are using it. I believe the OpenZFS initiative is one of the most important steps taken by the community in many years.
Social Event
This year's social event took place in the Open Air Museum in Maihaugen, where we were able to see, preserved in excellent condition, parts of the Norwegian houses from the 19th century through the late 20th century. The fun part was that every house was open and you were able to go inside, some of them with people dressed up in the fashion of the same years, talking about the age. I very much enjoyed it, as it was a great opportunity to learn more about Norwegian culture and history.
Next Year!
The most important key point during closing notes is always: "where will the next EuroBSDCon take place?!" This year the guessing game was:
- Beer will be cheaper than in Norway
- [picture of Schnitzel]
- Photo of...
Vienna!
Hope to see you all next year in Vienna!