I took part in BSDCan 2019 and during the event wrote a NetBSD version of truss, a ptrace(2)-powered syscall tracing utility from FreeBSD. I've finished the port after getting back home and published it to the NetBSD community. This work allowed me to validate the ptrace(2) interfaces in another application and catch new problems that affect every ptrace(2)-based debugger.
Changes in the basesystem
A number of modifications were introduced to the main NetBSD repository which include ptrace(2), kernel sanitizer, and kernel coverage.
Improvements in the ptrace(2) code:
- Reporting exec(3) events in the PT_SYSCALL mode. This simplifies debuggers adding MI interface to detect exec(3) of a tracee.
- Relax prohibition of Program Counter set to 0x0 in ptrace(2). In theory certain applications can map Virtual Address 0x0 and run code there. In practice this will probably never happen in modern computing.
- New ATF tests for user_va0_disable sysctl(3) node checking if PT_CONTINUE/PT_SYSCALL/PT_DETACH can set Program Counter to 0x0.
- Fixing kernel races with threading code when reports for events compare with process termination. This unbroke tracing Go programs, which call exit(2) before collecting all the running threads.
- Refreshed ptrace(2) and siginfo(2) documentation with recent changes and discoveries.
I mentor two GSoC projects and support community members working on kernel fuzzers. As a result of this work, fixes are landing in the distribution source code from time to time. We have achieved an important milestone with being able to run NetBSD/amd64 on real hardware with Kernel Undefined Behavior Sanitizer without any reports from boot and execution of ATF tests. This achievement will allow us to enable kUBSan reports as fatal ones and use in the fuzzing process, capturing bugs such as integer overflow, out of bounds array use, invalid shifts, etc.
I have also introduced a support of a new sysctl(3) operation: KERN_PROC_CWD.
This call retrieves the current working directory of a specified process.
This operation is used typically in terminal-related applications
(tmux, terminator, ...) and generic process status prompters (Python psutils).
I have found out that my terminal emulator after system upgrade doesn't work correctly
and it uses a fallback of resolving the symbolic link of /proc/*/cwd.
ptrace(2) utilities
In order to more easily validate the kernel interfaces I wrote a number of utility programs that use the ptrace(2) functionality. New programs reuse the previously announced picotrace (https://github.com/krytarowski/picotrace) code as a framework.
Some programs have been already published, other are still in progress and kept locally on my disk only. New programs that have been published so far:
- NetBSD truss - reimplementation from scratch of FreeBSD truss for the NetBSD kernel.
- sigtracer - stripped down picotrace reporting signal events.
- coredumper - crash detector and core(5) dumper regardless of signal handing or masking options in programs.
More elaborated introduction with examples is documented on the GitHub repository. I use these programs for profit as they save my precious time on debugging issues in programs and as validators for kernel interfaces as they are working closely over the kernel APIs (contrary to large code base of GDB or LLDB). I've already detected and verified various kernel ptrace(2) issues with these programs. With a debugger it's always unclear whether a problem is in the debugger, its libraries or the kernel.
Among the detected issues, the notable list of them is as follows:
- Reporting events to a debugger in the same time from two or more sources is racy and one report can be registered and other dropped.
- Either questionable or broken handling of process stopped by other process with a stop signal.
- Attaching to sleeping process terminates it as interrupted nanosleep does not return EINTR.
- Exiting of a process with running multiple threads broken (fixed).
- Attaching to applications due to unknown reasons can sometimes kill them.
- Callbacks from pthread_atfork(3) (can?) generate crash signals unhandled by a tracer.
Update in Problem Report entries
I've reiterated over reported bugs in the gnats tracking system. These problems were still valid in the beginning of this year and are now reported to be gone:
- kern/52548 (GDB breakpoint does not work properly in the binary created with golang on NetBSD (both pkgsrc/lang/go and pkgsrc/lang/go14).)
- kern/53120 (gdb issues with threaded programs)
- bin/49662 (gdb has trouble with threaded programs)
- port-arm/51677 (gdb can not debug threaded programs)
- kern/16284 (running function in stoped gdb causes signal 93)
- bin/49979 (gdb does not work properly on NetBSD-6.1.1/i386)
- kern/52628 (tracing 32-bit application on 64-bit kernel panic)
Summary
There are still sometimes new bugs reported in ptrace(2) or GDB, but they are usually against racy ATF test and complex programs. I can also observe a difference that simple and moderately complex programs usually work now and the reports are for heavy ones like firefox (multiple threads and multiple processes).
I estimate that there are still at least 3 critical threading issues to be resolved, races and use-after-free scenaris in vfork(2) and a dozen of other more generic problems typically in signal routing semantics.
Plan for the next milestone
Cover with regression tests the posix_spawn(2) interface, if needed correct the kernel code. This will be followed by resolving the use-after-free scenarios in vfork(2). This is expected to accomplish the tasks related to forking code.
My next goal on the roadmap is to return to LWP (threading) code and fix all currently known problems.
Independently I will keep supporting the work on kernel fuzzing projects within the GSoC context.
This work was sponsored by The NetBSD Foundation.
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL to chip in what you can:
The NetBSD Project is pleased to announce NetBSD 8.1, the first feature and stability maintenance release of the netbsd-8 stable branch.
Besides the workarounds for the latest CPU specific vulnerabilities, this also includes many bug fixes and a few selected new drivers. For more details and instructions see the 8.1 announcement.
Get NetBSD 8.1 from our CDN (provided by fastly) or one of the ftp mirrors.
Complete source and binaries for NetBSD are available for download at many sites around the world. A list of download sites providing FTP, AnonCVS, and other services may be found at https://www.NetBSD.org/mirrors/.
Upstream describes LLDB as a next generation, high-performance debugger. It is built on top of LLVM/Clang toolchain, and features great integration with it. At the moment, it primarily supports debugging C, C++ and ObjC code, and there is interest in extending it to more languages.
In February, I have started working on LLDB, as contracted by the NetBSD Foundation. So far I've been working on reenabling continuous integration, squashing bugs, improving NetBSD core file support and lately extending NetBSD's ptrace interface to cover more register types. You can read more about that in my Apr 2019 report.
In May, I was primarily continuing the work on new ptrace interface. Besides that, I've found and fixed a bug in ptrace() compat32 code, pushed LLVM buildbot to ‘green’ status and found some upstream LLVM regressions. More below.
Buildbot status update
Traditionally, let's start with buildbot updates. The buildbot is providing continuous integration for a number of LLVM projects on NetBSD, including LLDB, clang and clang's runtime libraries. Its available at: http://lab.llvm.org:8011/builders/netbsd-amd64.
Previously, the most significant problem in using buildbot was flakiness of LLDB tests which resulted in frequent false positives. I was able to finally reduce this effect through reducing the number of parallel test runs for LLDB tests. To avoid slowing down other test suites, I have used a sed hack that overrides job count directly in the specific lit invocation.
Additionally, I have fixed a few regressions during the period, notably:
worked around missing nexttowardl() in NetBSD 8 causing libc++ test failure, by using std::nextafter() in the problematic test: r360673,
fixed compiler path test to work correctly without specific linker being available: r360761,
fixed inferring source paths in libunwind that prevented the tests from finding libc++: r361931,
removed test case that relied on read() attempt from a directory producing very specific error message: r362404 (NB: this failed because NetBSD permits reading from directory descriptors).
Those fixes permitted the buildbot to become green for a short period of time. Sadly, shortly afterwards one of AMDGPU tests started failing and we are still trying to find the cause.
Adding register read/write tests to ATF tests
Last month, I have implemented a number of register reading/writing tests for LLDB. This month I've introduced matching tests inside NetBSD's ATF test suite. This provides the ability to test NetBSD's ptrace implementation directly on the large variety of platforms and kernels supported by NetBSD. With the dynamic development of NetBSD, running LLDB tests everywhere would not be feasible.
While porting the tests, I've made a number of improvements, some of them requested specifically by LLDB upstream. Those include:
starting to use better input/output operands for assembly, effectively reducing the number of direct register references and redundant code: r359978,
using more readable/predictable constants for register data, read part: r360041, write part: r360154,
using %0 and %1 operands to reference memory portably between i386 and amd64: r360148.
The relevant NetBSD commits for added tests are (using the git mirror):
general-purpose register reading tests: 7a58d92435a9,
fix to the above: split tests for reading i386 gp registers as not to require MMX: 06be77bbafa6,
r8..r15 amd64 register reading tests: 86f6b1d4dab6,
mm & xmm register reading tests: 3ef02e1666ae,
general-purpose register writing tests: 95bfedcb6a89,
mm & xmm register writing tests: 2c8335920f61.
While working on this, I've also noticed that struct fpreg and struct xmmregs are not fully specified on i386. In bbc3f184d470, I've added the fields needed to make use of those structures convenient.
Fixing compat32: request mapping and debug registers
Kamil has asked me to look into PR#54233 indicating problems with 32-bit application debugging on amd64. While the problem in question most likely combines multiple issues, one specifically related to my work was missing PT_*DBREGS support in compat32.
While working on this, I've found out that the functions responsible for implementing those requests were not called at all. After investigating, I've came to the following conclusion. The i386 userland code has passed PT_* request codes corresponding to i386 headers to the compat32 layer. The compat32 layer has passed those codes unmodified to the common kernel code and compared them to PT_* constants available in kernel code which happened to be amd64 constants.
This worked fine for low requests numbers that happened to match on both architectures. However, i386 adds two additional requests (PT_*XMMREGS) after PT_SETFPREGS, and therefore all remaining requests are offset.
To solve this, I've created a request code mapping function that converts i386 codes coming from userland to the matching amd64 values used in the kernel. For the time being, this supports only requests common to both architectures, and therefore PT_*XMMREGS can't be implemented without further hacking it.
Once I've managed to fix compat32, I went ahead to implement PT_*DBREGS in compat32. Kamil has made an initial implementation in the past but it was commented out and lacked input verification. However, I've chosen to change the implementation a bit and reuse x86_dbregs_read() and x86_dbregs_write() functions rather than altering pcb directly. I've also added the needed value checks for PT_SETDBREGS.
Both changes were committed to /usr/src:
Initial XSAVE work
In the previous report, I have been considering which approach to take in order to provide access to the additional FPU registers via ptrace. Eventually, the approach to expose the raw contents of XSAVE area got the blessing, and I've started implementing it.
However, this approach proved impractical. The XSAVE area in standard format (which we are using) consists of three parts: FXSAVE-compatible legacy area, XSAVE header and zero or more extended components. The offsets of those extended components turned out to be unpredictable and potentially differing between various CPUs. The architecture developer's manual indicates that the relevant offsets can be obtained using CPUID calls.
Apparently both Linux and FreeBSD did not take this into consideration when implementing their API, and they effectively require the caller to issue CPUID calls directly. While such an approach could be doable in NetBSD, it would prevent core dumps from working correctly on a different CPU. Therefore, it would be necessary to perform the calls in kernel instead, and include the results along with XSAVE data.
However, I believe that doing so would introduce unnecessary complexity for no clear gain. Therefore, I proposed two alternative solutions. They were to either:
copy XSAVE data into custom structure with predictable indices, or
implement separate PT_* requests for each component group, with separate data structure each.
Comparison of the two proposed solutions
Both solutions are roughly equivalent. The main difference between them is that the first solution covers all extended registers (and is future-extensible) in one request call, while the second one requires new pair of requests for each new register set.
I personally prefer the former solution because it reduces the number of ptrace calls needed to perform typical operations. This is especially relevant when reading registers whose contents are split between multiple components: YMM registers (whose lower bits are in SSE area), and lower ZMM registers (whose lower bits are YMM registers).
Example code reading the ZMM register using a single request solution would look like:
struct xstate xst; struct iovec iov; char zmm_reg[64]; iov.iov_base = &xst; iov.iov_len = sizeof(xst); ptrace(PT_GETXSTATE, child_pid, &iov, 0); // verify that all necessary components are available assert(xst.xs_xstate_bv & XCR0_SSE); assert(xst.xs_xstate_bv & XCR0_YMM_Hi128); assert(xst.xs_xstate_bv & XCR0_ZMM_Hi256); // combine the values memcpy(&zmm_reg[0], &xst.xs_fxsave.fx_xmm[0], 16); memcpy(&zmm_reg[16], &xst.xs_ymm_hi128.xs_ymm[0], 16); memcpy(&zmm_reg[32], &xst.xs_zmm_hi256.xs_zmm[0], 32);
For comparison, the equivalent code for the other variant would roughly be:
#if defined(__x86_64__) struct fpreg fpr; #else struct xmmregs fpr; #endif struct ymmregs ymmr; struct zmmregs zmmr; char zmm_reg[64]; #if defined(__x86_64__) ptrace(PT_GETFPREGS, child_pid, &fpr, 0); #else ptrace(PT_GETXMMREGS, child_pid, &fpr, 0); #endif ptrace(PT_GETYMMREGS, child_pid, &ymmr, 0); ptrace(PT_GETZMMREGS, child_pid, &zmmr, 0); memcpy(&zmm_reg[0], &fpr.fxstate.fx_xmm[0], 16); memcpy(&zmm_reg[16], &ymmr.xs_ymm_hi128.xs_ymm[0], 16); memcpy(&zmm_reg[32], &zmmr.xs_zmm_hi256.xs_zmm[0], 32);
I've submitted a patch set implementing the first solution, as it was easier to convert to from the initial approach. If the feedback indicates the preference of the other solution, a conversion to it should also be easier to the other way around. It is available on tech-kern mailing list: [PATCH 0/2] PT_{GET,SET}XSTATE implementation, WIP v1.
The initial implementation should support getting and setting x87, SSE, AVX and AVX-512 registers (i.e. all types currently enabled in the kernel). The tests cover all but AVX-512. I have tested it on native amd64 and i386, and via compat32.
Future plans
The most immediate goal is to finish the work on XSAVE. This includes responding to any feedback received, finding AVX-512 hardware to test it on, writing tests for AVX-512 registers and eventually committing the patches to the NetBSD kernel. Once this is done, I need to extend XSAVE support into core dumps, and implement userland-side of both into LLDB.
Besides that, the next items on TODO are:
Adding support for debug registers (moved from last month's TODO).
Adding support to backtrace through signal trampoline.
Working on i386 and aarch64 LLDB port.
In the meantime, Kamil's going to continue working on improving fork and thread support kernel-side, preparing it for my work LLDB-side.
This work is sponsored by The NetBSD Foundation
The NetBSD Foundation is a non-profit organization and welcomes any donations to help us continue funding projects and services to the open-source community. Please consider visiting the following URL to chip in what you can:
The first coding period of The Google Summer of Code has come to an end. It has been a great experience so far and I got the opportunity to learn a lot of new stuff. This is a report on the work I have during this coding period.
About TriforceAFL
TriforceAFL is a modified version of AFL that supports fuzzing using QEMU's full system emulation. This offers several advantages such as the fact that pieces of the kernel need not be recompiled with AFL or that the kernel does not need to be built with coverage support. More details on other advantages, design and implementation of TriforceAFL can be found here.
The TriforceLinuxSyscallFuzzer and the TriforceOpenBSDSyscallFuzzer are syscall fuzzers built on top of TriforceAFL. The end goal of this project is to adapt TriforceAFL for NetBSD syscall fuzzing.
Adapted TriforceAFL for pkgsrc-wip
One of the end goals of the project is to make the fuzzer available as a pkgsrc package. To do so, TriforceAFL had to be first ported to pkgsrc. TriforceAFL uses qemu, so the appropriate patches to qemu for NetBSD were applied and few other minor issues resolved. The working package is now available in pkgsrc-wip.
The NetBSD Syscall Fuzzer
TriforceNetBSDSyscallFuzzer can be now used to perform system call fuzzing of NetBSD kernels using AFL and QEMU. The setup scripts and the driver program are functioning. The syscalls list has been updated for NetBSD and basic input generation works. Documentation detailing the process of setup(of the NetBSD installation/kernel image), building and fuzzing along with the code is available on github.
The fuzzer functions properly and detects crashes which can be reproduced using the driver. Although it can severely benefit from better input generation and optimisation. This will be the focus in the next coding period.
Summary
In the coming weeks, the work of optimizing the fuzzer is to be done, so that it is more efficient at catching bugs, I am looking forward to doing so and making TriforceNetBSDSyscallFuzzer available on NetBSD through pkgsrc.
Lastly I would like to thank my mentor, Kamil Rytarowski for always helping me through the process and guiding me whenever I needed any help.
Prepared by Siddharth Muralee(@R3x) as a part of Google Summer of Code 2019
As a part of Google Summer of Code 19, I am working on improving the support for Syzkaller kernel fuzzer. Syzkaller is an unsupervised coverage-guided kernel fuzzer, that supports a variety of operating systems including NetBSD. This report details the work done during the first coding period.
Syzkaller
Initially, Syzkaller was developed with Linux kernel fuzzing in mind, but now it's being extended to support other OS kernels as well. The main target of the Syzkaller fuzzer is the system call layer.
Thanks to Google, we now have a 24/7 continuous fuzzing instance on a Google Cloud engine for NetBSD managed by Syzbot, a sub-system of the Syzkaller fuzzer. Syzbot maintains a dashboard where it reports all the bugs that were found. Syzbot also maintains a mailing list to report bugs found.
Syzbot is currently fuzzing NetBSD-HEAD which it updates from the Github mirror.
You can go through the Syzkaller documentation or take a look at my slides from Troopers 19 for learning more about Syzkaller.
Automated Image creation
Syzkaller required NetBSD images with ssh-keys for continous fuzzing. Due to frequent updates to HEAD sometimes images become unusable. So I decided to automate the process of creating NetBSD images with ssh-keys and appropriate settings.
Initial attempts with packer by Hashicorp failed because the boot sequence was controlled by waits and not output matching. So I wrote a shell script which basically adds a wrapper around anita to do the job.
Kcov(4) support
Kernel Code Coverage(KCov) is a compiler instrumented feature which helps us in finding the code paths executed inside the kernel for a certain set of system calls. An initial port was done by me with modifications by @kamil and @maxv.
Syzkaller uses coverage for modifying and mutating the arguments of syscalls. Coverage information for NetBSD is publicly available.
Sanitizers Support
Sanitizers are compiler instrumented tools to improve code correctness. Currently, NetBSD supports Kernel Address Sanitizer(KASAN) and Kernel Undefined behaviour Sanitizer(KUBSAN).
We use the Sanitizers to increase the chances of finding bugs. Syzkaller now compiles kernels with the Sanitizers.
Report Generation and Symbolization
Syzkaller logs the console and in the event of a crash it records the log to find out details about the crash, these details are then used to classify the crash and create a report.
For better crash reports, we decided to enable a ddb(4) shell on event of a kernel panic. This allowed us to print backtraces, details of locks and processes.
Also I added support for better Symbolization of the reports. Symbolization is adding more details in the crash report to make them easier for developers to go through. Currently we have added file names and line numbers for functions in the crash based on the kernel object (netbsd.gdb).
Initial backtrace :
do_ptrace() at netbsd:do_ptrace+0x33d sys_ptrace() at netbsd:sys_ptrace+0x71 sys_syscall() at netbsd:sys_syscall+0xf5
After Symbolization :
do_ptrace() at netbsd:do_ptrace+0x33d sys/kern/sys_ptrace_common.c:1430 sys_ptrace() at netbsd:sys_ptrace+0x71 sys/kern/sys_ptrace.c:218 sys_syscall() at netbsd:sys_syscall+0xf5 sy_call sys/sys/syscallvar.h:65 [inline]
Syscall Coverage Improvements
Syzkaller uses a pseudo-formal grammar for system calls. It uses the same to build programs to fuzz the kernel. The files related to the same are stored at sys/netbsd in the syzkaller repo. These files were a rough copy from the linux files with whatever was unable to compile removed.
We had to review all the existing syscall descriptions, find the missing ones and add them.
I wrote a python script which would help in finding out existing syscall descriptions and match them with the NetBSD description. The script also finds missing syscalls and logs them.
I have listed the system calls that are currently fuzzed with Syzkaller.
We are currently working on adding and improving descriptions for the more important syscalls. If you would like to see a NetBSD system call fuzzed you can reach out to us.
Summary
During the last month, I was focusing on improving support for NetBSD. I have managed to complete the tasks that we had planned for the first evaluation.
For the next coding period (28th June - 22nd July) I will be working on adding support for fuzzing the Network layer.
Last but not least, I want to thank my mentors, @kamil and @cryo for their useful suggestions and guidance. I would also like to thank Dmitry Vyukov, Google for helping with any issues faced with regard to Syzkaller. Finally, thanks to Google to give me a good chance to work with NetBSD community.
Prepared by Manikishan Ghantasala (shannu) as a part of Google Summer of Code 2019.
Greetings everyone, I am Manikishan an Undergraduate pursuing my Bachelors Degree in Computer Science from Amrita Vishwa Vidyapeetham, Amritapuri, Kerala, India. I have been very interested in working on the lower level development such as Operating Systems, Kernels, and Compilers. I have also worked on building a computer from scratch. The project is named From Nand To Tetris
. It had helped me elevate my interest in the field of Operating Systems and to apply for this organization. I am very grateful to be a part of this program and also would like to thank the community and my mentors for granting me the opportunity and being supportive at all times.
Regarding the first evaluation, it has been quite interesting working on Add KNF (NetBSD style) in clang-format project. I love the NetBSD community and look forward to continue. It has helped me to learn a lot during this period. It has been challenging and amazing so far.
This is a blog post about the work I have done prior to the first evaluation period.
What is clang-format?
Clang-format is a set of tools that format code built upon LibFormat. It supports some of the coding styles such as LLVM, Google, Chromium, Mozilla, Webkit which can be chosen by -style=<StyleName>. There is another option of writing a config file named .clang-format with custom style rules. My project is to add NetBSD KNF support to clang-format.
With the added support to clang-format, it will be able to format the code according to the NetBSD Style when run with -style=NetBSD.
Getting familiar to LLVM Source
For the first week, I went on exploring LLVM source to find out implementations for the similar style rules I have listed in my proposal. I have managed to figure out a way for the same with the help of my supportive mentors. I have implemented two styles:
- BitFieldDeclarationsOnePerLine
- SortNetBSDIncludes
About BitFieldDeclarationsOnePerLine
This rule lines up BitField declarations on consecutive lines with correct indentation.
Example:
Input:
unsigned int bas :3, hh : 4, jjj : 8;
unsigned int baz:1,
fuz:5,
zap:2;
Output:
unsigned int bas:3,
hh:4,
jjj:8;
unsigned int baz:1,
fuz:5,
zap:2;
Submitted a patch regarding this in the differential for review.
-> Patch: https://reviews.llvm.org/D63062
There is a bug in the implementation where the indentation breaks when there is a comment in between the bitfields, which I kept aside for getting on to go with the next style and will fix it in the coming weeks.
About SortNetBSDIncludes
Clang-format has a native SortIncludes style, which sorts all the headers in alphabetical order where NetBSD headers follow a special order due to dependencies between each header. After discussing with my mentors and in the tech-toolchain we have come up with more precise order in which we should follow for headers:
- <sys/param.h>
- <sys/types.h>
- <sys/*> -- kernel headers
- <uvm/*> -- vm headers
- <net*/*> -- network protocol headers
- <*fs/*> -- filesystem headers
- <dev/*> -- device driver headers
- <protocols/.h>
- <machine/*>
- <[arch]/*>
- < /usr includes next>
- <paths.h>
- User include files
I have made more smarter version with Regex with necessary changes from the approach I had before which was a harcoded one, the patch for this will be up by this weekend.
Plan for the next phase
I was a bit behind the proposed schedule because understanding the LLVM source took more time than expected. I am confident that I will be speeding up my progress in the upcoming phase and complete as many styles possible. The final plan is to add support for NetBSD in clang-format which can be used by --style=NetBSD.
Summary
In the coming weeks, I will fix and add all the missing features to existing styles and new ones, and concentrate on optimizing and testing following all guidelines of NetBSD and make sure a formatted code doesn't cause a build fail.
Lastly, I would like to thank my wonderful mentors Michal and Christos for helping me to the process and guiding me whenever needed.
Prepared by Manikishan Ghantasala (shannu) as a part of Google Summer of Code 2019.
Greetings everyone, I am Manikishan an Undergraduate pursuing my Bachelors Degree in Computer Science from Amrita Vishwa Vidyapeetham, Amritapuri, Kerala, India. I have been very interested in working on the lower level development such as Operating Systems, Kernels, and Compilers. I have also worked on building a computer from scratch. The project is named From Nand To Tetris
. It had helped me elevate my interest in the field of Operating Systems and to apply for this organization. I am very grateful to be a part of this program and also would like to thank the community and my mentors for granting me the opportunity and being supportive at all times.
Regarding the first evaluation, it has been quite interesting working on Add KNF (NetBSD style) in clang-format project. I love the NetBSD community and look forward to continue. It has helped me to learn a lot during this period. It has been challenging and amazing so far.
This is a blog post about the work I have done prior to the first evaluation period.
What is clang-format?
Clang-format is a set of tools that format code built upon LibFormat. It supports some of the coding styles such as LLVM, Google, Chromium, Mozilla, Webkit which can be chosen by -style=<StyleName>. There is another option of writing a config file named .clang-format with custom style rules. My project is to add NetBSD KNF support to clang-format.
With the added support to clang-format, it will be able to format the code according to the NetBSD Style when run with -style=NetBSD.
Getting familiar to LLVM Source
For the first week, I went on exploring LLVM source to find out implementations for the similar style rules I have listed in my proposal. I have managed to figure out a way for the same with the help of my supportive mentors. I have implemented two styles:
- BitFieldDeclarationsOnePerLine
- SortNetBSDIncludes
About BitFieldDeclarationsOnePerLine
This rule lines up BitField declarations on consecutive lines with correct indentation.
Example:
Input:
unsigned int bas :3, hh : 4, jjj : 8;
unsigned int baz:1,
fuz:5,
zap:2;
Output:
unsigned int bas:3,
hh:4,
jjj:8;
unsigned int baz:1,
fuz:5,
zap:2;
Submitted a patch regarding this in the differential for review.
-> Patch: https://reviews.llvm.org/D63062
There is a bug in the implementation where the indentation breaks when there is a comment in between the bitfields, which I kept aside for getting on to go with the next style and will fix it in the coming weeks.
About SortNetBSDIncludes
Clang-format has a native SortIncludes style, which sorts all the headers in alphabetical order where NetBSD headers follow a special order due to dependencies between each header. After discussing with my mentors and in the tech-toolchain we have come up with more precise order in which we should follow for headers:
- <sys/param.h>
- <sys/types.h>
- <sys/*> -- kernel headers
- <uvm/*> -- vm headers
- <net*/*> -- network protocol headers
- <*fs/*> -- filesystem headers
- <dev/*> -- device driver headers
- <protocols/.h>
- <machine/*>
- <[arch]/*>
- < /usr includes next>
- <paths.h>
- User include files
I have made more smarter version with Regex with necessary changes from the approach I had before which was a harcoded one, the patch for this will be up by this weekend.
Plan for the next phase
I was a bit behind the proposed schedule because understanding the LLVM source took more time than expected. I am confident that I will be speeding up my progress in the upcoming phase and complete as many styles possible. The final plan is to add support for NetBSD in clang-format which can be used by --style=NetBSD.
Summary
In the coming weeks, I will fix and add all the missing features to existing styles and new ones, and concentrate on optimizing and testing following all guidelines of NetBSD and make sure a formatted code doesn't cause a build fail.
Lastly, I would like to thank my wonderful mentors Michal and Christos for helping me to the process and guiding me whenever needed.
This report was written by Naveen Narayanan as part of Google Summer of Code 2019.
I have been working on porting Wine to amd64 on NetBSD as a GSoC 2019 project. Wine is a compatibility layer which allows running Microsoft Windows applications on POSIX-complaint operating systems. This report provides an overview of the progress of the project during the first coding period.
WINE on i386
Initially, when I started working on getting Wine-4.4 to build and run on NetBSD i386 the primary issue that I faced was Wine displaying black windows instead of UI, and this applied to any graphical program I tried running with Wine.
I suspected it , as it is related to graphics, to be an issue with the graphics driver or Xorg. Subsequently, I tried building modular Xorg, and I tried running Wine on it only to realize that Xorg being modular didn't affect it in the least. After having tried a couple of configurations, I realized that trying to hazard out every other probability is going to take an awful lot of time that I didn't have. This motivated me to bisect the repo using git, and find the first version of Wine which failed on NetBSD.
I started with the last version of Wine that worked on NetBSD which is 1.9.18. After two days of git bisect, I had the culprit. It was a commit that introduced a single clipboard manager thread per window station in Wine. Later, I found that pthread_create(3) calls were borked and didn't return anything. I tried to walk through the code to know where in pthreads lib was the program getting stuck at. Finally, I got a segfault.
I realized the address at which it segfaulted was identical to the address at which Wine has been throwing an unhandled page fault. I had a hunch that these two issues were some how correlated. After learning more about memory mapping and /proc, I was sure black windows was an effect of this unhandled page fault.
Eventually, I found that pthread_attr_setstack(3) was setting the guard size to 65536 bytes even though the man page said otherwise. And Wine relied on it not being set. This resulted in out-of-bound access which caused the unhandled page fault. After setting the guard size to 0 using pthread_attr_setguardsize(3), Wine started behaving fine.
I discussed about the patch with Wine devs, and they were happy to upstream it as long as it didn't cause any inadvertent issues on other platforms. Of course, who wouldn't want to play mario now?
WINE on amd64
Compiling Wine with 32 bit support is a bit tricky on amd64. I proceeded with the chroot approach as I wanted to know if Wine worked as promised on NetBSD, and if it didn't, to what degree, it required patching. So, as the first step, I compiled Wine on amd64 and it ran fine. The subsequent step was to build a chroot environment for i386. I corresponded with my mentors to learn more about compat32_netbsd and successfully built an i386 chroot. I had to compile Wine twice under i386 to have it inject the 32 bit components to 64 bit Wine for WoW64 support. And to my amazement, it worked!
Summary
I think Wine-4.4 is in pretty good shape as of right now, but packaging it is tricky especially since chroot isn't an option, as it is privileged. I have been writing compat32 packages for dependencies of Wine to have them crosscompiled on amd64. I shall be working on getting Wine crosscompiled for 32 bit support on amd64 during the next coding period. On a different note and a very important one, I would like to thank my mentors @christos, @leot, @maya, and @maxv for their valuable suggestions. From where I see it, I wouldn't have reached this phase if it weren't for them. I would also like to thank @kamil for his feedback and input.
This report was written by Naveen Narayanan as part of Google Summer of Code 2019.
I have been working on porting Wine to amd64 on NetBSD as a GSoC 2019 project. Wine is a compatibility layer which allows running Microsoft Windows applications on POSIX-complaint operating systems. This report provides an overview of the progress of the project during the first coding period.
WINE on i386
Initially, when I started working on getting Wine-4.4 to build and run on NetBSD i386 the primary issue that I faced was Wine displaying black windows instead of UI, and this applied to any graphical program I tried running with Wine.
I suspected it , as it is related to graphics, to be an issue with the graphics driver or Xorg. Subsequently, I tried building modular Xorg, and I tried running Wine on it only to realize that Xorg being modular didn't affect it in the least. After having tried a couple of configurations, I realized that trying to hazard out every other probability is going to take an awful lot of time that I didn't have. This motivated me to bisect the repo using git, and find the first version of Wine which failed on NetBSD.
I started with the last version of Wine that worked on NetBSD which is 1.9.18. After two days of git bisect, I had the culprit. It was a commit that introduced a single clipboard manager thread per window station in Wine. Later, I found that pthread_create(3) calls were borked and didn't return anything. I tried to walk through the code to know where in pthreads lib was the program getting stuck at. Finally, I got a segfault.
I realized the address at which it segfaulted was identical to the address at which Wine has been throwing an unhandled page fault. I had a hunch that these two issues were some how correlated. After learning more about memory mapping and /proc, I was sure black windows was an effect of this unhandled page fault.
Eventually, I found that pthread_attr_setstack(3) was setting the guard size to 65536 bytes even though the man page said otherwise. And Wine relied on it not being set. This resulted in out-of-bound access which caused the unhandled page fault. After setting the guard size to 0 using pthread_attr_setguardsize(3), Wine started behaving fine.
I discussed about the patch with Wine devs, and they were happy to upstream it as long as it didn't cause any inadvertent issues on other platforms. Of course, who wouldn't want to play mario now?
WINE on amd64
Compiling Wine with 32 bit support is a bit tricky on amd64. I proceeded with the chroot approach as I wanted to know if Wine worked as promised on NetBSD, and if it didn't, to what degree, it required patching. So, as the first step, I compiled Wine on amd64 and it ran fine. The subsequent step was to build a chroot environment for i386. I corresponded with my mentors to learn more about compat32_netbsd and successfully built an i386 chroot. I had to compile Wine twice under i386 to have it inject the 32 bit components to 64 bit Wine for WoW64 support. And to my amazement, it worked!
Summary
I think Wine-4.4 is in pretty good shape as of right now, but packaging it is tricky especially since chroot isn't an option, as it is privileged. I have been writing compat32 packages for dependencies of Wine to have them crosscompiled on amd64. I shall be working on getting Wine crosscompiled for 32 bit support on amd64 during the next coding period. On a different note and a very important one, I would like to thank my mentors @christos, @leot, @maya, and @maxv for their valuable suggestions. From where I see it, I wouldn't have reached this phase if it weren't for them. I would also like to thank @kamil for his feedback and input.
This report was written by Saurav Prakash as part of Google Summer of Code 2019.
My venture into the first phase of The Google Summer of Code is nearing an end. The experience was enriching in every dimension, and the learning exposure I was subjected to was genuinely worthwhile. Here is a brief report on the work I have performed during this coding period.
About HummingBoard Pulse
HummingBoard Pulse is an arm64 board developed by SolidRun, with iMX8 System-on-Chip, Dual/Quad core Cortex A53 processor. This project involves extending the GENERIC64 kernel to support iMX8 SoC.
Device Tree Files
We compare the compatible property of the root node in the device tree with the list of ARM platforms compiled into the kernel. We add the .dts file for imx8 which is compatible with the imx8mq.dtsi file adopted from the Linux mainline 5.1.4. At this early stage, nodes for uart, iomux and clock only were created.
Board Platform Code
The platform code provides SoC specific code needed early at boot. The arm_platform structure for imx8 is initialised here. It contains function pointers like .ap_attach_init_args, .ap_device_register, .ap_reset, .ap_delay, .ap_uart_freq, .ap_mpstart and a, platform_early_putchar_function so that we can send a character to UART.
Clock Driver
During the booting stage we only need to enable the uart clocks (IMX8MQ_CLK_UART*_ROOT) and its parents. This includes writing drivers for fixed clocks (OSC_25M, OSC_27M, OSC_32K, and more), fixed-factor clocks, divider clocks, mux clocks, composite clocks (IMX8MQ_CLK_UART*), gate clocks (IMX8MQ_CLK_UART*_ROOT), sccg-pll clocks, and frac-pll clocks.
UART Driver
The imx drivers are separated between "core driver" and "bus glue". <arch/arm/imx/imxuart.c> is a "core driver", and we need to write a fdt based "bus glue" for the custom bus, this non-fdt based imx code currently uses.
You can checkout the code from here.What's Next
The sccg-pll, frac-pll, composite clocks aren't complete yet. So the focus in the next weeks would be to complete the unfinished work as required for the board to be booting till root. Finalising the UART driver code. Modifying the drivers written now, to suit the needs later. And then moving on further to write iomux driver, and then a USB driver for a storage device to boot from.
Lastly I would like to thank my mentors, @jmcneill, @martin @Cryo for being a constant support and non-flickering guidance throughout the process.
This report was written by Saurav Prakash as part of Google Summer of Code 2019.
My venture into the first phase of The Google Summer of Code is nearing an end. The experience was enriching in every dimension, and the learning exposure I was subjected to was genuinely worthwhile. Here is a brief report on the work I have performed during this coding period.
About HummingBoard Pulse
HummingBoard Pulse is an arm64 board developed by SolidRun, with iMX8 System-on-Chip, Dual/Quad core Cortex A53 processor. This project involves extending the GENERIC64 kernel to support iMX8 SoC.
Device Tree Files
We compare the compatible property of the root node in the device tree with the list of ARM platforms compiled into the kernel. We add the .dts file for imx8 which is compatible with the imx8mq.dtsi file adopted from the Linux mainline 5.1.4. At this early stage, nodes for uart, iomux and clock only were created.
Board Platform Code
The platform code provides SoC specific code needed early at boot. The arm_platform structure for imx8 is initialised here. It contains function pointers like .ap_attach_init_args, .ap_device_register, .ap_reset, .ap_delay, .ap_uart_freq, .ap_mpstart and a, platform_early_putchar_function so that we can send a character to UART.
Clock Driver
During the booting stage we only need to enable the uart clocks (IMX8MQ_CLK_UART*_ROOT) and its parents. This includes writing drivers for fixed clocks (OSC_25M, OSC_27M, OSC_32K, and more), fixed-factor clocks, divider clocks, mux clocks, composite clocks (IMX8MQ_CLK_UART*), gate clocks (IMX8MQ_CLK_UART*_ROOT), sccg-pll clocks, and frac-pll clocks.
UART Driver
The imx drivers are separated between "core driver" and "bus glue". <arch/arm/imx/imxuart.c> is a "core driver", and we need to write a fdt based "bus glue" for the custom bus, this non-fdt based imx code currently uses.
You can checkout the code from here.What's Next
The sccg-pll, frac-pll, composite clocks aren't complete yet. So the focus in the next weeks would be to complete the unfinished work as required for the board to be booting till root. Finalising the UART driver code. Modifying the drivers written now, to suit the needs later. And then moving on further to write iomux driver, and then a USB driver for a storage device to boot from.
Lastly I would like to thank my mentors, @jmcneill, @martin @Cryo for being a constant support and non-flickering guidance throughout the process.