Fuzzing Golang msgpack for fun and panic

The Red Canary Product Security team is always on the lookout to improve the security of the software we use. Our mission is to help secure the technical assets Red Canary produces, which includes the open source projects we rely on. It’s a win-win-win because we’re able to:

• learn new and interesting application security techniques
• improve our own internal security posture
• improve the security posture of the open source ecosystem and broader tech community

This post will take you on a journey of fuzzing Golang MessagePack implementations, including a basic introduction to the concept of fuzzing, and will show you how we discovered a MessagePack denial-of-service vulnerability (GO-2022-0972, CVE-2022-41719).

So, what is fuzzing?

At the risk of oversimplifying, consider the following description of the fuzzing process:

1. Generate garbled data.
2. Run program with garbled data.
3. Look for crashes.
4. GOTO 1.

Fuzzing is often presented as a 5-dollar word for a 25-cent concept. It’s nothing more than generating weird data, sending it to a program, and looking for crashes. Crashes typically indicate a “security inflection point.” You’ve passed input the program didn’t expect, which may be exploited under the right conditions. Ultimately, it may not be a security vulnerability, but it’s a hint at where to look next.

So, the concept of fuzzing isn’t so scary after all, however, good fuzzers require quite a bit more smarts to do their thing. First, we should understand what applications and types of programs are good targets for fuzzing, then we can explore what it means to “generate garbled data.”

Fuzzing is particularly useful for applications that take in binary data. Good examples of fuzzable formats are binary JSON (BSON), MessagePack, ProtoBufs, compression formats, Base64 encoding, image and PDF applications, and HTTP/2 frames. In this post we’re going to focus on MessagePack because it’s a binary data format, and Red Canary uses it internally.

It’s like JSON, but smaller… Credit: MessagePack website

Human-readable file formats like JSON, YAML, and INI can be fuzzed too, but binary data is typically the sweet spot in my experience. Format-aware fuzzers exist, but they’re often not as turnkey as binary data fuzzers. Binary data fuzzers are often better at generalizing the “generating garbled data” step than human-readable formats. This is because all the things that make formats human-readable (whitespace, curly braces, commas, etc.) are noise to a fuzzer. Garbling this data generally results in a parse error rather than a meaningful application state change. Flipping bits and bytes in binary data tends to introduce more meaningful state changes in an application.

You’ve probably noticed that we keep saying “often” or “tends to.” Fuzzing is more of an art than science. Intuition will be your guide, not dogma. Take everything we’re saying with a grain of salt–it’s simply what I’ve learned in my modest experience fuzzing applications.

Before moving on to the good stuff, I think it’s worth calling out and explaining two of the aforementioned “smarts” that fuzzers commonly employ: mutation and coverage-guided execution.

Mutation

Mutation is simply changing data. A fuzzer mutates data like an evil alien mutates into its final form before eating you. This is the “generate garbled data” step. A list of smart mutations will help a fuzzer maximize the weirdness it can inflict upon your application. Like turning a 42 it sees in the binary data into a 0, or a -42, or a 43, or a 4294967295 (2^32 – 1), then feeding it to your application. Or turning a string joe into a eoj, or a JOE, or a \0 (null byte), or a "" (empty string). Fuzzers leverage these kinds of common mutators to push your application to boundaries that may crash it.

Coverage-guided execution

Coverage-guided execution is exactly like code test coverage. It is typically instrumented at the compilation phase for compiled languages, or integrated into the interpreter or a library wrapper for interpreted languages. All this means is that there’s an additional step before running the system under test that allows the fuzzer to track what parts of the program it’s executed thus far. i.e., what have I executed, and what have I not? This helps the fuzzer explore new execution paths in the program and reveal additional code paths. New code paths mean new opportunities for crashes. Fuzzers will often combine coverage-guided execution with mutation to prioritize exploring more of the program. In other words, the fuzzer will focus on mutating specific parts of a binary data blob if it means it is increasing its coverage. Once the fuzzer believes it has sufficiently exhausted that execution path, it will backtrack and prioritize mutating other portions of the blob.

Both mutation and coverage-guided execution will be explored more concretely as we dive into a fuzzing example using the Go programming language.

Fuzzing in Golang

Developers using the Go programming language, or Golang, have strongly embraced fuzz testing as a means of improving software quality assurance and security properties. In fact, Golang has first-party support for fuzzing in the standard library, and supports fuzzing in CI with the -fuzztime flag. However, this post is going to wind back the clock a few years before we reach our final destination. We’re going to be using a tool called go-fuzz to perform our fuzz testing. All the same concepts mentioned above still apply, we’re just going to be using an older tool. Let’s call it a throwback, or vintage, or something.

go-fuzz saw its first commit in 2015, and eventually brought fuzzing into the mainstream in the Golang community. You can tell a lot about a security tool by its trophy case, and go-fuzz has an extensive one. This success, combined with Google’s focus on fuzzing, make first-party fuzzing support a natural fit in the Golang ecosystem. First-party fuzzing support was initially proposed all the way back in 2017, and formalized as a proposal in 2021, with many references to go-fuzz and its effectiveness. The culmination of all this hard work was first-party fuzzing support released in Go 1.18 in early 2022.

Why the history lesson, and why are we using an older tool? The target application did not support Go 1.18 when I started fuzzing it, and if it ain’t broke, don’t fix it. go-fuzz proved more than adequate for performing the task.

Fuzzing msgpack

Okay, enough with the theory, let’s do some fuzzing. The target application is a Golang msgpack implementation. The first thing that needs to be done when fuzzing is to generate, or otherwise acquire, a corpus of seed data. This seed data is what gets mutated before being sent to the target application. Fortunately, we can use this library to generate the seed corpus that will eventually lead to its demise.

Code snippets used in this post are taken from the full example provided here. We started with a representative sample of native Golang data types that will be encoded into the MessagePack format:

type Primitive struct {
     Nil *int
     Bool bool
     Int int
     Uint uint
     Float32 float32
     Float64 float64
     String string
     Bytes []byte
}

type Struct struct {
     Int int
     String string
}

type Container struct {
     Array []int
     Map map[string]int
     Nested Struct
}

primitives := []Primitive{
      Primitive{Nil: nil, Bool: true, Int: -100, Uint: 0, Float32: 3.1415, Float64: -3.1415, String: "foo", Bytes: []byte("bar")},
      Primitive{Nil: nil, Bool: false, Int: 0, Uint: 100, Float32: 3.1415, Float64: -3.1415, String: "", Bytes: []byte("")},
      Primitive{Nil: nil, Bool: false, Int: 1000, Uint: 1000, Float32: 3.1415, Float64: -3.1415, String: "\n", Bytes: []byte("\n")},
}

containers := []Container{
     Container{Array: []int{1, 2, 3, 4, 5}, Map: map[string]int{"1": 1, "2": 2, "3": 3}, Nested: Struct{Int: 42, String: "bar"}},
     Container{Array: []int{}, Map: map[string]int{}, Nested: Struct{Int: 0, String: ""}},
}

…

data, err := msgpack.Marshal(primitive)