Skip Navigation
Get a Demo
 
 
 
 
 
 
 
 
 
Resources Blog Testing and validation

Fuzzing Golang msgpack for fun and panic

How the Red Canary Product Security Team found a vulnerability in a Go programming language MessagePack implementation

Matt Schwager

The Red Canary Product Security team is always on the lookout to improve the security of the software we use. Our mission is to help secure the technical assets Red Canary produces, which includes the open source projects we rely on. It’s a win-win-win because we’re able to:

  • learn new and interesting application security techniques
  • improve our own internal security posture
  • improve the security posture of the open source ecosystem and broader tech community

This post will take you on a journey of fuzzing Golang MessagePack implementations, including a basic introduction to the concept of fuzzing, and will show you how we discovered a MessagePack denial-of-service vulnerability (GO-2022-0972, CVE-2022-41719).

So, what is fuzzing?

At the risk of oversimplifying, consider the following description of the fuzzing process:

  1. Generate garbled data.
  2. Run program with garbled data.
  3. Look for crashes.
  4. GOTO 1.

Fuzzing is often presented as a 5-dollar word for a 25-cent concept. It’s nothing more than generating weird data, sending it to a program, and looking for crashes. Crashes typically indicate a “security inflection point.” You’ve passed input the program didn’t expect, which may be exploited under the right conditions. Ultimately, it may not be a security vulnerability, but it’s a hint at where to look next.

So, the concept of fuzzing isn’t so scary after all, however, good fuzzers require quite a bit more smarts to do their thing. First, we should understand what applications and types of programs are good targets for fuzzing, then we can explore what it means to “generate garbled data.”

Fuzzing is particularly useful for applications that take in binary data. Good examples of fuzzable formats are binary JSON (BSON), MessagePack, ProtoBufs, compression formats, Base64 encoding, image and PDF applications, and HTTP/2 frames. In this post we’re going to focus on MessagePack because it’s a binary data format, and Red Canary uses it internally.

Screenshot of MessagePack format

It’s like JSON, but smaller… Credit: MessagePack website

Human-readable file formats like JSON, YAML, and INI can be fuzzed too, but binary data is typically the sweet spot in my experience. Format-aware fuzzers exist, but they’re often not as turnkey as binary data fuzzers. Binary data fuzzers are often better at generalizing the “generating garbled data” step than human-readable formats. This is because all the things that make formats human-readable (whitespace, curly braces, commas, etc.) are noise to a fuzzer. Garbling this data generally results in a parse error rather than a meaningful application state change. Flipping bits and bytes in binary data tends to introduce more meaningful state changes in an application.

You’ve probably noticed that we keep saying “often” or “tends to.” Fuzzing is more of an art than science. Intuition will be your guide, not dogma. Take everything we’re saying with a grain of salt–it’s simply what I’ve learned in my modest experience fuzzing applications.

Before moving on to the good stuff, I think it’s worth calling out and explaining two of the aforementioned “smarts” that fuzzers commonly employ: mutation and coverage-guided execution.

Mutation

Mutation is simply changing data. A fuzzer mutates data like an evil alien mutates into its final form before eating you. This is the “generate garbled data” step. A list of smart mutations will help a fuzzer maximize the weirdness it can inflict upon your application. Like turning a 42 it sees in the binary data into a 0, or a -42, or a 43, or a 4294967295 (2^32 – 1), then feeding it to your application. Or turning a string joe into a eoj, or a JOE, or a \0 (null byte), or a "" (empty string). Fuzzers leverage these kinds of common mutators to push your application to boundaries that may crash it.

Coverage-guided execution

Coverage-guided execution is exactly like code test coverage. It is typically instrumented at the compilation phase for compiled languages, or integrated into the interpreter or a library wrapper for interpreted languages. All this means is that there’s an additional step before running the system under test that allows the fuzzer to track what parts of the program it’s executed thus far. i.e., what have I executed, and what have I not? This helps the fuzzer explore new execution paths in the program and reveal additional code paths. New code paths mean new opportunities for crashes. Fuzzers will often combine coverage-guided execution with mutation to prioritize exploring more of the program. In other words, the fuzzer will focus on mutating specific parts of a binary data blob if it means it is increasing its coverage. Once the fuzzer believes it has sufficiently exhausted that execution path, it will backtrack and prioritize mutating other portions of the blob.

Both mutation and coverage-guided execution will be explored more concretely as we dive into a fuzzing example using the Go programming language.

Fuzzing in Golang

Developers using the Go programming language, or Golang, have strongly embraced fuzz testing as a means of improving software quality assurance and security properties. In fact, Golang has first-party support for fuzzing in the standard library, and supports fuzzing in CI with the -fuzztime flag. However, this post is going to wind back the clock a few years before we reach our final destination. We’re going to be using a tool called go-fuzz to perform our fuzz testing. All the same concepts mentioned above still apply, we’re just going to be using an older tool. Let’s call it a throwback, or vintage, or something.

go-fuzz saw its first commit in 2015, and eventually brought fuzzing into the mainstream in the Golang community. You can tell a lot about a security tool by its trophy case, and go-fuzz has an extensive one. This success, combined with Google’s focus on fuzzing, make first-party fuzzing support a natural fit in the Golang ecosystem. First-party fuzzing support was initially proposed all the way back in 2017, and formalized as a proposal in 2021, with many references to go-fuzz and its effectiveness. The culmination of all this hard work was first-party fuzzing support released in Go 1.18 in early 2022.

Why the history lesson, and why are we using an older tool? The target application did not support Go 1.18 when I started fuzzing it, and if it ain’t broke, don’t fix it. go-fuzz proved more than adequate for performing the task.

Fuzzing msgpack

Okay, enough with the theory, let’s do some fuzzing. The target application is a Golang msgpack implementation. The first thing that needs to be done when fuzzing is to generate, or otherwise acquire, a corpus of seed data. This seed data is what gets mutated before being sent to the target application. Fortunately, we can use this library to generate the seed corpus that will eventually lead to its demise.

Code snippets used in this post are taken from the full example provided here. We started with a representative sample of native Golang data types that will be encoded into the MessagePack format:

type Primitive struct {
     Nil *int
     Bool bool
     Int int
     Uint uint
     Float32 float32
     Float64 float64
     String string
     Bytes []byte
}

type Struct struct {
     Int int
     String string
}

type Container struct {
     Array []int
     Map map[string]int
     Nested Struct
}

primitives := []Primitive{
      Primitive{Nil: nil, Bool: true, Int: -100, Uint: 0, Float32: 3.1415, Float64: -3.1415, String: "foo", Bytes: []byte("bar")},
      Primitive{Nil: nil, Bool: false, Int: 0, Uint: 100, Float32: 3.1415, Float64: -3.1415, String: "", Bytes: []byte("")},
      Primitive{Nil: nil, Bool: false, Int: 1000, Uint: 1000, Float32: 3.1415, Float64: -3.1415, String: "\n", Bytes: []byte("\n")},
}

containers := []Container{
     Container{Array: []int{1, 2, 3, 4, 5}, Map: map[string]int{"1": 1, "2": 2, "3": 3}, Nested: Struct{Int: 42, String: "bar"}},
     Container{Array: []int{}, Map: map[string]int{}, Nested: Struct{Int: 0, String: ""}},
}

…

data, err := msgpack.Marshal(primitive)

The Marshal call turned a Golang object into a byte-encoded MessagePack buffer. We chose these data types and values intentionally to exercise as much of the msgpack library as possible. We then wrote these bytes to a MessagePack file on disk for later use when fuzzing:

filename := filepath.Join("corpus", fmt.Sprintf("primitive-%d.mp", i))
err = os.WriteFile(filename, data, 0644)
if err != nil {
     panic(err)
}

Remember that the target application does not support Go 1.18, so we used 1.17:

$ go version
go version go1.17.13 darwin/amd64

By running the Golang script mentioned above, we generated a corpus of seed data:

$ go run main.go -generate
$ tree corpus/
corpus/
├── container-0.mp
├── container-1.mp
├── primitive-0.mp
├── primitive-1.mp
└── primitive-2.mp

0 directories, 5 files

The script also took a -filename argument for performing an unmarshal and printing the results:

$ go run main.go -filename corpus/primitive-0.mp
map[Bool:true Bytes:[98 97 114] Float32:3.1415 Float64:-3.1415 Int:-100 Nil:<nil> String:foo Uint:0]


Before fuzzing, we needed to change package main to package msgpackfuzz in the script so that go-fuzz could fuzz it as a library function:

$ sed -i '' 's/package main/package msgpackfuzz/' main.go


The following is the key library call that allows go-fuzz to do its thing:

func Fuzz(data []byte) int {
     var r interface{}

     err := msgpack.Unmarshal(data, &r)
     if err != nil {
           return 0
     }

     return 1
}

This API is as specified by the go-fuzz framework. In go-fuzz, returning 1 increases the coverage-guided priority of the given input during subsequent fuzzing, returning -1 decreases priority of the given input, and returning 0 has no effect. In this case we lower the error case priority because we’re explicitly looking for unhandled errors (panics), not handled errors.

go-fuzz works by processing the corpus of seed data on disk, mutating its contents according to its supported mutators, and sending the resultant data to this fuzz function. The function then sends it immediately to the msgpack Unmarshal function. So the target here is really the msgpack.Unmarshal function.

Finally, let’s fuzz the target. go-fuzz-build performs the coverage instrumentation mentioned above to aid in coverage-guided execution. Let’s roll:

$ go-fuzz-build
$ go-fuzz
2022/11/28 15:07:29 workers: 12, corpus: 5 (3s ago), crashers: 0, restarts: 1/0, execs: 0 (0/sec), cover: 0, uptime: 3s
2022/11/28 15:07:32 workers: 12, corpus: 5 (6s ago), crashers: 0, restarts: 1/0, execs: 0 (0/sec), cover: 0, uptime: 6s
2022/11/28 15:07:35 workers: 12, corpus: 5 (9s ago), crashers: 1, restarts: 1/1, execs: 2032 (226/sec), cover: 143, uptime: 9s
2022/11/28 15:07:38 workers: 12, corpus: 5 (12s ago), crashers: 1, restarts: 1/1, execs: 3550 (296/sec), cover: 143, uptime: 12s
2022/11/28 15:07:41 workers: 12, corpus: 5 (15s ago), crashers: 2, restarts: 1/1, execs: 5106 (340/sec), cover: 146, uptime: 15s
2022/11/28 15:07:44 workers: 12, corpus: 5 (18s ago), crashers: 3, restarts: 1/1, execs: 6531 (363/sec), cover: 146, uptime: 18s
2022/11/28 15:07:47 workers: 12, corpus: 5 (21s ago), crashers: 3, restarts: 1/1, execs: 7973 (380/sec), cover: 146, uptime: 21s
2022/11/28 15:07:50 workers: 12, corpus: 5 (24s ago), crashers: 5, restarts: 1/1, execs: 9548 (398/sec), cover: 146, uptime: 24s
2022/11/28 15:07:53 workers: 12, corpus: 5 (27s ago), crashers: 6, restarts: 1/1, execs: 11238 (416/sec), cover: 173, uptime: 27s
2022/11/28 15:07:56 workers: 12, corpus: 5 (30s ago), crashers: 7, restarts: 1/1, execs: 12923 (431/sec), cover: 173, uptime: 30s

Almost immediately we started generating crashing cases. go-fuzz conveniently places crash cases in the “crashers” directory along with the crash output:

$ ls crashers/
crashers/0653a222aab82b5a03416902a97a186ff4642ed8
crashers/0653a222aab82b5a03416902a97a186ff4642ed8.output
crashers/0653a222aab82b5a03416902a97a186ff4642ed8.quoted
crashers/0c85503447c44f4996c936a04e54a7d2909fb86d
crashers/0c85503447c44f4996c936a04e54a7d2909fb86d.output
crashers/0c85503447c44f4996c936a04e54a7d2909fb86d.quoted

We can re-use the script’s -filename argument to see what happens during the crash, but first we must switch back to package main to allow running the script directly:

$ sed -i '' 's/package msgpackfuzz/package main/' main.go
$ go run main.go -filename crashers/0653a222aab82b5a03416902a97a186ff4642ed8
panic: runtime error: index out of range [3] with length 3

goroutine 1 [running]:
github.com/shamaton/msgpack/v2/internal/decoding.(*decoder).asInterface(0x10c7140, 0xc00009c990, 0xc000096af0)
~/go/pkg/mod/github.com/shamaton/msgpack/v2@v2.1.0/internal/decoding/interface.go:10 +0xbea
github.com/shamaton/msgpack/v2/internal/decoding.(*decoder).asInterface(0xc0000be000, 0xc00009a270, 0x10159cb) 
 
~/go/pkg/mod/github.com/shamaton/msgpack/v2@v2.1.0/internal/decoding/interface.go:126 +0x9b3
github.com/shamaton/msgpack/v2/internal/decoding.(*decoder).decode(0xc0000be000, {0x10c5d00, 0xc00009a270, 0x10}, 0x0)

~/go/pkg/mod/github.com/shamaton/msgpack/v2@v2.1.0/internal/decoding/decoding.go:307 +0xb85
github.com/shamaton/msgpack/v2/internal/decoding.Decode({0xc0000bc000, 0x3, 0x200}, {0x10c0040, 0xc00009a270}, 0x0)

~/go/pkg/mod/github.com/shamaton/msgpack/v2@v2.1.0/internal/decoding/decoding.go:32 +0x1a5
github.com/shamaton/msgpack/v2.Unmarshal(...)
      ~/go/pkg/mod/github.com/shamaton/msgpack/v2@v2.1.0/msgpack.go:24
main.main()
      ~/rc/msgpackfuzz/main.go:136 +0x1a5
exit status 2

We’ve generated a malformed MessagePack file that crashes the msgpack library!

So what actually happened here? The msgpack library crashed at this line of code. The fuzzer sent in an offset larger than the size of the interface object it’s supposed to represent. The resultant panic error message confirms this: “index out of range [3] with length 3.” The mutator likely took in an offset of 2, incremented it to 3, passed the data to the Unmarshal call, then a panic occurred. In the fixed code we can see that a bounds check was added to correctly handle these types of errors.

If an adversary can input a MessagePack file or byte stream of their choosing this would be a security vulnerability. In this case, this panic would likely lead to denial-of-service of the application. In situations that use the Golang unsafe library, this could lead to more dangerous bugs like memory corruption.

Your turn

This post exists to serve as an introduction to fuzzing, and some of the thought processes that go into fuzzing an application. Following the documentation of go-fuzz or Golang’s first-party fuzzing support is easy enough, so I wanted to highlight some of the finer points of beginning your fuzzing journey, such as generating a corpus of seed data, prioritizing certain fuzz inputs over others, and leveraging mutators and coverage-guided execution. Remember, fuzzing is often more art than science.

The culmination of this fuzzing work resulted in GO-2022-0972 and CVE-2022-41719. This vulnerability report was processed through the new Go Vulnerability Database. That means that you can run the excellent, new govulncheck tool against your Golang codebase to see if you’re vulnerable.

The official MessagePack website includes a list of Golang implementations. I briefly performed some fuzz testing against each, and shamaton/msgpack was the only vulnerable implementation I could find:

That doesn’t mean there aren’t bugs to be found in the other libraries. Perhaps I just didn’t look hard enough. Happy fuzzing!

Timeline

 

Atomic Red Team year in review

 

Validate your defenses with Atomic Test Harnesses for Linux and macOS

 

A brief history of security testing

 

Explore the new Atomic Red Team website

Subscribe to our blog

 
 
Back to Top