ObjFile on stripped binaries

[stevemk14ebr]

On GO binaries with stripped .symtab sections the objfile tool fails to located the pclntab and fails to print embedded symbols. The table can still be found however by byte scanning the .text section of the binary for the header magic for the table:

go/src/debug/gosym/pclntab.go

Lines 169 to 170 in 89f687d

	const go12magic = 0xfffffffb
	const go116magic = 0xfffffffa

The gosym NewTable method does not require the .symtab section to correctly parse the pclntab table and is still useful for this case of stripped binaries. It may simply be passed an empty byte array as it's first argument:

go/src/cmd/internal/objfile/objfile.go

Line 152 in 2d0258d

return gosym.NewTable(symtab, gosym.NewLineTable(pclntab, textStart))

I propose that a simple byte scan be added in objfile.go if the .pcln() method fails to locate the table by the standard symbols lookup. By falling back to byte scanning the tool can still print much useful information.

[cherrymui]

Could you clarify what the problem is?

the objfile tool fails to located the pclntab and fails to print embedded symbols.

What is "the objfile tool"? go tool objdump? And what is "embedded symbols"?

go/src/cmd/internal/objfile/objfile.go
return gosym.NewTable(symtab, gosym.NewLineTable(pclntab, textStart))

That symtab is the .gosymtab section, instead of .symtab section. How do you strip the binary so it has strips that section? Also, that section is actually empty anyway in recent versions of Go.

Could you clarify what program/command you run that you see unexpected result? Thanks.

[stevemk14ebr]

Sure, sorry my wording wasn't exact. The issue is this:

go tool objdump ./mtisfun_win_stripped
objdump: disassemble ./fmtisfun_win_stripped: no runtime.pclntab symbol found

The binary in question has its .symtab section zeroed out (sometimes this is referred to as stripping symbols):

The reason this matters is because internally the objfile package attempts to resolve the pclntab by using this .symtab section:

go/src/cmd/internal/objfile/objfile.go

Line 148 in 2d0258d

textStart, symtab, pclntab, err := e.raw.pcln()

which calls (for PE's)

go/src/cmd/internal/objfile/pe.go

Lines 111 to 119 in 2d0258d

	if pclntab, err = loadPETable(f.pe, "runtime.pclntab", "runtime.epclntab"); err != nil {
	// We didn't find the symbols, so look for the names used in 1.3 and earlier.
	// TODO: Remove code looking for the old symbols when we no longer care about 1.3.
	var err2 error
	if pclntab, err2 = loadPETable(f.pe, "pclntab", "epclntab"); err2 != nil {
	return 0, nil, nil, err
	}
	}
	if symtab, err = loadPETable(f.pe, "runtime.symtab", "runtime.esymtab"); err != nil {

That symbol resolution fails, and therefore no golang symbols are parsed and the objdump tool faults. This is a poor failure case though as the pclntab section is (usually) still there! It can be located by performing a bytescan over the .text section for the magic pclntab magic. When this byte scan is done the objdump tool is able to work perfectly fine even on binaries where symbols are stripped like this.

[stevemk14ebr]

The patch is fairly simple, in objfile.go PCLineTable():

	textStart, symtab, pclntab, err := e.raw.pcln()
	if err != nil {
		textStart, textData, err := e.raw.text()
		// little endian go12magic
		pclntab_idx := bytes.Index(textData, []byte("\xFB\xFF\xFF\xFF\x00\x00"))
		if pclntab_idx == -1 {
			// big endian go12magic
			pclntab_idx = bytes.Index(textData, []byte("\xFF\xFF\xFF\xFB\x00\x00"))
			if pclntab_idx == -1 {
				// little endian go116magic
				pclntab_idx = bytes.Index(textData, []byte("\xFA\xFF\xFF\xFF\x00\x00"))
				if pclntab_idx == -1 {
					// big endian go116magic
					pclntab_idx = bytes.Index(textData, []byte("\xFF\xFF\xFF\xFA\x00\x00"))
					if pclntab_idx == -1 {
						return nil, err
					}
				}
			}
		}

		// The symbol table of the PE is stripped, but we still have the pclntab!
		fmt.Printf("found pclntab via signature! %X", textStart+uint64(pclntab_idx))

                // we don't know the epclntab but that's fine, the parser won't read extra!
		return gosym.NewTable([]byte{}, gosym.NewLineTable(textData[pclntab_idx:], textStart))
	}

As an aside, if the members of the LineTable:

go/src/debug/gosym/pclntab.go

Lines 40 to 69 in 89f687d

	type LineTable struct {
	Data []byte
	PC uint64
	Line int
	// This mutex is used to keep parsing of pclntab synchronous.
	mu sync.Mutex
	// Contains the version of the pclntab section.
	version version
	// Go 1.2/1.16 state
	binary binary.ByteOrder
	quantum uint32
	ptrsize uint32
	funcnametab []byte
	cutab []byte
	funcdata []byte
	functab []byte
	nfunctab uint32
	filetab []byte
	pctab []byte // points to the pctables.
	nfiletab uint32
	funcNames map[uint32]string // cache the function names
	strings map[uint32]string // interned substrings of Data, keyed by offset
	// fileMap varies depending on the version of the object file.
	// For ver12, it maps the name to the index in the file table.
	// For ver116, it maps the name to the offset in filetab.
	fileMap map[string]uint32
	}

and the symtab Table:

go/src/debug/gosym/symtab.go

Line 127 in 89f687d

go12line *LineTable // Go 1.2 line number table

could be capitalized to be exported I would appreciate it! I use this for some internal security tooling and have to maintain a set of patches to export these internal details.

[randall77]

How exactly was your symbol table stripped? Can you show us the exact commands you used starting with building your binary?
What platform are you on?

I'm not a fan of finding sections by looking for magic numbers. Magic numbers are useful if you know where they are and are just verifying them (e.g. at the start of file), but looking for them among a large corpus has a serious false positive problem. For example, the instruction sequence:

	movq	0xfbffffff(%rcx), %rcx
	addb	%al, (%rax)

Has one of your magic sequences in it.

[stevemk14ebr]

I work in security and often deal with malicious samples that are stripped by bad actors, this isn't a process i do. However you can generate a binary without symbols via: go build -ldflags="-s -w" <files>. I agree that searching for magic bytes is not ideal, of course there are false positives. This particular magic isn't exactly the greatest, i would argue it needs more entropy and should probably be a little longer, but it's possible to deal with false positives like this as the parser will simply fail and we can try all candidates until one works. It's also know that this section is within the .text section which avoid a whole host of problems with data colliding.

A future version of go could be really nice if it added a magic for the moduledata structure too :)

[randall77]

What platform are you on?

Still need an answer to this.

However you can generate a binary without symbols via: go build -ldflags="-s -w"

Works fine for me (darwin/amd64, go1.13.6):

$ go build -ldflags="-s -w" hello.go
$ go tool objdump hello
... lots of disassembly ...

Why use go tool objdump instead of binutil's objdump for this? If it was generated by a malicious actor, how do you even know the stripped binary is a Go binary?

[cherrymui]

On ELF, we look for ".gosymtab" section (NOT ".symtab"), instead of "runtime.symtab" symbol ( https://tip.golang.org/src/cmd/internal/objfile/elf.go#L67 ). Can we do the same for PE?

I also don't think that looking for magic numbers are the right answer here.

[stevemk14ebr]

@randall77 I am on windows/linux but it's kind of irrelevant here as the symbol table (.gosymtab, .symtab, whatever) can be stripped and the pclntab still be present. This issue highlights that the symbol table (.gosymtab, .symtab, whatever) is used to locate the .pclntab section which can fail. This failure to locate the pclntab doesn't mean it isn't there however, and we can find it to still make objdump useful.

Why use go tool objdump instead of binutil's objdump for this

I specifically care about getting the pclntab metadata here as it gives function names and start/stop boundaries.

how do you even know the stripped binary is a Go binary

as a reverse engineer it is incredibly easily to tell by just looking at the assembly patterns in IDA PRO

there's at least two unhandled failure cases:

1. When the symbol table itself is fully missing, ex: entire .gosymtab (ELF) or .symtab (PE) is just straight up missing
2. When the symbol table is there, but zero/malformed and the symbols cannot be located within these tables. You will get different errors for both of these cases

Now, the symbol table being corrupt/stomped doesn't matter as the pclntab is independent of that.

[cherrymui]

Please don't confuse ".gosymtab" and ".symtab". They are always two different sections, on any platform (PE/ELF/Mach-O/etc.). What strip tool do you use? I don't think it should strip ".gosymtab" section on any platform. (However, the section should be empty anyway.)

"runtime.symtab" is a symbol, which should be at the beginning of the ".gosymtab" section. Please be careful with the names. Thanks.

[randall77]

@randall77 I am on windows/linux but it's kind of irrelevant here

It's very relevant. Now that I know "windows", I can reproduce:

$ GOOS=windows go build -ldflags="-s -w" hello.go
$ go tool objdump hello.exe
objdump: disassemble hello.exe: no runtime.pclntab symbol found

[stevemk14ebr]

@cherrymui I am simply using the linker flags i specified earlier. I am sorry if I am not using the correct terms, I am referring to .gosymtab as a section for linux elf, and .symtab as a section for PE. These are what i refer to as the symbol tab as it seems GO uses them to store symbols inside of (such as to locate runtime.pclntab)

@randall77 same happens for linux but it fails finding the symbol table entirely:

go tool objdump hello
objdump: disassemble hello: no symbol section

[randall77]

I don't think I understand the motivation for fixing this at all. For the Go project generally, that is.
A user tried to remove symbolic information from their binary. We should respect that wish and not intentionally resurrect that symbolic information despite their wishes.

The fact that they are a malicious user is unfortunate, but not IMO enough motivation to have our general-purpose tools sidestep someone's attempt to remove symbolic information.
Besides, this is an arms race. As soon as we implement this, the obfuscator will notice and just zero out more stuff. I don't think we want to go down that road. Consider yourself lucky that our tools don't show that symbolic information by default but it is still hidden in there somewhere for you.

[randall77]

(It might be better to have go tool objdump dump the whole text section as one big assembly function instead of erroring out, though.)

[cherrymui]

I am referring to .gosymtab as a section for linux elf, and .symtab as a section for PE

They should not be the same section. They should serve completely different purpose. If they are, it seems a bug in the linker.

to locate runtime.pclntab

What I'm proposing is that we should not try to locate runtime.pclntab at all. We should put the pclntab in a separate section and locate that section instead, which we do for ELF and Mach-O.