While debugging a Go application, I needed to troubleshoot its connections to a remote HTTPS endpoint.

Without fear, I started the mighty Wireshark and prepared to inspect some TCP packets!

The application I needed to debug is a small sample application I wrote, so I was in complete control of the source code. Quite handy, as we will demonstrate. The application is part of my study on the Elastic APM Agents communication protocol. Is a sample Go HTTP server with a couple of endpoints that emit all possible Elastic APM data, using simple instrumentation of HTTP function handlers.

The first step were to determine the IP address and port/s the application was connecting to. After starting the application, in this case named eapm, I used nestat and grep to check which connection were opened.

$ netstat -alpn | grep 
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
tcp        0      0 10.10.10.10:55932     1.2.3.4:443      ESTABLISHED 658594/./eapm
tcp        0      0 10.10.10.10:35122     1.2.3.4:443      ESTABLISHED 658594/./eapm
tcp6       0      0 :::8080               :::*             LISTEN      658594/./eapm

IPs have been redacted, but so far so good, we have 2 outgoing connection to the same IP address and a listening server by our small service.

With this information, we can start searching around in Wireshark. As connections go to port 443 I was expecting encrypted traffic, which meant looking at the content would be impossible without a decryption key.

As this is a known problem, there are workarounds! Being in control of the client or the server of a TLS connection, is possible to get the decryption keys needed for inspecting the traffic. In this case we are in control of the client, as is a library in a program we control the source of.

Wireshark allow decrypting traffic, as long as the Pre-Master-Secret is known.

SSLKEYLOGFILE is an environment variable that can be set tell browsers to save SSL decryption keys to a specific path on the file system. With can then pass that file to Wireshark and decrypt any TLS traffic.

So far so good, but Golang does not support it out of the box. But it supports (at least as for Golang 1.20.4) the KeyLogWriter option in the tls.Config struct (docs). It accepts a io.Writer, which is handily returned by os.Create.

To print the keys the only thing left was to find out where in the code to add this code:

fi, _ := os.Create("/tmp/foo")
cfg := tls.Config{KeyLogWriter: fi}

In Elastic APM Go Agent, the HTTP client is configured in transport/http.go. The function creates a tls.Config and returns it after some initialization.

The complete code for this little mischief was:

func newEnvTLSClientConfig() (*tls.Config, error) {
	verifyServerCert, err := configutil.ParseBoolEnv(envVerifyServerCert, true)
	if err != nil {
		return nil, err
	}
	tlsClientConfig := &tls.Config{InsecureSkipVerify: !verifyServerCert}
	// [...] no changes here
	fi, _ := os.Create("/tmp/foo")
	tlsClientConfig.KeyLogWriter = fi
	return tlsClientConfig, nil
}

With this done, the only thing left was to add the /tmp/foo file in Wireshark, under Edit > Preferences > Protocols > TLS > (Pre)-Master-Secret log filename and I were done.

Filtering for destination IP allowed to view all interesting traffic, which was now fully decrypted. Horray 🎉