20.3 Using the gdbserver Program

gdbserver is a control program for Unix-like systems, which allows you to connect your program with a remote gdb via target remote—but without linking in the usual debugging stub.

gdbserver is not a complete replacement for the debugging stubs, because it requires essentially the same operating-system facilities that gdb itself does. In fact, a system that can run gdbserver to connect to a remote gdb could also run gdb locally! gdbserver is sometimes useful nevertheless, because it is a much smaller program than gdb itself. It is also easier to port than all of gdb, so you may be able to get started more quickly on a new system by using gdbserver. Finally, if you develop code for real-time systems, you may find that the tradeoffs involved in real-time operation make it more convenient to do as much development work as possible on another system, for example by cross-compiling. You can use gdbserver to make a similar choice for debugging.

gdb and gdbserver communicate via either a serial line or a TCP connection, using the standard gdb remote serial protocol.

Warning: gdbserver does not have any built-in security. Do not run gdbserver connected to any public network; a gdb connection to gdbserver provides access to the target system with the same privileges as the user running gdbserver.

20.3.1 Running gdbserver

Run gdbserver on the target system. You need a copy of the program you want to debug, including any libraries it requires. gdbserver does not need your program's symbol table, so you can strip the program if necessary to save space. gdb on the host system does all the symbol handling.

To use the server, you must tell it how to communicate with gdb; the name of your program; and the arguments for your program. The usual syntax is:

     target> gdbserver comm program [ args ... ]

comm is either a device name (to use a serial line), or a TCP hostname and portnumber, or - or stdio to use stdin/stdout of gdbserver. For example, to debug Emacs with the argument `foo.txt' and communicate with gdb over the serial port /dev/com1:

     target> gdbserver /dev/com1 emacs foo.txt

gdbserver waits passively for the host gdb to communicate with it.

To use a TCP connection instead of a serial line:

     target> gdbserver host:2345 emacs foo.txt

The only difference from the previous example is the first argument, specifying that you are communicating with the host gdb via TCP. The `host:2345' argument means that gdbserver is to expect a TCP connection from machine `host' to local TCP port 2345. (Currently, the `host' part is ignored.) You can choose any number you want for the port number as long as it does not conflict with any TCP ports already in use on the target system (for example, 23 is reserved for telnet).¹ You must use the same port number with the host gdb target remote command.

The stdio connection is useful when starting gdbserver with ssh:

     (gdb) target remote | ssh -T hostname gdbserver - hello

The `-T' option to ssh is provided because we don't need a remote pty, and we don't want escape-character handling. Ssh does this by default when a command is provided, the flag is provided to make it explicit. You could elide it if you want to.

Programs started with stdio-connected gdbserver have /dev/null for stdin, and stdout,stderr are sent back to gdb for display through a pipe connected to gdbserver. Both stdout and stderr use the same pipe.

20.3.1.1 Attaching to a Running Program

On some targets, gdbserver can also attach to running programs. This is accomplished via the --attach argument. The syntax is:

     target> gdbserver --attach comm pid

pid is the process ID of a currently running process. It isn't necessary to point gdbserver at a binary for the running process.

You can debug processes by name instead of process ID if your target has the pidof utility:

     target> gdbserver --attach comm `pidof program`

In case more than one copy of program is running, or program has multiple threads, most versions of pidof support the -s option to only return the first process ID.

20.3.1.2 Multi-Process Mode for gdbserver

When you connect to gdbserver using target remote, gdbserver debugs the specified program only once. When the program exits, or you detach from it, gdb closes the connection and gdbserver exits.

If you connect using target extended-remote, gdbserver enters multi-process mode. When the debugged program exits, or you detach from it, gdb stays connected to gdbserver even though no program is running. The run and attach commands instruct gdbserver to run or attach to a new program. The run command uses set remote exec-file (see set remote exec-file) to select the program to run. Command line arguments are supported, except for wildcard expansion and I/O redirection (see Arguments).

To start gdbserver without supplying an initial command to run or process ID to attach, use the --multi command line option. Then you can connect using target extended-remote and start the program you want to debug.

In multi-process mode gdbserver does not automatically exit unless you use the option --once. You can terminate it by using monitor exit (see Monitor Commands for gdbserver). Note that the conditions under which gdbserver terminates depend on how gdb connects to it (target remote or target extended-remote). The --multi option to gdbserver has no influence on that.

20.3.1.3 TCP port allocation lifecycle of gdbserver

This section applies only when gdbserver is run to listen on a TCP port.

gdbserver normally terminates after all of its debugged processes have terminated in target remote mode. On the other hand, for target extended-remote, gdbserver stays running even with no processes left. gdb normally terminates the spawned debugged process on its exit, which normally also terminates gdbserver in the target remote mode. Therefore, when the connection drops unexpectedly, and gdb cannot ask gdbserver to kill its debugged processes, gdbserver stays running even in the target remote mode.

When gdbserver stays running, gdb can connect to it again later. Such reconnecting is useful for features like disconnected tracing. For completeness, at most one gdb can be connected at a time.

By default, gdbserver keeps the listening TCP port open, so that additional connections are possible. However, if you start gdbserver with the --once option, it will stop listening for any further connection attempts after connecting to the first gdb session. This means no further connections to gdbserver will be possible after the first one. It also means gdbserver will terminate after the first connection with remote gdb has closed, even for unexpectedly closed connections and even in the target extended-remote mode. The --once option allows reusing the same port number for connecting to multiple instances of gdbserver running on the same host, since each instance closes its port after the first connection.

20.3.1.4 Other Command-Line Arguments for gdbserver

The --debug option tells gdbserver to display extra status information about the debugging process. The --remote-debug option tells gdbserver to display remote protocol debug output. These options are intended for gdbserver development and for bug reports to the developers.

The --wrapper option specifies a wrapper to launch programs for debugging. The option should be followed by the name of the wrapper, then any command-line arguments to pass to the wrapper, then -- indicating the end of the wrapper arguments.

gdbserver runs the specified wrapper program with a combined command line including the wrapper arguments, then the name of the program to debug, then any arguments to the program. The wrapper runs until it executes your program, and then gdb gains control.

You can use any program that eventually calls execve with its arguments as a wrapper. Several standard Unix utilities do this, e.g. env and nohup. Any Unix shell script ending with exec "$@" will also work.

For example, you can use env to pass an environment variable to the debugged program, without setting the variable in gdbserver's environment:

     $ gdbserver --wrapper env LD_PRELOAD=libtest.so -- :2222 ./testprog

20.3.2 Connecting to gdbserver

Run gdb on the host system.

First make sure you have the necessary symbol files. Load symbols for your application using the file command before you connect. Use set sysroot to locate target libraries (unless your gdb was compiled with the correct sysroot using --with-sysroot).

The symbol file and target libraries must exactly match the executable and libraries on the target, with one exception: the files on the host system should not be stripped, even if the files on the target system are. Mismatched or missing files will lead to confusing results during debugging. On gnu/Linux targets, mismatched or missing files may also prevent gdbserver from debugging multi-threaded programs.

Connect to your target (see Connecting to a Remote Target). For TCP connections, you must start up gdbserver prior to using the target remote command. Otherwise you may get an error whose text depends on the host system, but which usually looks something like `Connection refused'. Don't use the load command in gdb when using gdbserver, since the program is already on the target.

20.3.3 Monitor Commands for gdbserver

During a gdb session using gdbserver, you can use the monitor command to send special requests to gdbserver. Here are the available commands.

monitor help

List the available monitor commands.

monitor set debug 0

monitor set debug 1

Disable or enable general debugging messages.

monitor set remote-debug 0

monitor set remote-debug 1

Disable or enable specific debugging messages associated with the remote protocol (see Remote Protocol).

monitor set libthread-db-search-path [PATH]

When this command is issued, path is a colon-separated list of directories to search for libthread_db (see set libthread-db-search-path). If you omit path, `libthread-db-search-path' will be reset to its default value.

The special entry `$pdir' for `libthread-db-search-path' is not supported in gdbserver.

monitor exit

Tell gdbserver to exit immediately. This command should be followed by disconnect to close the debugging session. gdbserver will detach from any attached processes and kill any processes it created. Use monitor exit to terminate gdbserver at the end of a multi-process mode debug session.

20.3.4 Tracepoints support in gdbserver

On some targets, gdbserver supports tracepoints, fast tracepoints and static tracepoints.

For fast or static tracepoints to work, a special library called the in-process agent (IPA), must be loaded in the inferior process. This library is built and distributed as an integral part of gdbserver. In addition, support for static tracepoints requires building the in-process agent library with static tracepoints support. At present, the UST (LTTng Userspace Tracer, http://lttng.org/ust) tracing engine is supported. This support is automatically available if UST development headers are found in the standard include path when gdbserver is built, or if gdbserver was explicitly configured using --with-ust to point at such headers. You can explicitly disable the support using --with-ust=no.

There are several ways to load the in-process agent in your program:

Specifying it as dependency at link time

You can link your program dynamically with the in-process agent library. On most systems, this is accomplished by adding -linproctrace to the link command.

Using the system's preloading mechanisms

You can force loading the in-process agent at startup time by using your system's support for preloading shared libraries. Many Unixes support the concept of preloading user defined libraries. In most cases, you do that by specifying LD_PRELOAD=libinproctrace.so in the environment. See also the description of gdbserver's --wrapper command line option.

Using gdb to force loading the agent at run time

On some systems, you can force the inferior to load a shared library, by calling a dynamic loader function in the inferior that takes care of dynamically looking up and loading a shared library. On most Unix systems, the function is dlopen. You'll use the call command for that. For example:

          (gdb) call dlopen ("libinproctrace.so", ...)
     

Note that on most Unix systems, for the dlopen function to be available, the program needs to be linked with -ldl.

On systems that have a userspace dynamic loader, like most Unix systems, when you connect to gdbserver using target remote, you'll find that the program is stopped at the dynamic loader's entry point, and no shared library has been loaded in the program's address space yet, including the in-process agent. In that case, before being able to use any of the fast or static tracepoints features, you need to let the loader run and load the shared libraries. The simplest way to do that is to run the program to the main procedure. E.g., if debugging a C or C++ program, start gdbserver like so:

     $ gdbserver :9999 myprogram

Start GDB and connect to gdbserver like so, and run to main:

     $ gdb myprogram
     (gdb) target remote myhost:9999
     0x00007f215893ba60 in ?? () from /lib64/ld-linux-x86-64.so.2
     (gdb) b main
     (gdb) continue

The in-process tracing agent library should now be loaded into the process; you can confirm it with the info sharedlibrary command, which will list libinproctrace.so as loaded in the process. You are now ready to install fast tracepoints, list static tracepoint markers, probe static tracepoints markers, and start tracing.