binutils-gdb/gdbserver
Pedro Alves 028a46039a gdbserver: handle running threads in qXfer:threads:read
On some systems, the gdb.multi/multi-target.exp testcase occasionally
fails like so:

 Running src/gdb/testsuite/gdb.multi/multi-target.exp ...
 FAIL: gdb.multi/multi-target.exp: info-inferiors: multi_process=on: inferior 1: info connections
 FAIL: gdb.multi/multi-target.exp: info-inferiors: multi_process=on: inferior 1: info inferiors
 FAIL: gdb.multi/multi-target.exp: info-inferiors: multi_process=on: inferior 2: info connections
 FAIL: gdb.multi/multi-target.exp: info-inferiors: multi_process=on: inferior 2: info inferiors
 FAIL: gdb.multi/multi-target.exp: info-inferiors: multi_process=on: inferior 3: inferior 3
 ... many more cascading fails.

The problem starts when the testcase runs an inferior against GDBserver:

 (gdb) run
 Starting program: build/gdb/testsuite/outputs/gdb.multi/multi-target/multi-target
 Reading /lib64/ld-linux-x86-64.so.2 from remote target...
 warning: File transfers from remote targets can be slow. Use "set sysroot" to access files locally instead.
 Reading /lib64/ld-linux-x86-64.so.2 from remote target...
 Reading /lib64/ld-2.31.so from remote target...
 Reading /lib64/.debug/ld-2.31.so from remote target...
 Reading /usr/lib/debug//lib64/ld-2.31.so from remote target...
 Reading /usr/lib/debug/lib64//ld-2.31.so from remote target...
 Reading target:/usr/lib/debug/lib64//ld-2.31.so from remote target...
 Reading /lib/x86_64-linux-gnu/libpthread.so.0 from remote target...
 Reading /lib/x86_64-linux-gnu/libc.so.6 from remote target...
 Reading /lib/x86_64-linux-gnu/libc-2.31.so from remote target...
 Reading /lib/x86_64-linux-gnu/.debug/libc-2.31.so from remote target...
 Reading /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.31.so from remote target...
 Reading /usr/lib/debug//lib/x86_64-linux-gnu/libc-2.31.so from remote target...
 Remote connection closed
 ...

Note the "Remote connection closed" message.  That means GDBserver
exited abruptly.

I traced it down to the fact that GDB fetches the thread list from
GDBserver while the main thread of the process is still running.  On
my main system where I wrote the testcase, I have not observed the
failure because it is slow enough that the thread stops before
GDBserver fetches the thread list in the problem scenario which I'll
describe below.

With some --remote-debug logging from GDBserver side, we see the last
packets before the connection closes:

 ...
 getpkt ("vCont;c");  [no ack sent]
 putpkt ("$OK#9a"); [noack mode]
 getpkt ("Tp10f9a.10f9a");  [no ack sent]
 putpkt ("$OK#9a"); [noack mode]
 getpkt ("Hgp0.0");  [no ack sent]
 putpkt ("$OK#9a"); [noack mode]
 getpkt ("qXfer:threads:read::0,1000");  [no ack sent]

Note the vCont;c , which sets the program running, and then a
qXfer:threads:read packet at the end.

The problem happens when the thread list refresh (qXfer:threads:read)
is sent just while the main thread is running and it still hasn't
initialized its libpthread id internally.  In that state, the main
thread's lwp will remain with the thread_known flag clear.  See in
find_one_thread:

  /* If the new thread ID is zero, a final thread ID will be available
     later.  Do not enable thread debugging yet.  */
  if (ti.ti_tid == 0)
    return 0;

Now, back in server.cc, to handle the qXfer:threads:read, we reach
handle_qxfer_threads -> handle_qxfer_threads_proper, and the latter
then calls handle_qxfer_threads_worker for each known thread.  In
handle_qxfer_threads_worker, we call target_thread_handle.  This ends
up in thread_db_thread_handle, here:

  if (!lwp->thread_known && !find_one_thread (thread->id))
    return false;

Since the thread ID isn't known yet, we call find_one_thread.  This
calls into libthread_db.so, which accesses memory.  Because the
current thread is running, that fails and we throw an error, here:

  /* Get information about this thread.  */
  err = thread_db->td_ta_map_lwp2thr_p (thread_db->thread_agent, lwpid, &th);
  if (err != TD_OK)
    error ("Cannot get thread handle for LWP %d: %s",
	   lwpid, thread_db_err_str (err));

The current design is that whenever GDB-facing packets/requests need
to accesses memory, server.cc is supposed to prepare the target for
the access.  See gdb_read_memory / gdb_write_memory.  This preparation
means pausing threads if in non-stop mode (someday we could lift this
requirement, but we will still need to pause to access registers or do
other related ptrace accesses like PTRACE_GET_THREAD_AREA).  Note that
the multi-target.exp testcase forces "maint set target-non-stop on".

So the fix here is to prepare the target to access memory when
handling qXfer:threads:read too.

gdbserver/ChangeLog:

	* inferiors.cc (switch_to_process): New, moved here from
	thread-db.cc, and made extern.
	* inferiors.h (switch_to_process): Declare.
	* server.cc: Include "gdbsupport/scoped_restore.h".
	(handle_qxfer_threads_proper): Now returns bool.  Prepare to
	access memory around target calls.
	(handle_qxfer_threads): Handle errors.
	* thread-db.cc (switch_to_process): Moved to inferiors.cc.
2020-07-22 12:32:53 +01:00
..
.dir-locals.el
.gitattributes
.gitignore
acinclude.m4
aclocal.m4
ax.cc [gdbserver] Fix Wlto-type-mismatch for debug_agent 2020-06-12 18:36:56 +02:00
ax.h [gdbserver] Add missing include of gdbsupport/agent.h 2020-06-29 12:14:10 +02:00
ChangeLog gdbserver: handle running threads in qXfer:threads:read 2020-07-22 12:32:53 +01:00
config.in gdb, gdbserver: remove configure check for fs_base/gs_base in user_regs_struct 2020-04-27 10:47:50 -04:00
configure gdbserver: remove support for Neutrino 2020-06-12 16:06:41 -04:00
configure.ac gdbserver: remove support for Neutrino 2020-06-12 16:06:41 -04:00
configure.srv gdbserver: remove support for ARM/WinCE 2020-06-12 16:06:45 -04:00
debug.cc
debug.h
dll.cc
dll.h
fork-child.cc
gdb_proc_service.h
gdbreplay.cc
gdbthread.h
hostio-errno.cc
hostio.cc
hostio.h
i387-fp.cc
i387-fp.h
inferiors.cc gdbserver: handle running threads in qXfer:threads:read 2020-07-22 12:32:53 +01:00
inferiors.h gdbserver: handle running threads in qXfer:threads:read 2020-07-22 12:32:53 +01:00
linux-aarch32-low.cc
linux-aarch32-low.h
linux-aarch32-tdesc.cc
linux-aarch32-tdesc.h
linux-aarch64-ipa.cc
linux-aarch64-low.cc gdbserver/linux-low: delete 'linux_target_ops' and 'the_low_target' 2020-04-02 15:11:32 +02:00
linux-aarch64-tdesc.cc
linux-aarch64-tdesc.h
linux-amd64-ipa.cc
linux-arm-low.cc gdbserver/linux-low: delete 'linux_target_ops' and 'the_low_target' 2020-04-02 15:11:32 +02:00
linux-arm-tdesc.cc
linux-arm-tdesc.h
linux-i386-ipa.cc
linux-ia64-low.cc gdbserver/linux-ia64-low: fix a build-breaking typo 2020-05-16 10:45:40 +02:00
linux-low.cc gdb, gdbserver: make stopped_pids global variables static 2020-07-21 16:49:36 -04:00
linux-low.h gdbserver/linux-low: use std::list to store pending signals 2020-06-22 14:13:48 +02:00
linux-m68k-low.cc gdbserver/linux-low: delete 'linux_target_ops' and 'the_low_target' 2020-04-02 15:11:32 +02:00
linux-mips-low.cc gdbserver/linux-low: delete 'linux_target_ops' and 'the_low_target' 2020-04-02 15:11:32 +02:00
linux-nios2-low.cc gdbserver/linux-low: delete 'linux_target_ops' and 'the_low_target' 2020-04-02 15:11:32 +02:00
linux-ppc-ipa.cc
linux-ppc-low.cc gdbserver/linux-low: delete 'linux_target_ops' and 'the_low_target' 2020-04-02 15:11:32 +02:00
linux-ppc-tdesc-init.h
linux-riscv-low.cc gdbserver/linux-low: delete 'linux_target_ops' and 'the_low_target' 2020-04-02 15:11:32 +02:00
linux-s390-ipa.cc
linux-s390-low.cc gdbserver/linux-low: delete 'linux_target_ops' and 'the_low_target' 2020-04-02 15:11:32 +02:00
linux-s390-tdesc.h
linux-sh-low.cc gdbserver/linux-low: delete 'linux_target_ops' and 'the_low_target' 2020-04-02 15:11:32 +02:00
linux-sparc-low.cc gdbserver/linux-low: delete 'linux_target_ops' and 'the_low_target' 2020-04-02 15:11:32 +02:00
linux-tic6x-low.cc gdbserver/linux-low: delete 'linux_target_ops' and 'the_low_target' 2020-04-02 15:11:32 +02:00
linux-x86-low.cc gdbserver: fix memory leak when handling qsupported packet 2020-07-13 22:27:01 -04:00
linux-x86-tdesc.cc
linux-x86-tdesc.h
linux-xtensa-low.cc gdbserver/linux-low: delete 'linux_target_ops' and 'the_low_target' 2020-04-02 15:11:32 +02:00
Makefile.in gdb, gdbserver: remove ARM regdat files 2020-06-17 14:42:53 -04:00
mem-break.cc
mem-break.h
notif.cc
notif.h
proc-service.cc
proc-service.list
README gdbserver: small cleanup of README file 2020-06-12 16:01:35 -04:00
regcache.cc
regcache.h
remote-utils.cc gdbserver: remove support for Neutrino 2020-06-12 16:06:41 -04:00
remote-utils.h
server.cc gdbserver: handle running threads in qXfer:threads:read 2020-07-22 12:32:53 +01:00
server.h gdbserver: remove support for ARM/WinCE 2020-06-12 16:06:45 -04:00
symbol.cc
target.cc gdbserver: fix memory leak when handling qsupported packet 2020-07-13 22:27:01 -04:00
target.h gdbserver: fix memory leak when handling qsupported packet 2020-07-13 22:27:01 -04:00
tdesc.cc gdb/riscv: delete target descriptions when gdb exits 2020-07-17 21:15:32 +01:00
tdesc.h gdb/riscv: delete target descriptions when gdb exits 2020-07-17 21:15:32 +01:00
thread-db.cc gdbserver: handle running threads in qXfer:threads:read 2020-07-22 12:32:53 +01:00
tracepoint.cc
tracepoint.h
utils.cc Remove gdb_fildes_t 2020-04-13 14:10:04 -06:00
utils.h Remove gdb_fildes_t 2020-04-13 14:10:04 -06:00
win32-i386-low.cc Enable hardware breakpoints for gdbserver on Windows 2020-05-15 10:55:21 +02:00
win32-low.cc gdbserver: remove support for ARM/WinCE 2020-06-12 16:06:45 -04:00
win32-low.h gdbserver: remove support for ARM/WinCE 2020-06-12 16:06:45 -04:00
x86-low.cc
x86-low.h
x86-tdesc.h
xtensa-xtregs.cc

		   README for GDBserver & GDBreplay
		    by Stu Grossman and Fred Fish

Introduction:

This is GDBserver, a remote server for Un*x-like systems.  It can be used to
control the execution of a program on a target system from a GDB on a different
host.  GDB and GDBserver communicate using the standard remote serial protocol.
They communicate via either a serial line or a TCP connection.

For more information about GDBserver, see the GDB manual:

    https://sourceware.org/gdb/current/onlinedocs/gdb/Remote-Protocol.html

Usage (server (target) side):

First, you need to have a copy of the program you want to debug put onto
the target system.  The program can be stripped to save space if needed, as
GDBserver doesn't care about symbols.  All symbol handling is taken care of by
the GDB running on the host system.

To use the server, you log on to the target system, and run the `gdbserver'
program.  You must tell it (a) how to communicate with GDB, (b) the name of
your program, and (c) its arguments.  The general syntax is:

	target> gdbserver COMM PROGRAM [ARGS ...]

For example, using a serial port, you might say:

	target> gdbserver /dev/com1 emacs foo.txt

This tells GDBserver to debug emacs with an argument of foo.txt, and to
communicate with GDB via /dev/com1.  GDBserver now waits patiently for the
host GDB to communicate with it.

To use a TCP connection, you could say:

	target> gdbserver host:2345 emacs foo.txt

This says pretty much the same thing as the last example, except that we are
going to communicate with the host GDB via TCP.  The `host:2345' argument means
that we are expecting to see a TCP connection to local TCP port 2345.
(Currently, the `host' part is ignored.)  You can choose any number you want for
the port number as long as it does not conflict with any existing TCP ports on
the target system.  This same port number must be used in the host GDB's
`target remote' command, which will be described shortly. Note that if you chose
a port number that conflicts with another service, GDBserver will print an error
message and exit.

On some targets, GDBserver can also attach to running programs.  This is
accomplished via the --attach argument.  The syntax is:

	target> gdbserver --attach COMM PID

PID is the process ID of a currently running process.  It isn't necessary
to point GDBserver at a binary for the running process.

Usage (host side):

You need an unstripped copy of the target program on your host system, since
GDB needs to examine it's symbol tables and such.  Start up GDB as you normally
would, with the target program as the first argument.  (You may need to use the
--baud option if the serial line is running at anything except 9600 baud.)
Ie: `gdb TARGET-PROG', or `gdb --baud BAUD TARGET-PROG'.  After that, the only
new command you need to know about is `target remote'.  It's argument is either
a device name (usually a serial device, like `/dev/ttyb'), or a HOST:PORT
descriptor.  For example:

	(gdb) target remote /dev/ttyb

communicates with the server via serial line /dev/ttyb, and:

	(gdb) target remote the-target:2345

communicates via a TCP connection to port 2345 on host `the-target', where
you previously started up GDBserver with the same port number.  Note that for
TCP connections, you must start up GDBserver prior to using the `target remote'
command, otherwise you may get an error that looks something like
`Connection refused'.

Building GDBserver:

See the `configure.srv` file for the list of host triplets you can build
GDBserver for.

Building GDBserver for your host is very straightforward.  If you build
GDB natively on a host which GDBserver supports, it will be built
automatically when you build GDB.  You can also build just GDBserver:

	% mkdir obj
	% cd obj
	% path-to-toplevel-sources/configure --disable-gdb
	% make all-gdbserver

(If you have a combined binutils+gdb tree, you may want to also
disable other directories when configuring, e.g., binutils, gas, gold,
gprof, and ld.)

If you prefer to cross-compile to your target, then you can also build
GDBserver that way.  For example:

	% export CC=your-cross-compiler
	% path-to-topevel-sources/configure --disable-gdb
	% make all-gdbserver

Using GDBreplay:

A special hacked down version of GDBserver can be used to replay remote
debug log files created by GDB.  Before using the GDB "target" command to
initiate a remote debug session, use "set remotelogfile <filename>" to tell
GDB that you want to make a recording of the serial or tcp session.  Note
that when replaying the session, GDB communicates with GDBreplay via tcp,
regardless of whether the original session was via a serial link or tcp.

Once you are done with the remote debug session, start GDBreplay and
tell it the name of the log file and the host and port number that GDB
should connect to (typically the same as the host running GDB):

	$ gdbreplay logfile host:port

Then start GDB (preferably in a different screen or window) and use the
"target" command to connect to GDBreplay:

	(gdb) target remote host:port

Repeat the same sequence of user commands to GDB that you gave in the
original debug session.  GDB should not be able to tell that it is talking
to GDBreplay rather than a real target, all other things being equal.  Note
that GDBreplay echos the command lines to stderr, as well as the contents of
the packets it sends and receives.  The last command echoed by GDBreplay is
the next command that needs to be typed to GDB to continue the session in
sync with the original session.