Now a bunch of stuff that doesn't apply to ld or any normal use of
libctf, piled into one commit so that it's easier to ignore.
The cu-mapping machinery associates incoming compilation unit names with
outgoing names of CTF dictionaries that should correspond to them, for
non-gdb CTF consumers that would like to group multiple TUs into a
single child dict if conflicting types are found in it (the existing use
case is one kernel module, one child CTF dict, even if the kernel module
is composed of multiple CUs).
The upcoming deduplicator needs to track not only the mapping from
incoming CU name to outgoing dict name, but the inverse mapping from
outgoing dict name to incoming CU name, so it can work over every CTF
dict we might see in the output and link into it.
So rejig the ctf-link machinery to do that. Simultaneously (because
they are closely associated and were written at the same time), we add a
new CTF_LINK_EMPTY_CU_MAPPINGS flag to ctf_link, which tells the
ctf_link machinery to create empty child dicts for each outgoing CU
mapping even if no CUs that correspond to it exist in the link. This is
a bit (OK, quite a lot) of a waste of space, but some existing consumers
require it. (Nobody else should use it.)
Its value is not consecutive with existing CTF_LINK flag values because
we're about to add more flags that are conceptually closer to the
existing ones than this one is.
include/
* ctf-api.h (CTF_LINK_EMPTY_CU_MAPPINGS): New.
libctf/
* ctf-impl.h (ctf_file_t): Improve comments.
<ctf_link_cu_mapping>: Split into...
<ctf_link_in_cu_mapping>: ... this...
<ctf_link_out_cu_mapping>: ... and this.
* ctf-create.c (ctf_serialize): Adjust.
* ctf-open.c (ctf_file_close): Likewise.
* ctf-link.c (ctf_create_per_cu): Look things up in the
in_cu_mapping instead of the cu_mapping.
(ctf_link_add_cu_mapping): The deduplicating link will define
what happens if many FROMs share a TO.
(ctf_link_add_cu_mapping): Create in_cu_mapping and
out_cu_mapping. Do not create ctf_link_outputs here any more, or
create per-CU dicts here: they are already created when needed.
(ctf_link_one_variable): Log a debug message if we skip a
variable due to its type being concealed in a CU-mapped link.
(This is probably too common a case to make into a warning.)
(ctf_link): Create empty per-CU dicts if requested.
This rather large and intertwined pile of changes does three things:
First, it transitions from dprintf to ctf_err_warn for things the user might
care about: this one file is the major impetus for the ctf_err_warn
infrastructure, because things like file names are crucial in linker
error messages, and errno values are utterly incapable of
communicating them
Second, it stabilizes the ctf_link APIs: you can now call
ctf_link_add_ctf without a CTF argument (only a NAME), to lazily
ctf_open the file with the given NAME when needed, and close it as soon
as possible, to save memory. This is not an API change because a null
CTF argument was prohibited before now.
Since getting CTF directly from files uses ctf_open, passing in only a
NAME requires use of libctf, not libctf-nobfd. The linker's behaviour
is unchanged, as it still passes in a ctf_archive_t as before.
This also let us fix a leak: we were opening ctf_archives and their
containing ctf_files, then only closing the files and leaving the
archives open.
Third, this commit restructures the ctf_link_in_member argument used by
the CTF linking machinery and adjusts its users accordingly.
We drop two members:
- arcname, which is difficult to construct and then only used in error
messages (that were only dprintf()ed, so never seen!)
- share_mode, since we store the flags passed to ctf_link (including the
share mode) in a new ctf_file_t.ctf_link_flags to help dedup get hold
of it
We rename others whose existing names were fairly dreadful:
- done_main_member -> done_parent, using consistent terminology for .ctf
as the parent of all archive members
- main_input_fp -> in_fp_parent, likewise
- file_name -> in_file_name, likewise
We add one new member, cu_mapped.
Finally, we move the various frees of things like mapping table data to
the top-level ctf_link, since deduplicating links will want to do that
too.
include/
* ctf-api.h (ECTF_NEEDSBFD): New.
(ECTF_NERR): Adjust.
(ctf_link): Rename share_mode arg to flags.
libctf/
* Makefile.am: Set -DNOBFD=1 in libctf-nobfd, and =0 elsewhere.
* Makefile.in: Regenerated.
* ctf-impl.h (ctf_link_input_name): New.
(ctf_file_t) <ctf_link_flags>: New.
* ctf-create.c (ctf_serialize): Adjust accordingly.
* ctf-link.c: Define ctf_open as weak when PIC.
(ctf_arc_close_thunk): Remove unnecessary thunk.
(ctf_file_close_thunk): Likewise.
(ctf_link_input_name): New.
(ctf_link_input_t): New value of the ctf_file_t.ctf_link_input.
(ctf_link_input_close): Adjust accordingly.
(ctf_link_add_ctf_internal): New, split from...
(ctf_link_add_ctf): ... here. Return error if lazy loading of
CTF is not possible. Change to just call...
(ctf_link_add): ... this new function.
(ctf_link_add_cu_mapping): Transition to ctf_err_warn. Drop the
ctf_file_close_thunk.
(ctf_link_in_member_cb_arg_t) <file_name> Rename to...
<in_file_name>: ... this.
<arcname>: Drop.
<share_mode>: Likewise (migrated to ctf_link_flags).
<done_main_member>: Rename to...
<done_parent>: ... this.
<main_input_fp>: Rename to...
<in_fp_parent>: ... this.
<cu_mapped>: New.
(ctf_link_one_type): Adjuwt accordingly. Transition to
ctf_err_warn, removing a TODO.
(ctf_link_one_variable): Note a case too common to warn about.
Report in the debug stream if a cu-mapped link prevents addition
of a conflicting variable.
(ctf_link_one_input_archive_member): Adjust.
(ctf_link_lazy_open): New, open a CTF archive for linking when
needed.
(ctf_link_close_one_input_archive): New, close it again.
(ctf_link_one_input_archive): Adjust for lazy opening, member
renames, and ctf_err_warn transition. Move the
empty_link_type_mapping call to...
(ctf_link): ... here. Adjut for renamings and thunk removal.
Don't spuriously fail if some input contains no CTF data.
(ctf_link_write): ctf_err_warn transition.
* libctf.ver: Remove not-yet-stable comment.
This utility function is almost useless (all it does is casts the result
of a strerror) but has a seriously confusing name. Over and over again
I have accidentally called it instead of ctf_errmsg, and hidden a
time-bomb for myself in a hard-to-test error-handling path: since
ctf_strerror is just a strerror wrapper, it cannot handle CTF errnos,
unlike ctf_errmsg. It's astonishingly lucky that none of these errors
have crept into any commits to date.
Fuse it into ctf_errmsg and drop it.
libctf/
* ctf-impl.h (ctf_strerror): Delete.
* ctf-subr.c (ctf_strerror): Likewise.
* ctf-error.c (ctf_errmsg): Stop using ctf_strerror: just use
strerror directly.
When you link TUs that contain conflicting types together, the resulting
CTF section is an archive containing many CTF dicts. These dicts appear
in ctf_link_outputs of the shared dict, with each ctf_import'ing that
shared dict. ctf_importing a dict bumps its refcount to stop it going
away while it's in use -- but if the shared dict (whose refcount is
bumped) has the child dict (doing the bumping) in its ctf_link_outputs,
we have a refcount loop, since the child dict only un-ctf_imports and
drops the parent's refcount when it is freed, but the child is only
freed when the parent's refcount falls to zero.
(In the future, this will be able to go wrong on the inputs too, when an
ld -r'ed deduplicated output with conflicts is relinked. Right now this
cannot happen because we don't ctf_import such dicts at all. This will
be fixed in a later commit in this series.)
Fix this by introducing an internal-use-only ctf_import_unref function
that imports a parent dict *witthout* bumping the parent's refcount, and
using it when we create per-CU outputs. This function is only safe to
use if you know the parent cannot go away while the child exists: but if
the parent *owns* the child, as here, this is necessarily true.
Record in the ctf_file_t whether a parent was imported via ctf_import or
ctf_import_unref, so that if you do another ctf_import later on (or a
ctf_import_unref) it can decide whether to drop the refcount of the
existing parent being replaced depending on which function you used to
import that one. Adjust ctf_serialize so that rather than doing a
ctf_import (which is wrong if the original import was
ctf_import_unref'fed), we just copy the parent field and refcount over
and forcibly flip the unref flag on on the old copy we are going to
discard.
ctf_file_close also needs a bit of tweaking to only close the parent if
it was not imported with ctf_import_unref: while we're at it, guard
against repeated closes with a refcount of zero and stop them causing
double-frees, even if destruction of things freed *inside*
ctf_file_close cause such recursion.
Verified no leaks or accesses to freed memory after all of this with
valgrind. (It was leak-happy before.)
libctf/
* ctf-impl.c (ctf_file_t) <ctf_parent_unreffed>: New.
(ctf_import_unref): New.
* ctf-open.c (ctf_file_close) Drop the refcount all the way to
zero. Don't recurse back in if the refcount is already zero.
(ctf_import): Check ctf_parent_unreffed before deciding whether
to close a pre-existing parent. Set it to zero.
(ctf_import_unreffed): New, as above, setting
ctf_parent_unreffed to 1.
* ctf-create.c (ctf_serialize): Do not ctf_import into the new
child: use direct assignment, and set unreffed on the new and
old children.
* ctf-link.c (ctf_create_per_cu): Import the parent using
ctf_import_unreffed.
The name was just annoyingly long and I kept misspelling it.
It's also a bad name: it's not a mapping the type might be *used* in a
type mapping, but it is itself a representation of a type (a ctf_file_t
/ ctf_id_t pair), not of a mapping at all.
libctf/
* ctf-impl.h (ctf_link_type_mapping_key): Rename to...
(ctf_link_type_key): ... this, adjusting member prefixes to
match.
(ctf_hash_type_mapping_key): Rename to...
(ctf_hash_type_key): ... this.
(ctf_hash_eq_type_mapping_key): Rename to...
(ctf_hash_eq_type_key): ... this.
* ctf-hash.c (ctf_hash_type_mapping_key): Rename to...
(ctf_hash_type_key): ... this, and adjust for member name
changes.
(ctf_hash_eq_type_mapping_key): Rename to...
(ctf_hash_eq_type_key): ... this, and adjust for member name
changes.
* ctf-link.c (ctf_add_type_mapping): Adjust. Note the lack of
need for out-of-memory checking in this code.
(ctf_type_mapping): Adjust.
We've been using this for all of libctf's history in binutils: we should
check for it in configure.
libctf/
configure.ac: Check for vasprintf.
configure: Regenerated.
config.h.in: Likewise.
This is a perfectly possible case, and half of ctf_bfdopen_ctfsect
handled it fine. The other half hit a divide by zero or two before we
got that far, and had no code path to load the strtab from anywhere
in the absence of a symtab to point at it in any case.
So, as a fallback, if there is no symtab, try loading ".strtab"
explicitly by name, like we used to before we started looking for the
strtab the symtab used.
Of course, such a strtab is not kept hold of by BFD, so this means we
have to bring back the code to possibly explicitly free the strtab that
we read in.
libctf/
* ctf-impl.h (struct ctf_archive_internal) <ctfi_free_strsect>
New.
* ctf-open-bfd.c (ctf_bfdopen_ctfsect): Explicitly open a strtab
if the input has no symtab, rather than dividing by
zero. Arrange to free it later via ctfi_free_ctfsect.
* ctf-archive.c (ctf_new_archive_internal): Do not
ctfi_free_strsect by default.
(ctf_arc_close): Possibly free it here.
Now that we can have slices of anything terminating in an int, we must
dump things accordingly, or slices of typedefs appear as
c5b: __u8 -> 16c: __u8 -> 78: short unsigned int (size 0x2)
which is unhelpful. If things *are* printed as slices, the name is
missing:
a15: [slice 0x8:0x4]-> 16c: __u8 -> 78: short unsigned int (size 0x2)
And struct members give no clue they're a slice at all, which is a shame
since bitfields are the major use of this type kind:
[0x8] (ID 0xa15) (kind 10) __u8 dst_reg
Fix things so that everything slicelike or integral gets its encoding
printed, and everything with a name gets the name printed:
a15: __u8 [slice 0x8:0x4] (size 0x1) -> 1ff: __u8 (size 0x1) -> 37: unsigned char [0x0:0x8] (size 0x1)
[0x0] (ID 0xa15) (kind 10) __u8:4 (aligned at 0x1, format 0x2, offset:bits 0x8:0x4)
Bitfield struct members get a technically redundant but much
easier-to-understand dumping now:
[0x0] (ID 0x80000005) (kind 6) struct bpf_insn (aligned at 0x1)
[0x0] (ID 0x222) (kind 10) __u8 code (aligned at 0x1)
[0x8] (ID 0x1e9e) (kind 10) __u8 dst_reg:4 (aligned at 0x1, format 0x2, offset:bits 0x8:0x4)
[0xc] (ID 0x1e46) (kind 10) __u8 src_reg:4 (aligned at 0x1, format 0x2, offset:bits 0xc:0x4)
[0x10] (ID 0xf35) (kind 10) __s16 off (aligned at 0x2)
[0x20] (ID 0x1718) (kind 10) __s32 imm (aligned at 0x4)
This also fixes one place where a failure to format a type would be
erroneously considered an out-of-memory condition.
libctf/
* ctf-dump.c (ctf_is_slice): Delete, unnecessary.
(ctf_dump_format_type): improve slice formatting. Always print
the type size, even of slices.
(ctf_dump_member): Print slices (-> bitfields) differently from
non-slices. Failure to format a type is not an OOM.
If we get an error emitting a single type, variable, or label, right now
we emit the error into the ctf_dprintf stream and propagate the error
all the way up the stack, causing the entire output to be silently
truncated (unless libctf debugging is on).
Instead, emit an error and keep going. (This makes sense for this use
case: if you're dumping types and a type is corrupted, you want to
know!)
Not all instances of this are fixed in this commit, only ones associated
with type formatting: more fixes will come.
libctf/
* ctf-dump.c (ctf_dump_format_type): Emit a warning.
(ctf_dump_label): Swallow errors from ctf_dump_format_type.
(ctf_dump_objts): Likewise.
(ctf_dump_var): Likewise.
(ctf_dump_type): Do not emit a duplicate message. Move to
ctf_err_warning, and swallow all errors.
ctf_decl_sprintf builds up a formatted string in the ctf_decl_t's
cd_buf, but then on error this is hardly ever freed: we assume that
ctf_decl_fini frees it, but it leaks it instead.
Make it free it like any decent ADT should.
libctf/
* ctf-decl.c (ctf_decl_fini): Free the cd_buf.
(ctf_decl_buf): Once it escapes, don't try to free it later.
Somehow this never got implemented, which makes debugging any kind of
bug that has to do with argument types fantastically confusing, because
it *looks* like the func type takes no arguments though in fact it does.
This also lets us simplify the dumper slightly (and introduces our first
uses of ctf_assert and ctf_err_warn: there will be many more).
ctf_type_aname dumps function types without including the function
pointer name itself: ctf_dump search-and-replaces it in. This seems to
give the nicest-looking results for existing users of both, even if it
is a bit fiddly.
libctf/
* ctf-types.c (ctf_type_aname): Print arg types here...
* ctf-dump.c (ctf_dump_funcs): ... not here: but do substitute
in the type name here.
This commit adds a long-missing piece of infrastructure to libctf: the
ability to report errors and warnings using all the power of printf,
rather than being restricted to one errno value. Internally, libctf
calls ctf_err_warn() to add errors and warnings to a list: a new
iterator ctf_errwarning_next() then consumes this list one by one and
hands it to the caller, which can free it. New errors and warnings are
added until the list is consumed by the caller or the ctf_file_t is
closed, so you can dump them at intervals. The caller can of course
choose to print only those warnings it wants. (I am not sure whether we
want objdump, readelf or ld to print warnings or not: right now I'm
printing them, but maybe we only want to print errors? This entirely
depends on whether warnings are voluminous things describing e.g. the
inability to emit single types because of name clashes or something.
There are no users of this infrastructure yet, so it's hard to say.)
There is no internationalization here yet, but this at least adds a
place where internationalization can be added, to one of
ctf_errwarning_next or ctf_err_warn.
We also provide a new ctf_assert() function which uses this
infrastructure to provide non-fatal assertion failures while emitting an
assert-like string to the caller: to save space and avoid needlessly
duplicating unchanging strings, the assertion test is inlined but the
print-things-out failure case is not. All assertions in libctf will be
converted to use this machinery in future commits and propagate
assertion-failure errors up, so that the linker in particular cannot be
killed by libctf assertion failures when it could perfectly well just
print warnings and drop the CTF section.
include/
* ctf-api.h (ECTF_INTERNAL): Adjust error text.
(ctf_errwarning_next): New.
libctf/
* ctf-impl.h (ctf_assert): New.
(ctf_err_warning_t): Likewise.
(ctf_file_t) <ctf_errs_warnings>: Likewise.
(ctf_err_warn): New prototype.
(ctf_assert_fail_internal): Likewise.
* ctf-inlines.h (ctf_assert_internal): Likewise.
* ctf-open.c (ctf_file_close): Free ctf_errs_warnings.
* ctf-create.c (ctf_serialize): Copy it on serialization.
* ctf-subr.c (ctf_err_warn): New, add an error/warning.
(ctf_errwarning_next): New iterator, free and pass back
errors/warnings in succession.
* libctf.ver (ctf_errwarning_next): Add.
ld/
* ldlang.c (lang_ctf_errs_warnings): New, print CTF errors
and warnings. Assert when libctf asserts.
(lang_merge_ctf): Call it.
(land_write_ctf): Likewise.
binutils/
* objdump.c (ctf_archive_member): Print CTF errors and warnings.
* readelf.c (dump_ctf_archive_member): Likewise.
ctf_variable_iter was returning a (positive!) error code rather than
setting the error in the passed-in ctf_file_t.
Reviewed-by: Nick Alcock <nick.alcock@oracle.com>
libctf/
* ctf-types.c (ctf_variable_iter): Fix error return.
When wrapping qsort_r on a system like FreeBSD on which the compar
argument comes first, we wrap the passed arg in a thunk so we can pass
down both the caller-supplied comparator function and its argument. We
should pass the *argument* down to the comparator, not the thunk, which
is basically random nonsense on the stack from the point of view of the
caller of qsort_r.
libctf/
ctf-decls.h (ctf_qsort_compar_thunk): Fix arg passing.
This lets you iterate over dynhashes and dynsets using the _next API.
dynhashes can be iterated over in sorted order, which works by
populating an array of key/value pairs using ctf_dynhash_next itself,
then sorting it with qsort.
Convenience inline functions named ctf_dyn{hash,set}_cnext are also
provided that take (-> return) const keys and values.
libctf/
* ctf-impl.h (ctf_next_hkv_t): New, kv-pairs passed to
sorting functions.
(ctf_next_t) <u.ctn_sorted_hkv>: New, sorted kv-pairs for
ctf_dynhash_next_sorted.
<cu.ctn_h>: New, pointer to the dynhash under iteration.
<cu.ctn_s>: New, pointer to the dynset under iteration.
(ctf_hash_sort_f): Sorting function passed to...
(ctf_dynhash_next_sorted): ... this new function.
(ctf_dynhash_next): New.
(ctf_dynset_next): New.
* ctf-inlines.h (ctf_dynhash_cnext_sorted): New.
(ctf_dynhash_cnext): New.
(ctf_dynset_cnext): New.
* ctf-hash.c (ctf_dynhash_next_sorted): New.
(ctf_dynhash_next): New.
(ctf_dynset_next): New.
* ctf-util.c (ctf_next_destroy): Free the u.ctn_sorted_hkv if
needed.
(ctf_next_copy): Alloc-and-copy the u.ctn_sorted_hkv if needed.
The libctf machinery currently only provides one way to iterate over its
data structures: ctf_*_iter functions that take a callback and an arg
and repeatedly call it.
This *works*, but if you are doing a lot of iteration it is really quite
inconvenient: you have to package up your local variables into
structures over and over again and spawn lots of little functions even
if it would be clearer in a single run of code. Look at ctf-string.c
for an extreme example of how unreadable this can get, with
three-line-long functions proliferating wildly.
The deduplicator takes this to the Nth level. It iterates over a whole
bunch of things: if we'd had to use _iter-class iterators for all of
them there would be twenty additional functions in the deduplicator
alone, for no other reason than that the iterator API requires it.
Let's do something better. strtok_r gives us half the design: generators
in a number of other languages give us the other half.
The *_next API allows you to iterate over CTF-like entities in a single
function using a normal while loop. e.g. here we are iterating over all
the types in a dict:
ctf_next_t *i = NULL;
int *hidden;
ctf_id_t id;
while ((id = ctf_type_next (fp, &i, &hidden, 1)) != CTF_ERR)
{
/* do something with 'hidden' and 'id' */
}
if (ctf_errno (fp) != ECTF_NEXT_END)
/* iteration error */
Here we are walking through the members of a struct with CTF ID
'struct_type':
ctf_next_t *i = NULL;
ssize_t offset;
const char *name;
ctf_id_t membtype;
while ((offset = ctf_member_next (fp, struct_type, &i, &name,
&membtype)) >= 0
{
/* do something with offset, name, and membtype */
}
if (ctf_errno (fp) != ECTF_NEXT_END)
/* iteration error */
Like every other while loop, this means you have access to all the local
variables outside the loop while inside it, with no need to tiresomely
package things up in structures, move the body of the loop into a
separate function, etc, as you would with an iterator taking a callback.
ctf_*_next allocates 'i' for you on first entry (when it must be NULL),
and frees and NULLs it and returns a _next-dependent flag value when the
iteration is over: the fp errno is set to ECTF_NEXT_END when the
iteartion ends normally. If you want to exit early, call
ctf_next_destroy on the iterator. You can copy iterators using
ctf_next_copy, which copies their current iteration position so you can
remember loop positions and go back to them later (or ctf_next_destroy
them if you don't need them after all).
Each _next function returns an always-likely-to-be-useful property of
the thing being iterated over, and takes pointers to parameters for the
others: with very few exceptions all those parameters can be NULLs if
you're not interested in them, so e.g. you can iterate over only the
offsets of members of a structure this way:
while ((offset = ctf_member_next (fp, struct_id, &i, NULL, NULL)) >= 0)
If you pass an iterator in use by one iteration function to another one,
you get the new error ECTF_NEXT_WRONGFUN back; if you try to change
ctf_file_t in mid-iteration, you get ECTF_NEXT_WRONGFP back.
Internally the ctf_next_t remembers the iteration function in use,
various sizes and increments useful for almost all iterations, then
uses unions to overlap the actual entities being iterated over to keep
ctf_next_t size down.
Iterators available in the public API so far (all tested in actual use
in the deduplicator):
/* Iterate over the members of a STRUCT or UNION, returning each member's
offset and optionally name and member type in turn. On end-of-iteration,
returns -1. */
ssize_t
ctf_member_next (ctf_file_t *fp, ctf_id_t type, ctf_next_t **it,
const char **name, ctf_id_t *membtype);
/* Iterate over the members of an enum TYPE, returning each enumerand's
NAME or NULL at end of iteration or error, and optionally passing
back the enumerand's integer VALue. */
const char *
ctf_enum_next (ctf_file_t *fp, ctf_id_t type, ctf_next_t **it,
int *val);
/* Iterate over every type in the given CTF container (not including
parents), optionally including non-user-visible types, returning
each type ID and optionally the hidden flag in turn. Returns CTF_ERR
on end of iteration or error. */
ctf_id_t
ctf_type_next (ctf_file_t *fp, ctf_next_t **it, int *flag,
int want_hidden);
/* Iterate over every variable in the given CTF container, in arbitrary
order, returning the name and type of each variable in turn. The
NAME argument is not optional. Returns CTF_ERR on end of iteration
or error. */
ctf_id_t
ctf_variable_next (ctf_file_t *fp, ctf_next_t **it, const char **name);
/* Iterate over all CTF files in an archive, returning each dict in turn as a
ctf_file_t, and NULL on error or end of iteration. It is the caller's
responsibility to close it. Parent dicts may be skipped. Regardless of
whether they are skipped or not, the caller must ctf_import the parent if
need be. */
ctf_file_t *
ctf_archive_next (const ctf_archive_t *wrapper, ctf_next_t **it,
const char **name, int skip_parent, int *errp);
ctf_label_next is prototyped but not implemented yet.
include/
* ctf-api.h (ECTF_NEXT_END): New error.
(ECTF_NEXT_WRONGFUN): Likewise.
(ECTF_NEXT_WRONGFP): Likewise.
(ECTF_NERR): Adjust.
(ctf_next_t): New.
(ctf_next_create): New prototype.
(ctf_next_destroy): Likewise.
(ctf_next_copy): Likewise.
(ctf_member_next): Likewise.
(ctf_enum_next): Likewise.
(ctf_type_next): Likewise.
(ctf_label_next): Likewise.
(ctf_variable_next): Likewise.
libctf/
* ctf-impl.h (ctf_next): New.
(ctf_get_dict): New prototype.
* ctf-lookup.c (ctf_get_dict): New, split out of...
(ctf_lookup_by_id): ... here.
* ctf-util.c (ctf_next_create): New.
(ctf_next_destroy): New.
(ctf_next_copy): New.
* ctf-types.c (includes): Add <assert.h>.
(ctf_member_next): New.
(ctf_enum_next): New.
(ctf_type_iter): Document the lack of iteration over parent
types.
(ctf_type_next): New.
(ctf_variable_next): New.
* ctf-archive.c (ctf_archive_next): New.
* libctf.ver: Add new public functions.
This allows you to bump the refcount on a ctf_file_t, so that you can
smuggle it out of iterators which open and close the ctf_file_t for you
around the loop body (like ctf_archive_iter).
You still can't use this to preserve a ctf_file_t for longer than the
lifetime of its containing entity (e.g. ctf_archive).
include/
* ctf-api.h (ctf_ref): New.
libctf/
* libctf.ver (ctf_ref): New.
* ctf-open.c (ctf_ref): Implement it.
The internals of the deduplicator want to know if something is a type
that can have a forward to it fairly often, often enough that inlining
it brings a noticeable performance gain. Convert the one place in
libctf that can already benefit, even though it doesn't bring any sort
of performance gain there.
libctf/
* ctf-inlines.h (ctf_forwardable_kind): New.
* ctf-create.c (ctf_add_forward): Use it.
There are many places in the deduplicator which use hashtables as tiny
sets: keys with no value (and usually, but not always, no freeing
function) often with only one or a few members. For each of these, even
after the last change to not store the freeing functions, we are storing
a little malloced block for each item just to track the key/value pair,
and a little malloced block for the hash table itself just to track the
freeing function because we can't use libiberty hashtab's freeing
function because we are using that to free the little malloced per-item
block.
If we only have a key, we don't need any of that: we can ditch the
per-malloced block because we don't have a value, and we can ditch the
per-hashtab structure because we don't need to independently track the
freeing functions since libiberty hashtab is doing it for us. That
means we don't need an owner field in the (now nonexistent) item block
either.
Roughly speaking, this datatype saves about 25% in time and 20% in peak
memory usage for normal links, even fairly big ones. So this might seem
redundant, but it's really worth it.
Instead of a _lookup function, a dynset has two distinct functions:
ctf_dynset_exists, which returns true or false and an optional pointer
to the set member, and ctf_dynhash_lookup_any, which is used if all
members of the set are expected to be equivalent and we just want *any*
member and we don't care which one.
There is no iterator in this set of functions, not because we don't
iterate over dynset members -- we do, a lot -- but because the iterator
here is a member of an entirely new family of much more convenient
iteration functions, introduced in the next commit.
libctf/
* ctf-hash.c (ctf_dynset_eq_string): New.
(ctf_dynset_create): New.
(DYNSET_EMPTY_ENTRY_REPLACEMENT): New.
(DYNSET_DELETED_ENTRY_REPLACEMENT): New.
(key_to_internal): New.
(internal_to_key): New.
(ctf_dynset_insert): New.
(ctf_dynset_remove): New.
(ctf_dynset_destroy): New.
(ctf_dynset_lookup): New.
(ctf_dynset_exists): New.
(ctf_dynset_lookup_any): New.
(ctf_hash_insert_type): Coding style.
(ctf_hash_define_type): Likewise.
* ctf-impl.h (ctf_dynset_t): New.
(ctf_dynset_eq_string): New.
(ctf_dynset_create): New.
(ctf_dynset_insert): New.
(ctf_dynset_remove): New.
(ctf_dynset_destroy): New.
(ctf_dynset_lookup): New.
(ctf_dynset_exists): New.
(ctf_dynset_lookup_any): New.
* ctf-inlines.h (ctf_dynset_cinsert): New.
The libctf dynhash hashtab abstraction supports per-hashtab arbitrary
key/item freeing functions -- but it also has a constant slot type that
holds both key and value requested by the user, so it needs to use its
own freeing function to free that -- and it has nowhere to store the
freeing functions the caller requested.
So it copies them into every hash item, bloating every slot, even though
all items in a given hash table must have the same key and value freeing
functions.
So point back to the owner using a back-pointer, but don't even spend
space in the item or the hashtab allocating those freeing functions
unless necessary: if none are needed, we can simply arrange to not pass
in ctf_dynhash_item_free as a del_f to hashtab_create_alloc, and none of
those fields will ever be accessed.
The only downside is that this makes the code sensitive to the order of
fields in the ctf_helem_t and ctf_hashtab_t: but the deduplicator
allocates so many hash tables that doing this alone cuts memory usage
during deduplication by about 10%. (libiberty hashtab itself has a lot
of per-hashtab bloat: in the future we might trim that down, or make a
trimmer version.)
libctf/
* ctf-hash.c (ctf_helem_t) <key_free>: Remove.
<value_free>: Likewise.
<owner>: New.
(ctf_dynhash_item_free): Indirect through the owner.
(ctf_dynhash_create): Only pass in ctf_dynhash_item_free and
allocate space for the key_free and value_free fields fields
if necessary.
(ctf_hashtab_insert): Likewise. Fix OOM errno value.
(ctf_dynhash_insert): Only access ctf_hashtab's key_free and
value_free if they will exist. Set the slot's owner, but only
if it exists.
(ctf_dynhash_remove): Adjust.
Right now, if you insert a key/value pair into a dynhash, the old slot's
key is freed and the new one always assigned. This seemed sane to me
when I wrote it, but I got it wrong time and time again. It's much
less confusing to free the key passed in: if a key-freeing function
was passed, you are asserting that the dynhash owns the key in any
case, so if you pass in a key it is always buggy to assume it sticks
around. Freeing the old key means that you can't even safely look up a
key from out of a dynhash and hold on to it, because some other matching
key might force it to be freed at any time.
In the new model, you can always get a key out of a dynhash with
ctf_dynhash_lookup_kv and hang on to it until the kv-pair is actually
deleted from the dynhash. In the old model the pointer to the key might
be freed at any time if a matching key was inserted.
libctf/
* ctf-hash.c (ctf_hashtab_insert): Free the key passed in if
there is a key-freeing function and the key already exists.
Future commits will use these.
ctf_dynhash_elements: count elements in a dynhash
ctf_dynhash_lookup_kv: look up and return pointers to the original key
and value in a dynhash (the only way of getting
a reference to the original key)
ctf_dynhash_iter_find: iterate until an item is found, then return its
key
ctf_dynhash_cinsert: insert a const key / value into a dynhash (a thim
wrapper in a new header dedicated to inline
functions).
As with the rest of ctf_dynhash, this is not public API. No impact
on existing callers is expected.
libctf/
* ctf-inlines.h: New file.
* ctf-impl.h: Include it.
(ctf_hash_iter_find_f): New typedef.
(ctf_dynhash_elements): New.
(ctf_dynhash_lookup_kv): New.
(ctf_dynhash_iter_find): New.
* ctf-hash.c (ctf_dynhash_lookup_kv): New.
(ctf_traverse_find_cb_arg_t): New.
(ctf_hashtab_traverse_find): New.
(ctf_dynhash_iter_find): New.
(ctf_dynhash_elements): New.
Another count that was otherwise unavailable without doing expensive
operations.
include/
* ctf-api.h (ctf_archive_count): New.
libctf/
* ctf-archive.c (ctf_archive_count): New.
* libctf.ver: New public function.
This returns the number of members in a struct or union, or the number
of enumerations in an enum. (This was only available before now by
iterating across every member, but it can be returned much faster than
that.)
include/
* ctf-api.h (ctf_member_count): New.
libctf/
* ctf-types.c (ctf_member_count): New.
* libctf.ver: New public function.
This is just like ctf_type_kind, except that forwards get the
type of the thing being pointed to rather than CTF_K_FORWARD.
include/
* ctf-api.h (ctf_type_kind_forwarded): New.
libctf/
* ctf-types.c (ctf_type_kind_forwarded): New.
We already have a function ctf_type_aname_raw, which returns the raw
name of a type with no decoration for structures or arrays or anything
like that: just the underlying name of whatever it is that's being
ultimately pointed at.
But this can be inconvenient to use, becauswe it always allocates new
storage for the string and copies it in, so it can potentially fail.
Add ctf_type_name_raw, which just returns the string directly out of
libctf's guts: it will live until the ctf_file_t is closed (if we later
gain the ability to remove types from writable dicts, it will live as
long as the type lives).
Reimplement ctf_type_aname_raw in terms of it.
include/
* ctf-api.c (ctf_type_name_raw): New.
libctf/
* ctf-types.c (ctf_type_name_raw): New.
(ctf_type_aname_raw): Reimplement accordingly.
The deduplicator can emit enormous amounts of debugging output,
so much so that a later commit will introduce a new configure flag
that configures most of it out (and configures it out by default).
It became clear that when this configure flag is on, but debugging is
not enabled via the LIBCTF_DEBUG environment variable, up to 10% of
runtime can be spent on branch mispredictions checking the _libctf_debug
variable. Mark it unlikely to be set (when it is set, performance is
likely to be the least of your concerns).
libctf/
* ctf-subr.c (ctf_dprintf): _libctf_debug is unlikely to be set.
The archive machinery mmap()s its archives when possible: so it arranges
to do appropriately-sized unmaps by recording the unmap length in the
ctfa_magic value and unmapping that.
This brilliant (horrible) trick works less well when ctf_arc_bufopen is
called with an existing buffer (which might be a readonly mapping).
ctf_arc_bufopen always returns a ctf_archive_t wrapper, so record in
there the necessity to not unmap anything when a bufopen'ed archive is
closed again.
libctf/
* ctf-impl.h (struct ctf_archive_internal)
<ctfi_unmap_on_close>: New.
(ctf_new_archive_internal): Adjust.
* ctf-archive.c (ctf_new_archive_internal): Likewise.
Initialize ctfi_unmap_on_close. Adjust error path.
(ctf_arc_bufopen): Adjust ctf_new_archive_internal call
(unmap_on_close is 0).
(ctf_arc_close): Only unmap if ctfi_unmap_on_close.
* ctf-open-bfd.c (ctf_fdopen): Adjust.
Report them as such, rather than letting ctf_decl_sprintf wrongly
conclude that the printing of zero characters means we are out of
memory.
libctf/
* ctf-types.c (ctf_type_aname): Return ECTF_CORRUPT if
ints, floats or typedefs have no name. Fix comment typo.
It is perfectly valid C to say e.g.
typedef u64 int;
struct foo_t
{
const volatile u64 wibble:2;
};
i.e. bitfields have to be integral types, but they can be cv-qualified
integral types or typedefs of same, etc.
This is easy to fix: do a ctf_type_resolve_unsliced() at creation time
to ensure the ultimate type is integral, and ctf_type_resolve() at
lookup time so that if you somehow have e.g. a slice of a typedef of a
slice of a cv-qualified int, we pull the encoding that the topmost slice
is based on out of the subsidiary slice (and then modify it), not out of
the underlying int. (This last bit is rather academic right now, since
all slices override exactly the same properties of the underlying type,
but it's still the right thing to do.)
libctf/
* ctf-create.c (ctf_add_slice): Support slices of any kind that
resolves to an integral type.
* ctf-types.c (ctf_type_encoding): Resolve the type before
fishing its encoding out.
Without this, an empty dict that is written out immediately never gets
any content at all: even the header is left empty.
libctf/
* ctf-create.c (ctf_create): Mark dirty.
A Solaris-era bug causes us to check the offsets of types with no names
against the first such type when ctf_add_type()ing members to a struct
or union. Members with no names (i.e. anonymous struct/union members)
can appear as many times as you like in a struct/union, so this check
should be skipped in this case.
libctf/
* ctf-create.c (membcmp) Skip nameless members.
This matters for the case of unnamed bitfields, whose names are the null
string. These are special in that they are the only members whose
"names" are allowed to be duplicated in a single struct, but we were
only handling this for the case where name == NULL. Translate "" to
NULL to help callers.
libctf/
* ctf-create.c (ctf_add_member_offset): Support names of ""
as if they were the null pointer.
When opening, we consider a forward with a kind above the maximum
allowable set of kinds and a forward of kind CTF_K_UNKNOWN to be a
forward to a struct. Whatever CTF version it was that produced
forwards with no associated kind, it predates anything we can read:
remove this wart.
libctf/
* ctf-open.c (init_types): Remove typeless CTF_K_FORWARD
special-casing.
One spot was missed when we rejigged ctf_update into ctf_serialize and
allowed all operations on dynamic containers: ctf_type_reference of
slices. A dynamic slice's vlen state is stored in the dtu_slice member,
so fetch it from there.
libctf/
* ctf-types.c (ctf_type_reference): Add support for dynamic slices.
This is technically unnecessary -- the compiler is quite capable of
doing the range reduction for us -- but it does mean that all
assignments of a ctf_id_t to its final uint32_t representation now have
appropriate explicit casts.
libctf/
* ctf-create.c (ctf_serialize): Add cast.
(ctf_add_slice): Likewise.
ctf_add_function assumes that function types' arglists are of type
ctf_id_t. Since they are CTF IDs, they are 32 bits wide, a uint32_t:
unfortunately ctf_id_t is a forward-compatible user-facing 64 bits wide,
and should never ever reach the CTF storage level.
All the CTF code other than ctf_add_function correctly assumes that
function arglists outside dynamic containers are 32 bits wide, so the
serialization machinery ends up cutting off half the arglist, corrupting
all args but the first (a good sign is a bunch of args of ID 0, the
unimplemented type, popping up).
Fix this by copying the arglist into place item by item, casting it
properly, at the same time as we validate the arg types. Fix the type
of the dtu_argv in the dynamic container and drop the now-unnecessary
cast in the serializer.
libctf/
* ctf-impl.h (ctf_dtdef_t) <dtu_argv>: Fix type.
* ctf-create.c (ctf_add_function): Check for unimplemented type
and populate at the same time. Populate one-by-one, not via
memcpy.
(ctf_serialize): Remove unnecessary cast.
* ctf-types.c (ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise. Fix comment typo.
The deduplicating linker adds types from the linker inputs to the output
via the same API everyone else does, so it's important that we can emit
everything that the compiler wants us to. Unfortunately, the compiler
may represent the unimplemented type (used for compiler constructs that
CTF cannot currently encode) as type zero or as a type of kind
CTF_K_UNKNOWN, and we don't allow the addition of types that cite the
former.
Adding this support adds a tiny bit of extra complexity: additions of
structure members immediately following a member of the unimplemented
type must be via ctf_add_member_offset or ctf_add_member_encoded, since
we have no idea how big members of the unimplemented type are.
(Attempts to do otherwise return -ECTF_NONREPRESENTABLE, like other
attempts to do forbidden things with the unimplemented type.)
Even slices of the unimplemented type are permitted: this is the only
case in which you can slice a type that terminates in a non-integral
type, on the grounds that it was likely integral in the source code,
it's just that we can't represent that sort of integral type properly
yet.
libctf/
* ctf-create.c (ctf_add_reftype): Support refs to type zero.
(ctf_add_array): Support array contents of type zero.
(ctf_add_function): Support arguments and return types of
type zero.
(ctf_add_typedef): Support typedefs to type zero.
(ctf_add_member_offset): Support members of type zero,
unless added at unspecified (naturally-aligned) offset.
Jose Marchesi noted that the traditional-Unix error array in ctf-error.c
introduces one reloc per error to initialize the array: 58 so far. We
can reduce this to zero using an array of carefully-sized individual
members which is used to construct a string table, that is then
referenced by the lookup functions: but doing this automatically is a
pain.
Bruno Haible wrote suitable code years ago: I got permission to reuse it
(Bruno says "... which I hereby put in the public domain"); I modified
it a tiny bit (similarly to what Ulrich Drepper did in the dsohowto
text, but I redid it from scratch), commented it up a bit, and shifted
the error table into that form, migrating it into the new file
ctf-error.h.
This has the advantage that it spotted both typos in the text of the
errors in the comments in ctf-api.h and typos in the error defines in
the comments in ctf-error.c, and places where the two were simply not
in sync. All are now fixed.
One new constant exists in ctf-api.h: CTF_NERR, since the old method of
working out the number of errors in ctf-error.c was no longer usable,
and it seems that the number of CTF errors is something users might
reasonably want as well. It should be pretty easy to keep up to date as
new errors are introduced.
include/
* ctf-api.h (ECTF_*): Improve comments.
(ECTF_NERR): New.
libctf/
* ctf-error.c: Include <stddef.h>, for offsetof.
(_ctf_errlist): Migrate to...
(_ctf_errlist_t): ... this.
(_ctf_erridx): New, indexes into _ctf_errlist_t.
(_ctf_nerr): Remove.
(ctf_errmsg): Adjust accordingly.
* Makefile.am (BUILT_SOURCES): Note...
(ctf-error.h): ... this new rule.
* Makefile.in: Regenerate.
* mkerrors.sed: New, process ctf-api.h to generate ctf-error.h.
* .gitignore: New, ignore ctf-error.h.
readelf * readelf.c (parse_args): Silence potential warnings about a
memory resource leak when allocating space for ctf option values.
(dump_section_as_ctf): Fix typo checking dump_ctf_strtab_name
variable.
libctf * ctf-archive.c (ctf_arc_write): Avoid calling close twice on the
same file descriptor.
We were not using the right configure machinery to spot libintl on
platforms where it was required, leading to the spurious failure of
various configure tests (e.g. for things like ELF support in BFD).
libctf/
* aclocal.m4: Add config/gettext-sister.m4: Shuffle into
alphabetical order.
* configure.ac: Add ZW_GNU_GETTEXT_SISTER_DIR.
* config.h.in: Regenerated.
* Makefile.in: Likewise.
* configure: Likewise.
At least one C library (uclibc-ng) defines some of these only when
the compiler is GCC. We might as well test for all three cases and
handle any of them being missing.
Very similar code exists in libctf and split between elfcpp and gold:
fix both.
(Also sync up elfcpp with a change made to libctf swap.h a few months
ago: since there is no out-of-line definition of the bswap replacements,
they should be declared static inline, not just inline, to prevent the
linker generating out-of-line references to them.)
PR libctf/25120
libctf/
* configure.ac: Check for bswap_16, bswap_32, and bswap_64 decls.
* swap.h (bswap_16): Do not assume that presence of <byteswap.h>
means this is declared.
(bswap_32): Likewise.
(bswap_64): Likewise.
(bswap_identity_64): Remove, unused.
* configure: Regenerated.
* config.h.in: Likewise.
gold/
* configure.ac: Check for bswap_16, bswap_32, and bswap_64 decls.
* configure: Regenerated.
* config.h.in: Likewise.
elfcpp/
* elfcpp_swap.h (bswap_16): Do not assume that presence of
<byteswap.h> means this is declared. Make static inline, matching
recent change to libctf, since there is no non-inline definition
of these functions.
(bswap_32): Likewise.
(bswap_64): Likewise.
This keeps archive searching threadsafe using the new bsearch_r that was
just added to libiberty.
PR25120
libctf/
* ctf-archive.c (search_nametbl): No longer global: declare...
(ctf_arc_open_by_name_internal): ... here. Use bsearch_r.
(search_modent_by_name): Take and use ARG for the nametbl.
objdump and readelf have one major CTF-related behavioural difference:
objdump can read .ctf sections that contain CTF archives and extract and
dump their members, while readelf cannot. Since the linker often emits
CTF archives, this means that readelf intermittently and (from the
user's perspective) randomly fails to read CTF in files that ld emits,
with a confusing error message wrongly claiming that the CTF content is
corrupt. This is purely because the archive-opening code in libctf was
needlessly tangled up with the BFD code, so readelf couldn't use it.
Here, we disentangle it, moving ctf_new_archive_internal from
ctf-open-bfd.c into ctf-archive.c and merging it with the helper
function in ctf-archive.c it was already using. We add a new public API
function ctf_arc_bufopen, that looks very like ctf_bufopen but returns
an archive given suitable section data rather than a ctf_file_t: the
archive is a ctf_archive_t, so it can be called on raw CTF dictionaries
(with no archive present) and will return a single-member synthetic
"archive".
There is a tiny lifetime tweak here: before now, the archive code could
assume that the symbol section in the ctf_archive_internal wrapper
structure was always owned by BFD if it was present and should always be
freed: now, the caller can pass one in via ctf_arc_bufopen, wihch has
the usual lifetime rules for such sections (caller frees): so we add an
extra field to track whether this is an internal call from ctf-open-bfd,
in which case we still free the symbol section.
include/
* ctf-api.h (ctf_arc_bufopen): New.
libctf/
* ctf-impl.h (ctf_new_archive_internal): Declare.
(ctf_arc_bufopen): Remove.
(ctf_archive_internal) <ctfi_free_symsect>: New.
* ctf-archive.c (ctf_arc_close): Use it.
(ctf_arc_bufopen): Fuse into...
(ctf_new_archive_internal): ... this, moved across from...
* ctf-open-bfd.c: ... here.
(ctf_bfdopen_ctfsect): Use ctf_arc_bufopen.
* libctf.ver: Add it.
binutils/
* readelf.c (dump_section_as_ctf): Support .ctf archives using
ctf_arc_bufopen. Automatically load the .ctf member of such
archives as the parent of all other members, unless specifically
overridden via --ctf-parent. Split out dumping code into...
(dump_ctf_archive_member): ... here, as in objdump, and call
it once per archive member.
(dump_ctf_indent_lines): Code style fix.
The C namespace a forward is located in is always the same as the
namespace of the corresponding complete type: 'struct foo' is in the
struct namespace and does not collide with, say, 'union foo'.
libctf allowed for this in many places, but inconsistently: in
particular, forward *addition* never allowed for this, and was interning
forwards in the default namespace, which is always wrong, since you can
only forward structs, unions and enums, all of which are in their own
namespaces in C.
Forward removal needs corresponding adjustment to remove the names form
the right namespace, as does ctf_rollback.
libctf/
* ctf-create.c (ctf_add_forward): Intern in the right namespace.
(ctf_dtd_delete): Remove correspondingly.
(ctf_rollback): Likewise.
When we add a type from a dictionary and then try to add it again, we
should hand it back unchanged unless it is a structure, union or enum
with a different number of members. That's what the comment says we do.
Instead, we hand it back unchanged *only* if it is a structure, union or
enum with the same number of members: non-structs, unions and enums are
unconditionally added. This causes extreme type bloating and (in
conjunction with the bug fixed by the next commit) can easily lead to
the same type being mistakenly added to a dictionary more than once
(which, for forwards, was not banned and led to dictionary corruption).
libctf/
* ctf-create.c (ctf_add_type_internal): Hand back existing types
unchanged.
This is what ctf_add_forward is documented to do, but it's not what it
actually does: the code is quite happy to add forwards that duplicate
existing structs, etc.
This is obviously wrong and breaks both the nondeduplicating linker
and the upcoming deduplicator, as well as allowing ordinary callers of
ctf_add_type to corrupt the dictionary by just adding the same root-
visible forward more than once.
libctf/
* ctf-create.c (ctf_add_forward): Don't add forwards to
types that already exist.
We were accidentally interning newly-added and newly-opened
non-root-visible types into name tables, and removing names from name
tables when such types were removed. This is very wrong: the whole
point of non-root-visible types is they do not go in name tables and
cannot be looked up by name. This bug made non-root-visible types
basically identical to root-visible types, right back to the earliest
days of libctf in the Solaris era.
libctf/
* ctf-open.c (init_types): Only intern root-visible types.
* ctf-create.c (ctf_dtd_insert): Likewise.
(ctf_dtd_delete): Only remove root-visible types.
(ctf_rollback): Likewise.
(ctf_add_generic): Adjust.
(ctf_add_struct_sized): Adjust comment.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
* ctf-impl.h (ctf_dtd_insert): Adjust prototype.
This is similar to cbbbc402e0 and fixes
a link error with duplicately defined symbols on FreeBSD.
libctf/ChangeLog:
* swap.h (bswap_identity_64): Make static.
This commit fixes a compilation warning when compiling libctf
on MinGW:
libctf/ctf-dump.c:118:8: warning: implicit declaration of function
'asprintf'; did you mean 'vasprintf'? [-Wimplicit-function-declaration]
if (asprintf (&bit, " %lx: [slice 0x%x:0x%x]",
^~~~~~~~
vasprintf
MinGW doesn't provide that function, so we depend on the one provided
by libiberty. However, the declaration is guarded by HAVE_DECL_ASPRINTF,
which we do not have in libctf's config.h.
libctf/ChangeLog:
PR binutils/25155:
* configure.ac: Add AC_CHECK_DECLS([asprintf]).
* configure, config.h.in: Regenerate.
When building binutils with mingw-w64, I get the following errors:
make[4]: Entering directory '/home/simark/build/binutils-gdb-mingw/binutils'
/bin/sh ./libtool --tag=CC --mode=link ccache x86_64-w64-mingw32-gcc -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Wstack-usage=262144 -Wno-format -Werror -I/home/simark/src/binutils-gdb/binutils/../zlib -g3 -O0 -D__USE_MINGW_ACCESS -Wl,--stack,12582912 -o objdump.exe objdump.o dwarf.o prdbg.o rddbg.o debug.o stabs.o rdcoff.o bucomm.o version.o filemode.o elfcomm.o ../opcodes/libopcodes.la ../libctf/libctf.la ../bfd/libbfd.la ../libiberty/libiberty.a -lintl
libtool: link: ccache x86_64-w64-mingw32-gcc -W -Wall -Wstrict-prototypes -Wmissing-prototypes -Wshadow -Wstack-usage=262144 -Wno-format -Werror -I/home/simark/src/binutils-gdb/binutils/../zlib -g3 -O0 -D__USE_MINGW_ACCESS -Wl,--stack -Wl,12582912 -o .libs/objdump.exe objdump.o dwarf.o prdbg.o rddbg.o debug.o stabs.o rdcoff.o bucomm.o version.o filemode.o elfcomm.o ../opcodes/.libs/libopcodes.a ../libctf/.libs/libctf.a -L/home/simark/build/binutils-gdb-mingw/zlib ../bfd/.libs/libbfd.a -lz ../libiberty/libiberty.a -lintl
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: ../libctf/.libs/libctf.a(ctf-open.o): in function `flip_header':
/home/simark/src/binutils-gdb/libctf/ctf-open.c:964: undefined reference to `bswap_16'
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: /home/simark/src/binutils-gdb/libctf/ctf-open.c:967: undefined reference to `bswap_32'
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: /home/simark/src/binutils-gdb/libctf/ctf-open.c:968: undefined reference to `bswap_32'
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: /home/simark/src/binutils-gdb/libctf/ctf-open.c:969: undefined reference to `bswap_32'
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: /home/simark/src/binutils-gdb/libctf/ctf-open.c:970: undefined reference to `bswap_32'
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: /home/simark/src/binutils-gdb/libctf/ctf-open.c:971: undefined reference to `bswap_32'
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: ../libctf/.libs/libctf.a(ctf-open.o):/home/simark/src/binutils-gdb/libctf/ctf-open.c:972: more undefined references to `bswap_32' follow
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: ../libctf/.libs/libctf.a(ctf-open.o): in function `flip_types':
/home/simark/src/binutils-gdb/libctf/ctf-open.c:1112: undefined reference to `bswap_16'
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: /home/simark/src/binutils-gdb/libctf/ctf-open.c:1113: undefined reference to `bswap_16'
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: /home/simark/src/binutils-gdb/libctf/ctf-open.c:1132: undefined reference to `bswap_32'
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: /home/simark/src/binutils-gdb/libctf/ctf-open.c:1133: undefined reference to `bswap_32'
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: /home/simark/src/binutils-gdb/libctf/ctf-open.c:1134: undefined reference to `bswap_32'
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: /home/simark/src/binutils-gdb/libctf/ctf-open.c:1135: undefined reference to `bswap_32'
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: /home/simark/src/binutils-gdb/libctf/ctf-open.c:1144: undefined reference to `bswap_32'
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: ../libctf/.libs/libctf.a(ctf-open.o):/home/simark/src/binutils-gdb/libctf/ctf-open.c:1145: more undefined references to `bswap_32' follow
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: ../libctf/.libs/libctf.a(ctf-open.o): in function `ctf_bufopen_internal':
/home/simark/src/binutils-gdb/libctf/ctf-open.c:1342: undefined reference to `bswap_16'
/usr/lib/gcc/x86_64-w64-mingw32/9.2.0/../../../../x86_64-w64-mingw32/bin/ld: ../libctf/.libs/libctf.a(ctf-open-bfd.o): in function `ctf_fdopen':
/home/simark/src/binutils-gdb/libctf/ctf-open-bfd.c:268: undefined reference to `bswap_16'
Apparently [1], if we have a function with `inline` but not `static`,
there should be a compilation unit defining the symbol too.
Alternatively, making those functions `static` fixes that.
[1] https://stackoverflow.com/questions/16245521/c99-inline-function-in-c-file/16254679#16254679
libctf/ChangeLog:
* swap.h (bswap_16, bswap_32, bswap_64): Make static.
Change-Id: I8fd12aedf6c90f9b7418af948e5e0bae0c32eead
A little tabdamage predating the linker patch series has crept in.
New in v5.
libctf/
* ctf-open.c (ctf_bufopen_internal): Fix tabdamage.
* ctf-types.c (ctf_type_lname): Likewise.
Calling ctf_import (fp, NULL) to cancel out a pre-existing import leaked
the refcnt increment on the parent, so it could never be freed.
New in v4.
libctf/
* ctf-open.c (ctf_import): Do not leak a ctf_file_t ref on every
ctf_import after the first for a given file.
ctf_dump calls ctf_str_append extensively but never checks to see if it
returns NULL (on OOM). If it ever does, we truncate the string we are
appending to and leak it!
Instead, create a variant of ctf_str_append that returns the *original
string* on OOM, and use it in ctf-dump. It is far better to omit a tiny
piece of a dump on OOM than to omit a bigger piece, and it is also
better to do this in what is after all purely debugging code than it is
to uglify ctf-dump.c with huge numbers of checks for the out-of-memory
case. Slightly truncated debugging output is better than no debugging
output at all and an out-of-memory message.
New in v4.
libctf/
* ctf-impl.h (ctf_str_append_noerr): Declare.
* ctf-util.c (ctf_str_append_noerr): Define in terms of
ctf_str_append.
* ctf-dump.c (str_append): New, call it.
(ctf_dump_format_type): Use str_append, not ctf_str_append.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
(ctf_dump_funcs): Likewise.
(ctf_dump_var): Likewise.
(ctf_dump_member): Likewise.
(ctf_dump_type): Likewise.
(ctf_dump): Likewise.
These just get in the way of auditing for erroneous usage of strdup and
add a huge irregular surface of "ctf_malloc or malloc? ctf_free or free?
ctf_strdup or strdup?"
ctf_malloc and ctf_free usage has not reliably matched up for many
years, if ever, making the whole game pointless.
Go back to malloc, free, and strdup like everyone else: while we're at
it, fix a bunch of places where we weren't properly checking for OOM.
This changes the interface of ctf_cuname_set and ctf_parent_name_set,
which could strdup but could not return errors (like ENOMEM).
New in v4.
include/
* ctf-api.h (ctf_cuname_set): Can now fail, returning int.
(ctf_parent_name_set): Likewise.
libctf/
* ctf-impl.h (ctf_alloc): Remove.
(ctf_free): Likewise.
(ctf_strdup): Likewise.
* ctf-subr.c (ctf_alloc): Remove.
(ctf_free): Likewise.
* ctf-util.c (ctf_strdup): Remove.
* ctf-create.c (ctf_serialize): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_function): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
(ctf_compress_write): Likewise.
(ctf_write_mem): Likewise.
* ctf-decl.c (ctf_decl_push): Likewise.
(ctf_decl_fini): Likewise.
(ctf_decl_sprintf): Likewise. Check for OOM.
* ctf-dump.c (ctf_dump_append): Use malloc, not ctf_alloc; free, not
ctf_free; strdup, not ctf_strdup.
(ctf_dump_free): Likewise.
(ctf_dump): Likewise.
* ctf-open.c (upgrade_types_v1): Likewise.
(init_types): Likewise.
(ctf_file_close): Likewise.
(ctf_bufopen_internal): Likewise. Check for OOM.
(ctf_parent_name_set): Likewise: report the OOM to the caller.
(ctf_cuname_set): Likewise.
(ctf_import): Likewise.
* ctf-string.c (ctf_str_purge_atom_refs): Use malloc, not ctf_alloc;
free, not ctf_free; strdup, not ctf_strdup.
(ctf_str_free_atom): Likewise.
(ctf_str_create_atoms): Likewise.
(ctf_str_add_ref_internal): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_write_strtab): Likewise.
If you call ctf_type_encoding() on a slice, you are meant to get the
encoding of the slice with the format of the underlying type. If
you call it on a non-int, non-fp, non-slice, you're meant to get the
error ECTF_INTNOTFP.
None of this was implemented for types in the dynamic space (which, now,
is *all* types in writable containers). Instead, we were always
returning the encoding as if it were a float, which for all other types
consulted the wrong part of a discriminated union and returned garbage.
(Curiously, existing users were more disturbed by the lack of an error
in the non-int/fp/slice case than they were about getting garbage back.)
libctf/
* ctf-types.c (ctf_type_encoding): Fix the dynamic case to
work right for non-int/fps.
The code was meant to handle this, but accidentally dereferenced the
null pointer before checking it for nullity.
v5: fix tabdamage.
libctf/
* ctf-types.c (ctf_type_name): Don't strlen a potentially-
null pointer.
The code to handle structures (and unions) that refer to themselves in
ctf_add_type is extremely dodgy. It works by looking through the list
of not-yet-committed types for a structure with the same name as the
structure in question and assuming, if it finds it, that this must be a
reference to the same type. This is a linear search that gets ever
slower as the dictionary grows, requiring you to call ctf_update at
intervals to keep performance tolerable: but if you do that, you run
into the problem that if a forward declared before the ctf_update is
changed to a structure afterwards, ctf_update explodes.
The last commit fixed most of this: this commit can use it, adding a new
ctf_add_processing hash that tracks source type IDs that are currently
being processed and uses it to avoid infinite recursion rather than the
dynamic type list: we split ctf_add_type into a ctf_add_type_internal,
so that ctf_add_type itself can become a wrapper that empties out this
being-processed hash once the entire recursive type addition is over.
Structure additions themselves avoid adding their dependent types
quite so much by checking the type mapping and avoiding re-adding types
we already know we have added.
We also add support for adding forwards to dictionaries that already
contain the thing they are a forward to: we just silently return the
original type.
v4: return existing struct/union/enum types properly, rather than using
an uninitialized variable: shrinks sizes of CTF sections back down
to roughly where they were in v1/v2 of this patch series.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_add_processing>: New.
* ctf-open.c (ctf_file_close): Free it.
* ctf-create.c (ctf_serialize): Adjust.
(membcmp): When reporting a conflict due to an error, report the
error.
(ctf_add_type): Turn into a ctf_add_processing wrapper. Rename to...
(ctf_add_type_internal): ... this. Hand back types we are already
in the middle of adding immediately. Hand back structs/unions with
the same number of members immediately. Do not walk the dynamic
list. Call ctf_add_type_internal, not ctf_add_type. Handle
forwards promoted to other types and the inverse case identically.
Add structs to the mapping as soon as we intern them, before they
gain any members.
The method of operation of libctf when the dictionary is writable has
before now been that types that are added land in the dynamic type
section, which is a linked list and hash of IDs -> dynamic type
definitions (and, recently a hash of names): the DTDs are a bit of CTF
representing the ctf_type_t and ad hoc C structures representing the
vlen. Historically, libctf was unable to do anything with these types,
not even look them up by ID, let alone by name: if you wanted to do that
say if you were adding a type that depended on one you just added) you
called ctf_update, which serializes all the DTDs into a CTF file and
reopens it, copying its guts over the fp it's called with. The
ctf_updated types are then frozen in amber and unchangeable: all lookups
will return the types in the static portion in preference to the dynamic
portion, and we will refuse to re-add things that already exist in the
static portion (and, of late, in the dynamic portion too). The libctf
machinery remembers the boundary between static and dynamic types and
looks in the right portion for each type. Lots of things still don't
quite work with dynamic types (e.g. getting their size), but enough
works to do a bunch of additions and then a ctf_update, most of the
time.
Except it doesn't, because ctf_add_type finds it necessary to walk the
full dynamic type definition list looking for types with matching names,
so it gets slower and slower with every type you add: fixing this
requires calling ctf_update periodically for no other reason than to
avoid massively slowing things down.
This is all clunky and very slow but kind of works, until you consider
that it is in fact possible and indeed necessary to modify one sort of
type after it has been added: forwards. These are necessarily promoted
to structs, unions or enums, and when they do so *their type ID does not
change*. So all of a sudden we are changing types that already exist in
the static portion. ctf_update gets massively confused by this and
allocates space enough for the forward (with no members), but then emits
the new dynamic type (with all the members) into it. You get an
assertion failure after that, if you're lucky, or a coredump.
So this commit rejigs things a bit and arranges to exclusively use the
dynamic type definitions in writable dictionaries, and the static type
definitions in readable dictionaries: we don't at any time have a mixture
of static and dynamic types, and you don't need to call ctf_update to
make things "appear". The ctf_dtbyname hash I introduced a few months
ago, which maps things like "struct foo" to DTDs, is removed, replaced
instead by a change of type of the four dictionaries which track names.
Rather than just being (unresizable) ctf_hash_t's populated only at
ctf_bufopen time, they are now a ctf_names_t structure, which is a pair
of ctf_hash_t and ctf_dynhash_t, with the ctf_hash_t portion being used
in readonly dictionaries, and the ctf_dynhash_t being used in writable
ones. The decision as to which to use is centralized in the new
functions ctf_lookup_by_rawname (which takes a type kind) and
ctf_lookup_by_rawhash, which it calls (which takes a ctf_names_t *.)
This change lets us switch from using static to dynamic name hashes on
the fly across the entirety of libctf without complexifying anything: in
fact, because we now centralize the knowledge about how to map from type
kind to name hash, it actually simplifies things and lets us throw out
quite a lot of now-unnecessary complexity, from ctf_dtnyname (replaced
by the dynamic half of the name tables), through to ctf_dtnextid (now
that a dictionary's static portion is never referenced if the dictionary
is writable, we can just use ctf_typemax to indicate the maximum type:
dynamic or non-dynamic does not matter, and we no longer need to track
the boundary between the types). You can now ctf_rollback() as far as
you like, even past a ctf_update or for that matter a full writeout; all
the iteration functions work just as well on writable as on read-only
dictionaries; ctf_add_type no longer needs expensive duplicated code to
run over the dynamic types hunting for ones it might be interested in;
and the linker no longer needs a hack to call ctf_update so that calling
ctf_add_type is not impossibly expensive.
There is still a bit more complexity: some new code paths in ctf-types.c
need to know how to extract information from dynamic types. This
complexity will go away again in a few months when libctf acquires a
proper intermediate representation.
You can still call ctf_update if you like (it's public API, after all),
but its only effect now is to set the point to which ctf_discard rolls
back.
Obviously *something* still needs to serialize the CTF file before
writeout, and this job is done by ctf_serialize, which does everything
ctf_update used to except set the counter used by ctf_discard. It is
automatically called by the various functions that do CTF writeout:
nobody else ever needs to call it.
With this in place, forwards that are promoted to non-forwards no longer
crash the link, even if it happens tens of thousands of types later.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_names_t): New.
(ctf_lookup_t) <ctf_hash>: Now a ctf_names_t, not a ctf_hash_t.
(ctf_file_t) <ctf_structs>: Likewise.
<ctf_unions>: Likewise.
<ctf_enums>: Likewise.
<ctf_names>: Likewise.
<ctf_lookups>: Improve comment.
<ctf_ptrtab_len>: New.
<ctf_prov_strtab>: New.
<ctf_str_prov_offset>: New.
<ctf_dtbyname>: Remove, redundant to the names hashes.
<ctf_dtnextid>: Remove, redundant to ctf_typemax.
(ctf_dtdef_t) <dtd_name>: Remove.
<dtd_data>: Note that the ctt_name is now populated.
(ctf_str_atom_t) <csa_offset>: This is now the strtab
offset for internal strings too.
<csa_external_offset>: New, the external strtab offset.
(CTF_INDEX_TO_TYPEPTR): Handle the LCTF_RDWR case.
(ctf_name_table): New declaration.
(ctf_lookup_by_rawname): Likewise.
(ctf_lookup_by_rawhash): Likewise.
(ctf_set_ctl_hashes): Likewise.
(ctf_serialize): Likewise.
(ctf_dtd_insert): Adjust.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
(ctf_list_empty_p): Likewise.
(ctf_str_remove_ref): Likewise.
(ctf_str_add): Returns uint32_t now.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Now returns a boolean (int).
* ctf-string.c (ctf_strraw_explicit): Check the ctf_prov_strtab
for strings in the appropriate range.
(ctf_str_create_atoms): Create the ctf_prov_strtab. Detect OOM
when adding the null string to the new strtab.
(ctf_str_free_atoms): Destroy the ctf_prov_strtab.
(ctf_str_add_ref_internal): Add make_provisional argument. If
make_provisional, populate the offset and fill in the
ctf_prov_strtab accordingly.
(ctf_str_add): Return the offset, not the string.
(ctf_str_add_ref): Likewise.
(ctf_str_add_external): Return a success integer.
(ctf_str_remove_ref): New, remove a single ref.
(ctf_str_count_strtab): Do not count the initial null string's
length or the existence or length of any unreferenced internal
atoms.
(ctf_str_populate_sorttab): Skip atoms with no refs.
(ctf_str_write_strtab): Populate the nullstr earlier. Add one
to the cts_len for the null string, since it is no longer done
in ctf_str_count_strtab. Adjust for csa_external_offset rename.
Populate the csa_offset for both internal and external cases.
Flush the ctf_prov_strtab afterwards, and reset the
ctf_str_prov_offset.
* ctf-create.c (ctf_grow_ptrtab): New.
(ctf_create): Call it. Initialize new fields rather than old
ones. Tell ctf_bufopen_internal that this is a writable dictionary.
Set the ctl hashes and data model.
(ctf_update): Rename to...
(ctf_serialize): ... this. Leave a compatibility function behind.
Tell ctf_simple_open_internal that this is a writable dictionary.
Pass the new fields along from the old dictionary. Drop
ctf_dtnextid and ctf_dtbyname. Use ctf_strraw, not dtd_name.
Do not zero out the DTD's ctt_name.
(ctf_prefixed_name): Rename to...
(ctf_name_table): ... this. No longer return a prefixed name: return
the applicable name table instead.
(ctf_dtd_insert): Use it, and use the right name table. Pass in the
kind we're adding. Migrate away from dtd_name.
(ctf_dtd_delete): Adjust similarly. Remove the ref to the
deleted ctt_name.
(ctf_dtd_lookup_type_by_name): Remove.
(ctf_dynamic_type): Always return NULL on read-only dictionaries.
No longer check ctf_dtnextid: check ctf_typemax instead.
(ctf_snapshot): No longer use ctf_dtnextid: use ctf_typemax instead.
(ctf_rollback): Likewise. No longer fail with ECTF_OVERROLLBACK. Use
ctf_name_table and the right name table, and migrate away from
dtd_name as in ctf_dtd_delete.
(ctf_add_generic): Pass in the kind explicitly and pass it to
ctf_dtd_insert. Use ctf_typemax, not ctf_dtnextid. Migrate away
from dtd_name to using ctf_str_add_ref to populate the ctt_name.
Grow the ptrtab if needed.
(ctf_add_encoded): Pass in the kind.
(ctf_add_slice): Likewise.
(ctf_add_array): Likewise.
(ctf_add_function): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_reftype): Likewise. Initialize the ctf_ptrtab, checking
ctt_name rather than dtd_name.
(ctf_add_struct_sized): Pass in the kind. Use
ctf_lookup_by_rawname, not ctf_hash_lookup_type /
ctf_dtd_lookup_type_by_name.
(ctf_add_union_sized): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_type): Likewise.
(ctf_compress_write): Call ctf_serialize: adjust for ctf_size not
being initialized until after the call.
(ctf_write_mem): Likewise.
(ctf_write): Likewise.
* ctf-archive.c (arc_write_one_ctf): Likewise.
* ctf-lookup.c (ctf_lookup_by_name): Use ctf_lookuup_by_rawhash, not
ctf_hash_lookup_type.
(ctf_lookup_by_id): No longer check the readonly types if the
dictionary is writable.
* ctf-open.c (init_types): Assert that this dictionary is not
writable. Adjust to use the new name hashes, ctf_name_table,
and ctf_ptrtab_len. GNU style fix for the final ptrtab scan.
(ctf_bufopen_internal): New 'writable' parameter. Flip on LCTF_RDWR
if set. Drop out early when dictionary is writable. Split the
ctf_lookups initialization into...
(ctf_set_cth_hashes): ... this new function.
(ctf_simple_open_internal): Adjust. New 'writable' parameter.
(ctf_simple_open): Adjust accordingly.
(ctf_bufopen): Likewise.
(ctf_file_close): Destroy the appropriate name hashes. No longer
destroy ctf_dtbyname, which is gone.
(ctf_getdatasect): Remove spurious "extern".
* ctf-types.c (ctf_lookup_by_rawname): New, look up types in the
specified name table, given a kind.
(ctf_lookup_by_rawhash): Likewise, given a ctf_names_t *.
(ctf_member_iter): Add support for iterating over the
dynamic type list.
(ctf_enum_iter): Likewise.
(ctf_variable_iter): Likewise.
(ctf_type_rvisit): Likewise.
(ctf_member_info): Add support for types in the dynamic type list.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-link.c (ctf_accumulate_archive_names): No longer call
ctf_update.
(ctf_link_write): Likewise.
(ctf_link_intern_extern_string): Adjust for new
ctf_str_add_external return value.
(ctf_link_add_strtab): Likewise.
* ctf-util.c (ctf_list_empty_p): New.
GCC can emit references to type 0 to indicate that this type is one that
is not representable in the version of CTF it emits (for instance,
version 3 cannot encode vector types). Type 0 is already used in the
function section to indicate padding inserted to skip functions we do
not want to encode the type of, so using zero in this way is a good
extension of the format: but libctf reports such types as ECTF_BADID,
which is indistinguishable from file corruption via links to truly
nonexistent types with IDs like 0xDEADBEEF etc, which we really do want
to stop for.
In particular, this stops all traversals of types dead at this point,
preventing us from even dumping CTF files containing unrepresentable
types to see what's going on!
So add a new error, ECTF_NONREPRESENTABLE, which is returned by
recursive type resolution when a reference to a zero type is found. (No
zero type is ever emitted into the CTF file by GCC, only references to
one). We can't do much with types that are ultimately nonrepresentable,
but we can do enough to keep functioning.
Adjust ctf_add_type to ensure that top-level types of type zero and
structure and union members of ultimate type zero are simply skipped
without reporting an error, so we can copy structures and unions that
contain nonrepresentable members (skipping them and leaving a hole where
they would be, so no consumers downstream of the linker need to worry
about this): adjust the dumper so that we dump members of
nonrepresentable types in a simple form that indicates
nonrepresentability rather than terminating the dump, and do not falsely
assume all errors to be -ENOMEM: adjust the linker so that types that
fail to get added are simply skipped, so that both nonrepresentable
types and outright errors do not terminate the type addition, which
could skip many valid types and cause further errors when variables of
those types are added.
In future, when we gain the ability to call back to the linker to report
link-time type resolution errors, we should report failures to add all
but nonrepresentable types. But we can't do that yet.
v5: Fix tabdamage.
include/
* ctf-api.h (ECTF_NONREPRESENTABLE): New.
libctf/
* ctf-types.c (ctf_type_resolve): Return ECTF_NONREPRESENTABLE on
type zero.
* ctf-create.c (ctf_add_type): Detect and skip nonrepresentable
members and types.
(ctf_add_variable): Likewise for variables pointing to them.
* ctf-link.c (ctf_link_one_type): Do not warn for nonrepresentable
type link failure, but do warn for others.
* ctf-dump.c (ctf_dump_format_type): Likewise. Do not assume all
errors to be ENOMEM.
(ctf_dump_member): Likewise.
(ctf_dump_type): Likewise.
(ctf_dump_header_strfield): Do not assume all errors to be ENOMEM.
(ctf_dump_header_sectfield): Do not assume all errors to be ENOMEM.
(ctf_dump_header): Likewise.
(ctf_dump_label): likewise.
(ctf_dump_objts): likewise.
(ctf_dump_funcs): likewise.
(ctf_dump_var): likewise.
(ctf_dump_str): Likewise.
This lets other programs read and write CTF-format data.
Two versioned shared libraries are created: libctf.so and
libctf-nobfd.so. They contain identical content except that
libctf-nobfd.so contains no references to libbfd and does not implement
ctf_open, ctf_fdopen, ctf_bfdopen or ctf_bfdopen_ctfsect, so it can be
used by programs that cannot use BFD, like readelf.
The soname major version is presently .0 until the linker API
stabilizes, when it will flip to .1 and hopefully never change again.
New in v3.
v4: libtoolize and turn into a pair of shared libraries. Drop
--enable-install-ctf: now controlled by --enable-shared and
--enable-install-libbfd, like everything else.
v5: Add ../bfd to ACLOCAL_AMFLAGS and AC_CONFIG_MACRO_DIR. Fix tabdamage.
* Makefile.def (host_modules): libctf is no longer no_install.
* Makefile.in: Regenerated.
libctf/
* configure.ac (AC_DISABLE_SHARED): New, like opcodes/.
(LT_INIT): Likewise.
(AM_INSTALL_LIBBFD): Likewise.
(dlopen): Note why this is necessary in a comment.
(SHARED_LIBADD): Initialize for possibly-PIC libiberty: derived from
opcodes/.
(SHARED_LDFLAGS): Likewise.
(BFD_LIBADD): Likewise, for libbfd.
(BFD_DEPENDENCIES): Likewise.
(VERSION_FLAGS): Initialize, using a version script if ld supports
one, or libtool -export-symbols-regex otherwise.
(AC_CONFIG_MACRO_DIR): Add ../BFD.
* Makefile.am (ACLOCAL_AMFLAGS): Likewise.
(INCDIR): New.
(AM_CPPFLAGS): Use $(srcdir), not $(top_srcdir).
(noinst_LIBRARIES): Replace with...
[INSTALL_LIBBFD] (lib_LTLIBRARIES): This, or...
[!INSTALL_LIBBFD] (noinst_LTLIBRARIES): ... this, mentioning new
libctf-nobfd.la as well.
[INSTALL_LIBCTF] (include_HEADERS): Add the CTF headers.
[!INSTALL_LIBCTF] (include_HEADERS): New, empty.
(libctf_a_SOURCES): Rename to...
(libctf_nobfd_la_SOURCES): ... this, all of libctf other than
ctf-open-bfd.c.
(libctf_la_SOURCES): Now derived from libctf_nobfd_la_SOURCES,
with ctf-open-bfd.c added.
(libctf_nobfd_la_LIBADD): New, using @SHARED_LIBADD@.
(libctf_la_LIBADD): New, using @BFD_LIBADD@ as well.
(libctf_la_DEPENDENCIES): New, using @BFD_DEPENDENCIES@.
* Makefile.am [INSTALL_LIBCTF]: Use it.
* aclocal.m4: Add ../bfd/acinclude.m4, ../config/acx.m4, and the
libtool macros.
* libctf.ver: New, everything is version LIBCTF_1.0 currently (even
the unstable components).
* Makefile.in: Regenerated.
* config.h.in: Likewise.
* configure: Likewise.
binutils/
* Makefile.am (LIBCTF): Mention the .la file.
(LIBCTF_NOBFD): New.
(readelf_DEPENDENCIES): Use it.
(readelf_LDADD): Likewise.
* Makefile.in: Regenerated.
ld/
* configure.ac (TESTCTFLIB): Set to the .so or .a, like TESTBFDLIB.
* Makefile.am (TESTCTFLIB): Use it.
(LIBCTF): Use the .la file.
(check-DEJAGNU): Use it.
* Makefile.in: Regenerated.
* configure: Likewise.
include/
* ctf-api.h: Note the instability of the ctf_link interfaces.
When we do a ctf_fdopen, we open things via bfd_fdopenr and set up a
hook to close the bfd again... but then we never actually call that hook
from anywhere, so we eventually leak every bfd we open.
Fix this by calling the hook (if set) in ctf_arc_close.
New in v3.
libctf/
* ctf-archive.c (ctf_arc_close): Call ctfi_bfd_close if set.
* ctf-open-bfd.c (ctf_bfdclose): Fix comment.
Without this, the FD is only closed when the CTF file is, leading to
running out of fds on (e.g.) very large links.
New in v3.
libctf/
* ctf-open-bfd.c (ctf_fdopen): Call bfd_set_cacheable.
This hoary old header defines things like MAX that users of libctf might
perfectly reasonably define themselves.
The CTF headers do not need it: move it into libctf/ctf-impl.h instead.
include/
* ctf-api.h (includes): No longer include <sys/param.h>.
libctf/
* ctf-impl.h (includes): Include <sys/param.h> here.
When a ctf_dynhash_insert() finds a slot already existing, it should
call the key and value free functions on the existing key and value and
move the passed-in key into place, so that the lifetime rules for hash
keys are always the same no matter whether the key existed or not but
neither are the keys or values leaked.
New in v3.
v5: fix tabdamage.
libctf/
* ctf-hash.c (ctf_hashtab_insert): Pass in the key and value
freeing functions: if set, free the key and value if the slot
already exists. Always reassign the key.
(ctf_dynhash_insert): Adjust call appropriately.
(ctf_hash_insert_type): Likewise.
This machinery has been broken for as long as Solaris has existed.
Forwards are meant to encode "struct foo;", "enum foo;" or "union
foo;". Obviously these all exist in distinct namespaces, so forwards
store the type kind they forward to in their ctt_type member
(which makes conceptual sense if you squint at it). The addition
machinery uses this to promote forwards to the appropriate type as
needed.
Unfortunately ctf_add_type does not: it checks the global namespace
(which is always wrong), and so fails with a spurious conflict if you
have, say, a typedef and then a forward comes along with the same name,
even if it's a forward to something like a struct. (This was observed
with <libio.h>, which has "struct _IO_FILE;" and also
"typedef struct _IO_FILE _IO_FILE"). We should look at the recorded
type kind and look in the appropriate namespace. We should also,
when creating the forward in the new container, use that type kind,
rather than just defaulting to CTF_K_STRUCT and hoping that what
eventually comes along is a struct.
This bug is as old as the first implementation of ctf_add_type in
Solaris. But we also want a new feature for the linker, closely-related
and touching the same code so we add it here: not only do we want a
forward followed by a struct/union/enum to promote the forward, but
we want want a struct/union/enum followed by a forward to act as a NOP
and return the existing type, because when we're adding many files
in succession to a target link, there will often be already-promoted
forwards (in the shape of a struct/union/enum) that want to unify
with duplicate forwards coming from other object files.
v5: fix tabdamage.
libctf/
* ctf-create.c (ctf_add_type): Look up and use the forwarded-to
type kind. Allow forwards to unify with pre-existing structs/
unions/enums.
Once the deduplicator is capable of actually detecting conflicting types
with the same name (i.e., not yet) we will place such conflicting types,
and types that depend on them, into CTF dictionaries that are the child
of the main dictionary we usually emit: currently, this will lead to the
.ctf section becoming a CTF archive rather than a single dictionary,
with the default-named archive member (_CTF_SECTION, or NULL) being the
main shared dictionary with most of the types in it.
By default, the sections are named after the compilation unit they come
from (complete path and all), with the cuname field in the CTF header
providing further evidence of the name without requiring the caller to
engage in tiresome parsing. But some callers may not wish the mapping
from input CU to output sub-dictionary to be purely CU-based.
The machinery here allows this to be freely changed, in two ways:
- callers can call ctf_link_add_cu_mapping to specify that a single
input compilation unit should have its types placed in some other CU
if they conflict: the CU will always be created, even if empty, so
the consuming program can depend on its existence. You can map
multiple input CUs to one output CU to force all their types to be
merged together: if some of *those* types conflict, the behaviour is
currently unspecified (the new deduplicator will specify it).
- callers can call ctf_link_set_memb_name_changer to provide a function
which is passed every CTF sub-dictionary name in turn (including
_CTF_SECTION) and can return a new name, or NULL if no change is
desired. The mapping from input to output names should not map two
input names to the same output name: if this happens, the two are not
merged but will result in an archive with two members with the same
name (technically valid, but it's hard to access the second
same-named member: you have to do an iteration over archive members).
This is used by the kernel's ctfarchive machinery (not yet upstream) to
encode CTF under member names like {module name}.ctf rather than
.ctf.CU, but it is anticipated that other large projects may wish to
have their own storage for CTF outside of .ctf sections and may wish to
have new naming schemes that suit their special-purpose consumers.
New in v3.
v4: check for strdup failure.
v5: fix tabdamage.
include/
* ctf-api.h (ctf_link_add_cu_mapping): New.
(ctf_link_memb_name_changer_f): New.
(ctf_link_set_memb_name_changer): New.
libctf/
* ctf-impl.h (ctf_file_t) <ctf_link_cu_mappping>: New.
<ctf_link_memb_name_changer>: Likewise.
<ctf_link_memb_name_changer_arg>: Likewise.
* ctf-create.c (ctf_update): Update accordingly.
* ctf-open.c (ctf_file_close): Likewise.
* ctf-link.c (ctf_create_per_cu): Apply the cu mapping.
(ctf_link_add_cu_mapping): New.
(ctf_link_set_memb_name_changer): Likewise.
(ctf_change_parent_name): New.
(ctf_name_list_accum_cb_arg_t) <dynames>: New, storage for names
allocated by the caller's ctf_link_memb_name_changer.
<ndynames>: Likewise.
(ctf_accumulate_archive_names): Call the ctf_link_memb_name_changer.
(ctf_link_write): Likewise (for _CTF_SECTION only): also call
ctf_change_parent_name. Free any resulting names.
The compiler describes the name and type of all file-scope variables in
this section. Merging it at link time requires using the type mapping
added in the previous commit to determine the appropriate type for the
variable in the output, given its type in the input: we check the shared
container first, and if the type doesn't exist there, it must be a
conflicted type in the per-CU child, and the variable should go there
too. We also put the variable in the per-CU child if a variable with
the same name but a different type already exists in the parent: we
ignore any such conflict in the child because CTF cannot represent such
things, nor can they happen unless a third-party linking program has
overridden the mapping of CU to CTF archive member name (using machinery
added in a later commit).
v3: rewritten using an algorithm that actually works in the case of
conflicting names. Some code motion from the next commit. Set
the per-CU parent name.
v4: check for strdup failure.
v5: fix tabdamage.
include/
* ctf-api.h (ECTF_INTERNAL): New.
libctf/
* ctf-link.c (ctf_create_per_cu): New, refactored out of...
(ctf_link_one_type): ... here, with parent-name setting added.
(check_variable): New.
(ctf_link_one_variable): Likewise.
(ctf_link_one_input_archive_member): Call it.
* ctf-error.c (_ctf_errlist): Updated with new errors.
This lets you call ctf_type_mapping (dest_fp, src_fp, src_type_id)
and get told what type ID the corresponding type has in the target
ctf_file_t. This works even if it was added by a recursive call, and
because it is stored in the target ctf_file_t it works even if we
had to add one type to multiple ctf_file_t's as part of conflicting
type handling.
We empty out this mapping after every archive is linked: because it maps
input to output fps, and we only visit each input fp once, its contents
are rendered entirely useless every time the source fp changes.
v3: add several missing mapping additions. Add ctf_dynhash_empty, and
empty after every input archive.
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t): New field ctf_link_type_mapping.
(struct ctf_link_type_mapping_key): New.
(ctf_hash_type_mapping_key): Likewise.
(ctf_hash_eq_type_mapping_key): Likewise.
(ctf_add_type_mapping): Likewise.
(ctf_type_mapping): Likewise.
(ctf_dynhash_empty): Likewise.
* ctf-open.c (ctf_file_close): Update accordingly.
* ctf-create.c (ctf_update): Likewise.
(ctf_add_type): Populate the mapping.
* ctf-hash.c (ctf_hash_type_mapping_key): Hash a type mapping key.
(ctf_hash_eq_type_mapping_key): Check the key for equality.
(ctf_dynhash_insert): Fix comment typo.
(ctf_dynhash_empty): New.
* ctf-link.c (ctf_add_type_mapping): New.
(ctf_type_mapping): Likewise.
(empty_link_type_mapping): New.
(ctf_link_one_input_archive): Call it.
This is the start of work on the core of the linking mechanism for CTF
sections. This commit handles the type and string sections.
The linker calls these functions in sequence:
ctf_link_add_ctf: to add each CTF section in the input in turn to a
newly-created ctf_file_t (which will appear in the output, and which
itself will become the shared parent that contains types that all
TUs have in common (in all link modes) and all types that do not
have conflicting definitions between types (by default). Input files
that are themselves products of ld -r are supported, though this is
not heavily tested yet.
ctf_link: called once all input files are added to merge the types in
all the input containers into the output container, eliminating
duplicates.
ctf_link_add_strtab: called once the ELF string table is finalized and
all its offsets are known, this calls a callback provided by the
linker which returns the string content and offset of every string in
the ELF strtab in turn: all these strings which appear in the input
CTF strtab are eliminated from it in favour of the ELF strtab:
equally, any strings that only appear in the input strtab will
reappear in the internal CTF strtab of the output.
ctf_link_shuffle_syms (not yet implemented): called once the ELF symtab
is finalized, this calls a callback provided by the linker which
returns information on every symbol in turn as a ctf_link_sym_t. This
is then used to shuffle the function info and data object sections in
the CTF section into symbol table order, eliminating the index
sections which map those sections to symbol names before that point.
Currently just returns ECTF_NOTYET.
ctf_link_write: Returns a buffer containing either a serialized
ctf_file_t (if there are no types with conflicting definitions in the
object files in the link) or a ctf_archive_t containing a large
ctf_file_t (the common types) and a bunch of small ones named after
individual CUs in which conflicting types are found (containing the
conflicting types, and all types that reference them). A threshold
size above which compression takes place is passed as one parameter.
(Currently, only gzip compression is supported, but I hope to add lzma
as well.)
Lifetime rules for this are simple: don't close the input CTF files
until you've called ctf_link for the last time. We do not assume
that symbols or strings passed in by the callback outlast the
call to ctf_link_add_strtab or ctf_link_shuffle_syms.
Right now, the duplicate elimination mechanism is the one already
present as part of the ctf_add_type function, and is not particularly
good: it misses numerous actual duplicates, and the conflicting-types
detection hardly ever reports that types conflict, even when they do
(one of them just tends to get silently dropped): it is also very slow.
This will all be fixed in the next few weeks, but the fix hardly touches
any of this code, and the linker does work without it, just not as
well as it otherwise might. (And when no CTF section is present,
there is no effect on performance, of course. So only people using
a trunk GCC with not-yet-committed patches will even notice. By the
time it gets upstream, things should be better.)
v3: Fix error handling.
v4: check for strdup failure.
v5: fix tabdamage.
include/
* ctf-api.h (struct ctf_link_sym): New, a symbol in flight to the
libctf linking machinery.
(CTF_LINK_SHARE_UNCONFLICTED): New.
(CTF_LINK_SHARE_DUPLICATED): New.
(ECTF_LINKADDEDLATE): New, replacing ECTF_UNUSED.
(ECTF_NOTYET): New, a 'not yet implemented' message.
(ctf_link_add_ctf): New, add an input file's CTF to the link.
(ctf_link): New, merge the type and string sections.
(ctf_link_strtab_string_f): New, callback for feeding strtab info.
(ctf_link_iter_symbol_f): New, callback for feeding symtab info.
(ctf_link_add_strtab): New, tell the CTF linker about the ELF
strtab's strings.
(ctf_link_shuffle_syms): New, ask the CTF linker to shuffle its
symbols into symtab order.
(ctf_link_write): New, ask the CTF linker to write the CTF out.
libctf/
* ctf-link.c: New file, linking of the string and type sections.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
* ctf-impl.h (ctf_file_t): New fields ctf_link_inputs,
ctf_link_outputs.
* ctf-create.c (ctf_update): Update accordingly.
* ctf-open.c (ctf_file_close): Likewise.
* ctf-error.c (_ctf_errlist): Updated with new errors.
We weren't correctly detecting when there were no functions to dump in
the function info table, because we were checking for ECTF_NOTYPEDAT,
which means there are no *data objects* to dump.
Adjust accordingly.
libctf/
* ctf-dump.c (ctf_dump_funcs): Check the right error value.
Use the recently-added ctf_type_iter_all function to iterate over
non-root types, too, indicating them via {....} surrounding the type
description in the dump.
libctf/
* ctf-dump.c (ctf_dump): Use ctf_type_iter_all to dump types, not
ctf_type_iter.
(ctf_dump_type): Pass down the flag from ctf_type_iter_all.
(ctf_dump_format_type): Add non-root-type { } notation.
Add root flag to prototype.
(ctf_dump_label): Adjust accordingly.
(ctf_dump_objts): Likewise.
(ctf_dump_var): Likewise.
We were freeing the compressed data buffer twice if compression failed.
v4: Fix commit message.
v5: fix tabdamage.
libctf/
* ctf-create.c (ctf_compress_write): Fix double-free.
Before now, we've been able to write CTF files to gzFile descriptors or
fds, and CTF archives to named files only.
Make this a bit less irregular by allowing CTF archives to be written
to fds with the new function ctf_arc_write_fd: also allow CTF
files to be written to a new memory buffer via ctf_write_mem.
(It would be nice to complete things by adding a new function to write
CTF archives to memory, but this is too difficult to do given the short
time the linker is expected to be writing them out: we will transition
to a better format in format v4, though we will always support reading
CTF archives that are stored in .ctf sections.)
include/
* ctf-api.h (ctf_arc_write_fd): New.
(ctf_write_mem): Likewise.
(ctf_gzwrite): Spacing fix.
libctf/
* ctf-archive.c (ctf_arc_write): Split off, and reimplement in terms
of...
(ctf_arc_write_fd): ... this new function.
* ctf-create.c (ctf_write_mem): New.
The CTF file format has always supported "external strtabs", which
internally are strtab offsets with their MSB on: such refs
get their strings from the strtab passed in at CTF file open time:
this is usually intended to be the ELF strtab, and that's what this
implementation is meant to support, though in theory the external
strtab could come from anywhere.
This commit adds support for these external strings in the ctf-string.c
strtab tracking layer. It's quite easy: we just add a field csa_offset
to the atoms table that tracks all strings: this field tracks the offset
of the string in the ELF strtab (with its MSB already on, courtesy of a
new macro CTF_SET_STID), and adds a new function that sets the
csa_offset to the specified offset (plus MSB). Then we just need to
avoid writing out strings to the internal strtab if they have csa_offset
set, and note that the internal strtab is shorter than it might
otherwise be.
(We could in theory save a little more time here by eschewing sorting
such strings, since we never actually write the strings out anywhere,
but that would mean storing them separately and it's just not worth the
complexity cost until profiling shows it's worth doing.)
We also have to go through a bit of extra effort at variable-sorting
time. This was previously using direct references to the internal
strtab: it couldn't use ctf_strptr or ctf_strraw because the new strtab
is not yet ready to put in its usual field (in a ctf_file_t that hasn't
even been allocated yet at this stage): but now we're using the external
strtab, this will no longer do because it'll be looking things up in the
wrong strtab, with disastrous results. Instead, pass the new internal
strtab in to a new ctf_strraw_explicit function which is just like
ctf_strraw except you can specify a ne winternal strtab to use.
But even now that it is using a new internal strtab, this is not quite
enough: it can't look up strings in the external strtab because ld
hasn't written it out yet, and when it does will write it straight to
disk. Instead, when we write the internal strtab, note all the offset
-> string mappings that we have noted belong in the *external* strtab to
a new "synthetic external strtab" dynhash, ctf_syn_ext_strtab, and look
in there at ctf_strraw time if it is set. This uses minimal extra
memory (because only strings in the external strtab that we actually use
are stored, and even those come straight out of the atoms table), but
let both variable sorting and name interning when ctf_bufopen is next
called work fine. (This also means that we don't need to filter out
spurious ECTF_STRTAB warnings from ctf_bufopen but can pass them back to
the caller, once we wrap ctf_bufopen so that we have a new internal
variant of ctf_bufopen etc that we can pass the synthetic external
strtab to. That error has been filtered out since the days of Solaris
libctf, which didn't try to handle the problem of getting external
strtabs right at construction time at all.)
v3: add the synthetic strtab and all associated machinery.
v5: fix tabdamage.
include/
* ctf.h (CTF_SET_STID): New.
libctf/
* ctf-impl.h (ctf_str_atom_t) <csa_offset>: New field.
(ctf_file_t) <ctf_syn_ext_strtab>: Likewise.
(ctf_str_add_ref): Name the last arg.
(ctf_str_add_external) New.
(ctf_str_add_strraw_explicit): Likewise.
(ctf_simple_open_internal): Likewise.
(ctf_bufopen_internal): Likewise.
* ctf-string.c (ctf_strraw_explicit): Split from...
(ctf_strraw): ... here, with new support for ctf_syn_ext_strtab.
(ctf_str_add_ref_internal): Return the atom, not the
string.
(ctf_str_add): Adjust accordingly.
(ctf_str_add_ref): Likewise. Move up in the file.
(ctf_str_add_external): New: update the csa_offset.
(ctf_str_count_strtab): Only account for strings with no csa_offset
in the internal strtab length.
(ctf_str_write_strtab): If the csa_offset is set, update the
string's refs without writing the string out, and update the
ctf_syn_ext_strtab. Make OOM handling less ugly.
* ctf-create.c (struct ctf_sort_var_arg_cb): New.
(ctf_update): Handle failure to populate the strtab. Pass in the
new ctf_sort_var arg. Adjust for ctf_syn_ext_strtab addition.
Call ctf_simple_open_internal, not ctf_simple_open.
(ctf_sort_var): Call ctf_strraw_explicit rather than looking up
strings by hand.
* ctf-hash.c (ctf_hash_insert_type): Likewise (but using
ctf_strraw). Adjust to diagnose ECTF_STRTAB nonetheless.
* ctf-open.c (init_types): No longer filter out ECTF_STRTAB.
(ctf_file_close): Destroy the ctf_syn_ext_strtab.
(ctf_simple_open): Rename to, and reimplement as a wrapper around...
(ctf_simple_open_internal): ... this new function, which calls
ctf_bufopen_internal.
(ctf_bufopen): Rename to, and reimplement as a wrapper around...
(ctf_bufopen_internal): ... this new function, which sets
ctf_syn_ext_strtab.
The existing function ctf_type_iter lets you iterate over root-visible
types (types you can look up by name). There is no way to iterate over
non-root-visible types, which is troublesome because both the linker
and dumper want to do that.
So add a new function that can do it: the callback it takes accepts
an extra parameter which indicates whether the type is root-visible
or not.
include/
* ctf-api.h (ctf_type_all_f): New.
(ctf_type_iter_all): New.
libctf/
* ctf_types.c (ctf_type_iter_all): New.
No code handles these yet, but our latest GCC patches are generating
them, so we have to be ready for them or erroneously conclude that we
have file corruption.
(This simultaneously fixes a longstanding bug, concealed because nothing
was generating anything in the object or function info sections, where
the end of the section was being tested against the wrong thing: it
would have walked over the entire contents of the variable section and
treated them as part of the function info section. This had to change
now anyway because the new sections have landed in between.)
include/
* ctf.h: Add object index and function index sections. Describe
them. Improve the description of the variable section and clarify
the constraints on backward-pointing type nodes.
(ctf_header): Add cth_objtidxoff, cth_funcidxoff.
libctf/
* ctf-open.c (init_symtab): Check for overflow against the right
section.
(upgrade_header): Set cth_objtidxoff, cth_funcidxoff to zero-length.
(upgrade_types_v1): Note that these sections are not checked.
(flip_header): Endian-swap the header fields.
(flip_ctf): Endian-swap the sections.
(flip_objts): Update comment.
(ctf_bufopen): Check header offsets and alignment for validity.
The code in ctf_bfdopen_ctfsect (which is the ultimate place where you
end up if you use ctf_open to open a CTF file and pull in the ELF string
and symbol tables) was written before it was possible to actually test
it, since the linker was not written. Now it is, it turns out that the
previous code was completely nonfunctional: it assumed that you could
load the symbol table via bfd_section_from_elf_index (...,elf_onesymtab())
and the string table via bfd_section_from_elf_index on the sh_link.
Unfortunately BFD loads neither of these sections in the conventional
fashion it uses for most others: the symbol table is immediately
converted into internal form (which is useless for our purposes, since
we also have to work in the absence of BFD for readelf, etc) and the
string table is loaded specially via bfd_elf_get_str_section which is
private to bfd/elf.c.
So make this function public, export it in elf-bfd.h, and use it from
libctf, which does something similar to what bfd_elf_sym_name and
bfd_elf_string_from_elf_section do. Similarly, load the symbol table
manually using bfd_elf_get_elf_syms and throw away the internal form
it generates for us (we never use it).
BFD allocates the strtab for us via bfd_alloc, so we can leave BFD to
deallocate it: we allocate the symbol table ourselves before calling
bfd_elf_get_elf_syms, so we still have to free it.
Also change the rules around what you are allowed to provide: It is
useful to provide a string section but no symbol table, because CTF
sections can legitimately have no function info or data object sections
while relying on the ELF strtab for some of their strings. So allow
that combination.
v4: adjust to upstream changes. ctf_bfdopen_ctfsect's first parameter
is potentially unused again (if BFD is not in use for this link
due to not supporting an ELF target).
v5: fix tabdamage.
bfd/
* elf-bfd.h (bfd_elf_get_str_section): Add.
* elf.c (bfd_elf_get_str_section): No longer static.
libctf/
* ctf-open-bfd.c: Add <assert.h>.
(ctf_bfdopen_ctfsect): Open string and symbol tables using
techniques borrowed from bfd_elf_sym_name.
(ctf_new_archive_internal): Improve comment.
* ctf-archive.c (ctf_arc_close): Do not free the ctfi_strsect.
* ctf-open.c (ctf_bufopen): Allow opening with a string section but
no symbol section, but not vice versa.
The CTF header has before now been thrown away too soon to be dumped
using the ctf_dump() machinery used by objdump and readelf: instead, a
kludge involving debugging-priority dumps of the header offsets on every
open was used.
Replace this with proper first-class dumping machinery just like
everything else in the CTF file, and have objdump and readelf use it.
(The dumper already had an enum value in ctf_sect_names_t for this
purpose, waiting to be used.)
v5: fix tabdamage.
libctf/
* ctf-impl.h (ctf_file_t): New field ctf_openflags.
* ctf-open.c (ctf_bufopen): Set it. No longer dump header offsets.
* ctf-dump.c (dump_header): New function, dump the CTF header.
(ctf_dump): Call it.
(ctf_dump_header_strfield): New function.
(ctf_dump_header_sectfield): Likewise.
binutils/
* objdump.c (dump_ctf_archive_member): Dump the CTF header.
* readelf.c (dump_section_as_ctf): Likewise.
libctf supports dynamic upgrading of the type table as file format
versions change, but before now has not supported changes to the CTF
header. Doing this is complicated by the baroque storage method used:
the CTF header is kept prepended to the rest of the CTF data, just as
when read from the file, and written out from there, and is
endian-flipped in place.
This makes accessing it needlessly hard and makes it almost impossible
to make the header larger if we add fields. The general storage
machinery around the malloced ctf pointer (the 'ctf_base') is also
overcomplicated: the pointer is sometimes malloced locally and sometimes
assigned from a parameter, so freeing it requires checking to see if
that parameter was used, needlessly coupling ctf_bufopen and
ctf_file_close together.
So split the header out into a new ctf_file_t.ctf_header, which is
written out explicitly: squeeze it out of the CTF buffer whenever we
reallocate it, and use ctf_file_t.ctf_buf to skip past the header when
we do not need to reallocate (when no upgrading or endian-flipping is
required). We now track whether the CTF base can be freed explicitly
via a new ctf_dynbase pointer which is non-NULL only when freeing is
possible.
With all this done, we can upgrade the header on the fly and add new
fields as desired, via a new upgrade_header function in ctf-open.
As with other forms of upgrading, libctf upgrades older headers
automatically to the latest supported version at open time.
For a first use of this field, we add a new string field cth_cuname, and
a corresponding setter/getter pair ctf_cuname_set and ctf_cuname: this
is used by debuggers to determine whether a CTF section's types relate
to a single compilation unit, or to all compilation units in the
program. (Types with ambiguous definitions in different CUs have only
one of these types placed in the top-level shared .ctf container: the
rest are placed in much smaller per-CU containers, which have the shared
container as their parent. Since CTF must be useful in the absence of
DWARF, we store the names of the relevant CUs ourselves, so the debugger
can look them up.)
v5: fix tabdamage.
include/
* ctf-api.h (ctf_cuname): New function.
(ctf_cuname_set): Likewise.
* ctf.h: Improve comment around upgrading, no longer
implying that v2 is the target of upgrades (it is v3 now).
(ctf_header_v2_t): New, old-format header for backward
compatibility.
(ctf_header_t): Add cth_cuname: this is the first of several
header changes in format v3.
libctf/
* ctf-impl.h (ctf_file_t): New fields ctf_header, ctf_dynbase,
ctf_cuname, ctf_dyncuname: ctf_base and ctf_buf are no longer const.
* ctf-open.c (ctf_set_base): Preserve the gap between ctf_buf and
ctf_base: do not assume that it is always sizeof (ctf_header_t).
Print out ctf_cuname: only print out ctf_parname if set.
(ctf_free_base): Removed, ctf_base is no longer freed: free
ctf_dynbase instead.
(ctf_set_version): Fix spacing.
(upgrade_header): New, in-place header upgrading.
(upgrade_types): Rename to...
(upgrade_types_v1): ... this. Free ctf_dynbase, not ctf_base. No
longer track old and new headers separately. No longer allow for
header sizes explicitly: squeeze the headers out on upgrade (they
are preserved in fp->ctf_header). Set ctf_dynbase, ctf_base and
ctf_buf explicitly. Use ctf_free, not ctf_free_base.
(upgrade_types): New, also handle ctf_parmax updating.
(flip_header): Flip ctf_cuname.
(flip_types): Flip BUF explicitly rather than deriving BUF from
BASE.
(ctf_bufopen): Store the header in fp->ctf_header. Correct minimum
required alignment of objtoff and funcoff. No longer store it in
the ctf_buf unless that buf is derived unmodified from the input.
Set ctf_dynbase where ctf_base is dynamically allocated. Drop locals
that duplicate fields in ctf_file: move allocation of ctf_file
further up instead. Call upgrade_header as needed. Move
version-specific ctf_parmax initialization into upgrade_types. More
concise error handling.
(ctf_file_close): No longer test for null pointers before freeing.
Free ctf_dyncuname, ctf_dynbase, and ctf_header. Do not call
ctf_free_base.
(ctf_cuname): New.
(ctf_cuname_set): New.
* ctf-create.c (ctf_update): Populate ctf_cuname.
(ctf_gzwrite): Write out the header explicitly. Remove obsolescent
comment.
(ctf_write): Likewise.
(ctf_compress_write): Get the header from ctf_header, not ctf_base.
Fix the compression length: fp->ctf_size never counted the CTF
header. Simplify the compress call accordingly.
With a glibc before 2.9 (such as 2.8), there's <endian.h> but no
htole64 or le64toh, so you get, compiling binutils for any target:
libtool: link: gcc -W -Wall -Wstrict-prototypes -Wmissing-prototypes \
-Wshadow -Werror -I/x/binutils/../zlib -g -O2 -o objdump \
objdump.o dwarf.o prdbg.o rddbg.o debug.o stabs.o rdcoff.o \
bucomm.o version.o filemode.o elfcomm.o ../opcodes/.libs/libopcodes.a \
../libctf/libctf.a ../bfd/.libs/libbfd.a -L/x/obj/b/zlib -lz ../libiberty/libiberty.a -ldl
../libctf/libctf.a(ctf-archive.o): In function `ctf_archive_raw_iter_internal':
/x/src/libctf/ctf-archive.c:543: undefined reference to `le64toh'
/x/src/libctf/ctf-archive.c:550: undefined reference to `le64toh'
/x/src/libctf/ctf-archive.c:551: undefined reference to `le64toh'
/x/src/libctf/ctf-archive.c:551: undefined reference to `le64toh'
/x/src/libctf/ctf-archive.c:554: undefined reference to `le64toh'
../libctf/libctf.a(ctf-archive.o):/x/src/libctf/ctf-archive.c:545: more undefined references to `le64toh' follow
(etc)
Also, I see no bswap_identity_64 *anywhere* except in libctf/swap.h
(including current glibc) and I don't think calling an "identity"-
function is better than just plain "#define foo(x) (x)" anyway.
(Where does the idea of a bytestap.h bswap_identity_64 come from?)
Speaking of that, I should mention that I instrumented the condition
to observe that the WORDS_BIGENDIAN case passes too for a presumed
big-endian target and glibc-2.8: there is a bswap_64 present for that
version. Curiously, no test-case regressed with that instrumentation.
For the record, constructing binary blobs using text source to run
tests on, can be done by linking to --oformat binary (with most ELF
targets), but I guess that's seen as unnecessary roundabout perhaps
checking in binary files in the test-suite would be ok these days.
[...]
[nca: trimmed commit log slightly, updated changelog]
v5: fix tabdamage.
libctf/
* ctf-endian.h: Don't assume htole64 and le64toh are always
present if HAVE_ENDIAN_H; also check if htole64 is defined.
[!WORDS_BIGENDIAN] (htole64, le64toh): Define as identity,
not bswap_identity_64.
The first two of these allow you to get function type info and args out
of the types section give a type ID: astonishingly, this was missing
from libctf before now: so even though types of kind CTF_K_FUNCTION were
supported, you couldn't find out anything about them. (The existing
ctf_func_info and ctf_func_args only allow you to get info about
functions in the function section, i.e. given symbol table indexes, not
type IDs.)
The second of these allows you to get the raw undecorated name out of
the CTF section (strdupped for safety) without traversing subtypes to
build a full C identifier out of it. It's useful for things that are
already tracking the type kind etc and just need an unadorned name.
include/
* ctf-api.h (ECTF_NOTFUNC): Fix description.
(ctf_func_type_info): New.
(ctf_func_type_args): Likewise.
libctf/
* ctf-types.c (ctf_type_aname_raw): New.
(ctf_func_type_info): Likewise.
(ctf_func_type_args): Likewise.
* ctf-error.c (_ctf_errlist): Fix description.
The first ctf_snapshot called after CTF file creation yields a snapshot
handle that always yields a spurious ECTF_OVERROLLBACK error ("Attempt
to roll back past a ctf_update") on ctf_rollback(), even if ctf_update
has never been called.
The fix is to start with a ctf_snapshot value higher than the zero value
that ctf_snapshot_lu ("last update CTF snapshot value") is initialized
to.
libctf/
* ctf-create.c (ctf_create): Fix off-by-one error.
ctf.h states:
> [...] the CTF string table does not contain any duplicated strings.
Unfortunately this is entirely untrue: libctf has before now made no
attempt whatsoever to deduplicate the string table. It computes the
string table's length on the fly as it adds new strings to the dynamic
CTF file, and ctf_update() just writes each string to the table and
notes the current write position as it traverses the dynamic CTF file's
data structures and builds the final CTF buffer. There is no global
view of the strings and no deduplication.
Fix this by erasing the ctf_dtvstrlen dead-reckoning length, and adding
a new dynhash table ctf_str_atoms that maps unique strings to a list
of references to those strings: a reference is a simple uint32_t * to
some value somewhere in the under-construction CTF buffer that needs
updating to note the string offset when the strtab is laid out.
Adding a string is now a simple matter of calling ctf_str_add_ref(),
which adds a new atom to the atoms table, if one doesn't already exist,
and adding the location of the reference to this atom to the refs list
attached to the atom: this works reliably as long as one takes care to
only call ctf_str_add_ref() once the final location of the offset is
known (so you can't call it on a temporary structure and then memcpy()
that structure into place in the CTF buffer, because the ref will still
point to the old location: ctf_update() changes accordingly).
Generating the CTF string table is a matter of calling
ctf_str_write_strtab(), which counts the length and number of elements
in the atoms table using the ctf_dynhash_iter() function we just added,
populating an array of pointers into the atoms table and sorting it into
order (to help compressors), then traversing this table and emitting it,
updating the refs to each atom as we go. The only complexity here is
arranging to keep the null string at offset zero, since a lot of code in
libctf depends on being able to leave strtab references at 0 to indicate
'no name'. Once the table is constructed and the refs updated, we know
how long it is, so we can realloc() the partial CTF buffer we allocated
earlier and can copy the table on to the end of it (and purge the refs
because they're not needed any more and have been invalidated by the
realloc() call in any case).
The net effect of all this is a reduction in uncompressed strtab sizes
of about 30% (perhaps a quarter to a half of all strings across the
Linux kernel are eliminated as duplicates). Of course, duplicated
strings are highly redundant, so the space saving after compression is
only about 20%: when the other non-strtab sections are factored in, CTF
sizes shrink by about 10%.
No change in externally-visible API or file format (other than the
reduction in pointless redundancy).
libctf/
* ctf-impl.h: (struct ctf_strs_writable): New, non-const version of
struct ctf_strs.
(struct ctf_dtdef): Note that dtd_data.ctt_name is unpopulated.
(struct ctf_str_atom): New, disambiguated single string.
(struct ctf_str_atom_ref): New, points to some other location that
references this string's offset.
(struct ctf_file): New members ctf_str_atoms and ctf_str_num_refs.
Remove member ctf_dtvstrlen: we no longer track the total strlen
as we add strings.
(ctf_str_create_atoms): Declare new function in ctf-string.c.
(ctf_str_free_atoms): Likewise.
(ctf_str_add): Likewise.
(ctf_str_add_ref): Likewise.
(ctf_str_purge_refs): Likewise.
(ctf_str_write_strtab): Likewise.
(ctf_realloc): Declare new function in ctf-util.c.
* ctf-open.c (ctf_bufopen): Create the atoms table.
(ctf_file_close): Destroy it.
* ctf-create.c (ctf_update): Copy-and-free it on update. No longer
special-case the position of the parname string. Construct the
strtab by calling ctf_str_add_ref and ctf_str_write_strtab after the
rest of each buffer element is constructed, not via open-coding:
realloc the CTF buffer and append the strtab to it. No longer
maintain ctf_dtvstrlen. Sort the variable entry table later, after
strtab construction.
(ctf_copy_membnames): Remove: integrated into ctf_copy_{s,l,e}members.
(ctf_copy_smembers): Drop the string offset: call ctf_str_add_ref
after buffer element construction instead.
(ctf_copy_lmembers): Likewise.
(ctf_copy_emembers): Likewise.
(ctf_create): No longer maintain the ctf_dtvstrlen.
(ctf_dtd_delete): Likewise.
(ctf_dvd_delete): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_variable): Likewise.
(membadd): Likewise.
* ctf-util.c (ctf_realloc): New, wrapper around realloc that aborts
if there are active ctf_str_num_refs.
(ctf_strraw): Move to ctf-string.c.
(ctf_strptr): Likewise.
* ctf-string.c: New file, strtab manipulation.
* Makefile.am (libctf_a_SOURCES): Add it.
* Makefile.in: Regenerate.
There are two, ctf_dynhash_iter and ctf_dynhash_iter_remove: the latter
lets you return a nonzero value to remove the element being iterated
over.
Used in the next commit.
libctf/
* ctf-impl.h (ctf_hash_iter_f): New.
(ctf_dynhash_iter): New declaration.
(ctf_dynhash_iter_remove): New declaration.
* ctf-hash.c (ctf_dynhash_iter): Define.
(ctf_dynhash_iter_remove): Likewise.
(ctf_hashtab_traverse): New.
(ctf_hashtab_traverse_remove): Likewise.
(struct ctf_traverse_cb_arg): Likewise.
(struct ctf_traverse_remove_cb_arg): Likewise.
We must call htab_remove_elt with an element (in this case, a mocked-up
one with only the key populated, since no reasonable hash function will
need the other fields), not with the key alone.
libctf/
* ctf-hash.c (ctf_dynhash_remove): Call with a mocked-up element.
We were sometimes printing hex values without prefixing them with '0x',
leading to confusion about what base the numbers were actually in.
libctf/
* ctf-dump.c (ctf_dump_format_type): Prefix hex strings with 0x.
(ctf_dump_funcs): Likewise.
ctf_open (or, rather, ctf_fdopen, which underlies it) has several
endianness problems, even though it was written after the
endian-swapping code was implemented, so should have been endian-aware.
Even though the comment right above the relevant check says that it wil
check for CTF magic in any endianness, it only checks in the native
endianness, so opening raw LE CTF files on BE, or vice-versa, will fail.
It also checks the CTF version by hand, without ever endianness-swapping
the header, so that too will fail, and is entirely redundant because
ctf_simple_open does the job properly in any case. We have a similar
problem in the next if block, which checks for raw CTF archives: we are
checking in the native endianness while we should be doing a le64toh()
on it to check in little-endian form only: so opening CTF archives
created on the local machine will fail if the local machine is
big-endian.
Adding insult to injury, if ctf_simple_open then fails, we go on and try
to turn it into a single-element CTF archive regardless, throwing the
error away. Since this involves dereferencing null pointers it is not
likely to work very well.
libctf/
* ctf-open-bfd.c: Add swap.h and ctf-endian.h.
(ctf_fdopen): Check for endian-swapped raw CTF magic, and
little-endian CTF archive magic. Do not check the CTF version:
ctf_simple_open does that in endian-safe ways. Do not dereference
null pointers on open failure.
Testing of the first code to generate CTF_K_SLICEs on big-endian
revealed a bunch of new problems in this area. Most importantly, the
trick we did earlier to avoid wasting two bytes on padding in the
ctf_slice_t is best avoided: because it leads to the whole file after
that point no longer being naturally aligned, all multibyte accesses
from then on must use memmove() to avoid unaligned access on platforms
where that is fatal. In future, this is planned, but for now we are
still doing direct access in many places, so we must revert to making
ctf_slice_t properly aligned for storage in an array.
Rather than wasting bytes on padding, we boost the size of cts_offset
and cts_bits. This is still a waste of space (we cannot have offsets or
bits in bitfields > 256) but it cannot be avoided for now, and slices
are not so common that this will be a serious problem.
A possibly-worse endianness problem fixed at the same time involves
a codepath used only for foreign-endian, uncompressed CTF files, where
we were not copying the actual CTF data into the buffer, leading to
libctf reading only zeroes (or, possibly, uninitialized garbage).
Finally, when we read in a CTF file, we copy the header and work from
the copy. We were flipping the endianness of the header copy, and of
the body of the file buffer, but not of the header in the file buffer
itself: so if we write the file back out again we end up with an
unreadable frankenfile with header and body of different endiannesses.
Fix by flipping both copies of the header.
include/
* ctf.h (ctf_slice_t): Make cts_offset and cts_bits unsigned
short, so following structures are properly aligned.
libctf/
* ctf-open.c (get_vbytes_common): Return the new slice size.
(ctf_bufopen): Flip the endianness of the CTF-section header copy.
Remember to copy in the CTF data when opening an uncompressed
foreign-endian CTF file. Prune useless variable manipulation.
If we see a CTF type with a kind we do not recognize in its ctt_info
during opening, we cannot skip it and continue opening the file: if the
type kind is unknown, we do not know how long its vlen is, and we cannot
have skipped past it: so if we continue reading we will almost certainly
read in part of the vlen as if it were a new ctf_type_t.
Avoid this trouble by considering unknown type kinds to be a reason to
return ECTF_CORRUPT, just like everything else that reads in type kinds
does.
libctf/
* ctf-open.c (ctf_types): Fail when unidentified type kinds are
seen.
This is an essential first piece of info needed to debug both libctf
writing and reading problems, and we weren't recording it anywhere!
(This is a short-term fix: fairly soon, we will record all of this in a
form that outlives ctf_bufopen, and then ctf_dump() will be able to dump
it like it can everything else.)
libctf/
* ctf-open.c (ctf_bufopen): Dump header offsets into the debugging
output.
This allocator has the ostensible benefit that it lets us mprotect() the
memory used for CTF storage: but in exchange for this it adds
considerable complexity, since we have to track allocation sizes
ourselves for use at freeing time, note whether the data we are storing
was ctf_data_alloc()ed or not so we know if we can safely mprotect()
it... and while the mprotect()ing has found few bugs, it *has* been the
cause of more than one due to errors in all this tracking leading to us
mprotect()ing bits of the heap and stuff like that.
We are about to start composing CTF buffers from pieces so that we can
do usage-based optimizations on the strtab. This means we need
realloc(), which needs nonportable mremap() and *more* tracking of the
*original* allocation size, and the complexity and bureaucracy of all of
this is just too high for its negligible benefits.
Drop the whole thing and just use malloc() like everyone else. It knows
better than we do when it is safe to use mmap() under the covers,
anyway.
While we're at it, don't leak the entire buffer if ctf_compress_write()
fails to compress it.
libctf/
* ctf-subr.c (_PAGESIZE): Remove.
(ctf_data_alloc): Likewise.
(ctf_data_free): Likewise.
(ctf_data_protect): Likewise.
* ctf-impl.h: Remove declarations.
* ctf-create.c (ctf_update): No longer call ctf_data_protect: use
ctf_free, not ctf_data_free.
(ctf_compress_write): Use ctf_data_alloc, not ctf_alloc. Free
the buffer again on compression error.
* ctf-open.c (ctf_set_base): No longer track the size: call
ctf_free, not ctf_data_free.
(upgrade_types): Likewise. Call ctf_alloc, not ctf_data_alloc.
(ctf_bufopen): Likewise. No longer call ctf_data_protect.
We were missing several cases where dynhash insertion might fail, likely
due to OOM but possibly for other reasons. Pass the errors on.
libctf/
* ctf-create.c (ctf_dtd_insert): Pass on error returns from
ctf_dynhash_insert.
(ctf_dvd_insert): Likewise.
(ctf_add_generic): Likewise.
(ctf_add_variable): Likewise.
* ctf-impl.h: Adjust declarations.
Not all platforms have it. Use libiberty xstrndup() instead.
(The include of libiberty.h happens in an unusual place due to the
requirements of synchronization of most source files between this
project and another that does not use libiberty. It serves to pull
libiberty.h in for all source files in libctf/, which does the trick.)
Tested on x86_64-pc-linux-gnu, x86_64-unknown-freebsd12.0,
sparc-sun-solaris2.11, i686-pc-cygwin, i686-w64-mingw32.
libctf/
* ctf-decls.h: Include <libiberty.h>.
* ctf-lookup.c (ctf_lookup_by_name): Call xstrndup(), not strndup().
Unsigned long will always be adequate (the only cases involving an
ssize_t are cases in which no error can be generated, or in which
negative output would require a seriously corrupted file: the latter has
been rewritten on a branch in any case).
Tested on x86_64-pc-linux-gnu, x86_64-unknown-freebsd12.0,
sparc-sun-solaris2.11, i686-pc-cygwin, i686-w64-mingw32.
libctf/
* ctf-dump.c (ctf_dump_format_type): Cast size_t's used in printf()s.
(ctf_dump_objts): Likewise.
(ctf_dump_funcs): Likewise.
(ctf_dump_member): Likewise.
(ctf_dump_str): Likewise.
Tested on x86_64-pc-linux-gnu, x86_64-unknown-freebsd12.0,
sparc-sun-solaris2.11, i686-pc-cygwin, i686-w64-mingw32.
libctf/
* ctf-archive.c (arc_mmap_header): Mark fd as potentially unused.
* ctf-subr.c (ctf_data_protect): Mark both args as potentially unused.
Too many platforms don't support it, and we can always safely use %lu or
%li anyway, because the only uses are in debugging output.
libctf/
* ctf-archive.c (ctf_arc_write): Eschew %zi format specifier.
(ctf_arc_open_by_offset): Likewise.
* ctf-create.c (ctf_add_type): Likewise.
On x86-64 Fedora 29, I tried to build a mingw-hosted gdb that targets
ppc-linux. You can do this with:
../binutils-gdb/configure --host=i686-w64-mingw32 --target=ppc-linux \
--disable-{binutils,gas,gold,gprof,ld}
The build failed with these errors in libctf:
In file included from ../../binutils-gdb/libctf/ctf-create.c:20:
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_encoded':
../../binutils-gdb/libctf/ctf-create.c:803:59: error: 'NBBY' undeclared (first use in this function)
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c:803:59: note: each undeclared identifier is reported only once for each function it appears in
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_slice':
../../binutils-gdb/libctf/ctf-create.c:862:59: error: 'NBBY' undeclared (first use in this function)
dtd->dtd_data.ctt_size = clp2 (P2ROUNDUP (ep->cte_bits, NBBY) / NBBY);
^~~~
../../binutils-gdb/libctf/ctf-impl.h:254:42: note: in definition of macro 'P2ROUNDUP'
#define P2ROUNDUP(x, align) (-(-(x) & -(align)))
^~~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_member_offset':
../../binutils-gdb/libctf/ctf-create.c:1341:21: error: 'NBBY' undeclared (first use in this function)
off += lsize * NBBY;
^~~~
../../binutils-gdb/libctf/ctf-create.c: In function 'ctf_add_type':
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: unknown conversion type character 'z' in format [-Wformat=]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../binutils-gdb/libctf/ctf-create.c:1823:35: note: format string is defined here
"union size differs, old %zi, new %zi\n",
^
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: unknown conversion type character 'z' in format [-Wformat=]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
../../binutils-gdb/libctf/ctf-create.c:1823:44: note: format string is defined here
"union size differs, old %zi, new %zi\n",
^
../../binutils-gdb/libctf/ctf-create.c:1822:16: warning: too many arguments for format [-Wformat-extra-args]
ctf_dprintf ("Conflict for type %s against ID %lx: "
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This patch fixes the actual errors in here. I did not try to fix the
printf warnings, though I think someone ought to.
Ok?
libctf/ChangeLog
2019-06-04 Tom Tromey <tromey@adacore.com>
* ctf-create.c (ctf_add_encoded, ctf_add_slice)
(ctf_add_member_offset): Use CHAR_BIT, not NBBY.
(Not tested on any such platforms, since I don't have access to any at
the moment. Testing encouraged.)
libctf/
* configure.ac: Check for O_CLOEXEC.
* ctf-decls.h (O_CLOEXEC): Define (to 0), if need be.
* config.h.in: Regenerate.
We cannot just look for any declaration of qsort_r, because some
operating systems have a qsort_r that has a different prototype
but which still has a pair of pointers in the right places (the last two
args are interchanged): so use AC_LINK_IFELSE to check for both
known variants of qsort_r(), and swap their args into a consistent order
in a suitable inline function. (The code for this is taken almost
unchanged from gnulib.)
(Now we are not using AC_LIBOBJ any more, we can use a better name for
the qsort_r replacement as well.)
libctf/
* qsort_r.c: Rename to...
* ctf-qsort_r.c: ... this.
(_quicksort): Define to ctf_qsort_r.
* ctf-decls.h (qsort_r): Remove.
(ctf_qsort_r): Add.
(struct ctf_qsort_arg): New, transport the real ARG and COMPAR.
(ctf_qsort_compar_thunk): Rearrange the arguments to COMPAR.
* Makefile.am (libctf_a_LIBADD): Remove.
(libctf_a_SOURCES): New, add ctf-qsort_r.c.
* ctf-archive.c (ctf_arc_write): Call ctf_qsort_r, not qsort_r.
* ctf-create.c (ctf_update): Likewise.
* configure.ac: Check for BSD versus GNU qsort_r signature.
* Makefile.in: Regenerate.
* config.h.in: Likewise.
* configure: Likewise.
- Use of nonportable <endian.h>
- Use of qsort_r
- Use of zlib without appropriate magic to pull in the binutils zlib
- Use of off64_t without checking (fixed by dropping the unused fields
that need off64_t entirely)
- signedness problems due to long being too short a type on 32-bit
platforms: ctf_id_t is now 'unsigned long', and CTF_ERR must be
used only for functions that return ctf_id_t
- One lingering use of bzero() and of <sys/errno.h>
All fixed, using code from gnulib where possible.
Relatedly, set cts_size in a couple of places it was missed
(string table and symbol table loading upon ctf_bfdopen()).
binutils/
* objdump.c (make_ctfsect): Drop cts_type, cts_flags, and
cts_offset.
* readelf.c (shdr_to_ctf_sect): Likewise.
include/
* ctf-api.h (ctf_sect_t): Drop cts_type, cts_flags, and cts_offset.
(ctf_id_t): This is now an unsigned type.
(CTF_ERR): Cast it to ctf_id_t. Note that it should only be used
for ctf_id_t-returning functions.
libctf/
* Makefile.am (ZLIB): New.
(ZLIBINC): Likewise.
(AM_CFLAGS): Use them.
(libctf_a_LIBADD): New, for LIBOBJS.
* configure.ac: Check for zlib, endian.h, and qsort_r.
* ctf-endian.h: New, providing htole64 and le64toh.
* swap.h: Code style fixes.
(bswap_identity_64): New.
* qsort_r.c: New, from gnulib (with one added #include).
* ctf-decls.h: New, providing a conditional qsort_r declaration,
and unconditional definitions of MIN and MAX.
* ctf-impl.h: Use it. Do not use <sys/errno.h>.
(ctf_set_errno): Now returns unsigned long.
* ctf-util.c (ctf_set_errno): Adjust here too.
* ctf-archive.c: Use ctf-endian.h.
(ctf_arc_open_by_offset): Use memset, not bzero. Drop cts_type,
cts_flags and cts_offset.
(ctf_arc_write): Drop debugging dependent on the size of off_t.
* ctf-create.c: Provide a definition of roundup if not defined.
(ctf_create): Drop cts_type, cts_flags and cts_offset.
(ctf_add_reftype): Do not check if type IDs are below zero.
(ctf_add_slice): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_member_offset): Cast error-returning ssize_t's to size_t
when known error-free. Drop CTF_ERR usage for functions returning
int.
(ctf_add_member_encoded): Drop CTF_ERR usage for functions returning
int.
(ctf_add_variable): Likewise.
(enumcmp): Likewise.
(enumadd): Likewise.
(membcmp): Likewise.
(ctf_add_type): Likewise. Cast error-returning ssize_t's to size_t
when known error-free.
* ctf-dump.c (ctf_is_slice): Drop CTF_ERR usage for functions
returning int: use CTF_ERR for functions returning ctf_type_id.
(ctf_dump_label): Likewise.
(ctf_dump_objts): Likewise.
* ctf-labels.c (ctf_label_topmost): Likewise.
(ctf_label_iter): Likewise.
(ctf_label_info): Likewise.
* ctf-lookup.c (ctf_func_args): Likewise.
* ctf-open.c (upgrade_types): Cast to size_t where appropriate.
(ctf_bufopen): Likewise. Use zlib types as needed.
* ctf-types.c (ctf_member_iter): Drop CTF_ERR usage for functions
returning int.
(ctf_enum_iter): Likewise.
(ctf_type_size): Likewise.
(ctf_type_align): Likewise. Cast to size_t where appropriate.
(ctf_type_kind_unsliced): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_encoding): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_value): Likewise.
(ctf_type_rvisit): Likewise.
* ctf-open-bfd.c (ctf_bfdopen): Drop cts_type, cts_flags and
cts_offset.
(ctf_simple_open): Likewise.
(ctf_bfdopen_ctfsect): Likewise. Set cts_size properly.
* Makefile.in: Regenerate.
* aclocal.m4: Likewise.
* config.h: Likewise.
* configure: Likewise.
All machinery works as on ELF, except for automatic loading of ELF
string and symbol tables in the BFD-style open machinery.
* Makefile.def (dependencies): configure-libctf depends on all-bfd
and all its deps.
* Makefile.in: Regenerated.
libctf/
* configure.in: Check for bfd_section_from_elf_index.
* configure: Regenerate.
* config.h.in [HAVE_BFD_ELF]: Likewise.
* libctf/ctf_open_bfd (ctf_bfdopen_ctfsect): Use it.
abfd is potentially unused now.
This ties libctf into the build system, and makes binutils depend on it
(used by the next commits).
* Makefile.def (host_modules): Add libctf.
* Makefile.def (dependencies): Likewise.
libctf depends on zlib, libiberty, and bfd.
* Makefile.in: Regenerated.
* configure.ac (host_libs): Add libctf.
* configure: Regenerated.
libctf/
* Makefile.am: New.
* Makefile.in: Regenerated.
* config.h.in: Likewise.
* aclocal.m4: Likewise.
* configure: Likewise.
This introduces ctf_dump(), an iterator which returns a series of
strings, each representing a debugging dump of one item from a given
section in the CTF file. The items may be multiline: a callback is
provided to allow the caller to decorate each line as they desire before
the line is returned.
libctf/
* ctf-dump.c: New.
include/
* ctf-api.h (ctf_dump_decorate_f): New.
(ctf_dump_state_t): new.
(ctf_dump): New.
This facility allows you to associate regions of type IDs with *labels*,
a labelled tiling of the type ID space. You can use these to define
CTF containers with distinct parents for distinct ranges of the ID
space, or to assist with parallelization of CTF processing, or for any
other purpose you can think of.
Notably absent from here (though declared in the API header) is any way
to define new labels: this will probably be introduced soon, as part of
the linker deduplication work. (One existed in the past, but was deeply
tied to the Solaris CTF file generator and had to be torn out.)
libctf/
* ctf-labels.c: New.
include/
* ctf-api.h (ctf_label_f): New.
(ctf_label_set): New.
(ctf_label_get): New.
(ctf_label_topmost): New.
(ctf_label_info): New.
(ctf_label_iter): New.
This old Solaris standard allows callers to specify that they are
expecting one particular API and/or CTF file format from the library.
libctf/
* ctf-impl.h (_libctf_version): New declaration.
* ctf-subr.c (_libctf_version): Define it.
(ctf_version): New.
include/
* ctf-api.h (ctf_version): New.
ctf_add_type() allows you to copy types, and all the types they depend
on, from one container to another (writable) container. This lets a
program maintaining multiple distinct containers (not in a parent-child
relationship) introduce types that depend on types in one container in
another writable one, by copying the necessary types.
libctf/
* ctf-create.c (enumcmp): New.
(enumadd): Likewise.
(membcmp): Likewise.
(membadd): Likewise.
(ctf_add_type): Likewise.
These functions allow you to look up types given a name in a simple
subset of C declarator syntax (no function pointers), to look up the
types of variables given a name, and to look up the types of data
objects and the type signatures of functions given symbol table offsets.
(Despite its name, one function in this commit, ctf_lookup_symbol_name(),
is for the internal use of libctf only, and does not appear in any
public header files.)
libctf/
* ctf-lookup.c (isqualifier): New.
(ctf_lookup_by_name): Likewise.
(struct ctf_lookup_var_key): Likewise.
(ctf_lookup_var): Likewise.
(ctf_lookup_variable): Likewise.
(ctf_lookup_symbol_name): Likewise.
(ctf_lookup_by_symbol): Likewise.
(ctf_func_info): Likewise.
(ctf_func_args): Likewise.
include/
* ctf-api.h (ctf_func_info): New.
(ctf_func_args): Likewise.
(ctf_lookup_by_symbol): Likewise.
(ctf_lookup_by_symbol): Likewise.
(ctf_lookup_variable): Likewise.
Finally we get to the functions used to actually look up and enumerate
properties of types in a container (names, sizes, members, what type a
pointer or cv-qual references, determination of whether two types are
assignment-compatible, etc).
With a very few exceptions these do not work for types newly added via
ctf_add_*(): they only work on types in read-only containers, or types
added before the most recent call to ctf_update().
This also adds support for lookup of "variables" (string -> type ID
mappings) and for generation of C type names corresponding to a type ID.
libctf/
* ctf-decl.c: New file.
* ctf-types.c: Likewise.
* ctf-impl.h: New declarations.
include/
* ctf-api.h (ctf_visit_f): New definition.
(ctf_member_f): Likewise.
(ctf_enum_f): Likewise.
(ctf_variable_f): Likewise.
(ctf_type_f): Likewise.
(ctf_type_isparent): Likewise.
(ctf_type_ischild): Likewise.
(ctf_type_resolve): Likewise.
(ctf_type_aname): Likewise.
(ctf_type_lname): Likewise.
(ctf_type_name): Likewise.
(ctf_type_sizee): Likewise.
(ctf_type_align): Likewise.
(ctf_type_kind): Likewise.
(ctf_type_reference): Likewise.
(ctf_type_pointer): Likewise.
(ctf_type_encoding): Likewise.
(ctf_type_visit): Likewise.
(ctf_type_cmp): Likewise.
(ctf_type_compat): Likewise.
(ctf_member_info): Likewise.
(ctf_array_info): Likewise.
(ctf_enum_name): Likewise.
(ctf_enum_value): Likewise.
(ctf_member_iter): Likewise.
(ctf_enum_iter): Likewise.
(ctf_type_iter): Likewise.
(ctf_variable_iter): Likewise.
These functions let you open an ELF file with a customarily-named CTF
section in it, automatically opening the CTF file or archive and
associating the symbol and string tables in the ELF file with the CTF
container, so that you can look up the types of symbols in the ELF file
via ctf_lookup_by_symbol(), and so that strings can be shared between
the ELF file and CTF container, to save space.
It uses BFD machinery to do so. This has now been lightly tested and
seems to work. In particular, if you already have a bfd you can pass
it in to ctf_bfdopen(), and if you want a bfd made for you you can
call ctf_open() or ctf_fdopen(), optionally specifying a target (or
try once without a target and then again with one if you get
ECTF_BFD_AMBIGUOUS back).
We use a forward declaration for the struct bfd in ctf-api.h, so that
ctf-api.h users are not required to pull in <bfd.h>. (This is mostly
for the sake of readelf.)
libctf/
* ctf-open-bfd.c: New file.
* ctf-open.c (ctf_close): New.
* ctf-impl.h: Include bfd.h.
(ctf_file): New members ctf_data_mmapped, ctf_data_mmapped_len.
(ctf_archive_internal): New members ctfi_abfd, ctfi_data,
ctfi_bfd_close.
(ctf_bfdopen_ctfsect): New declaration.
(_CTF_SECTION): likewise.
include/
* ctf-api.h (struct bfd): New forward.
(ctf_fdopen): New.
(ctf_bfdopen): Likewise.
(ctf_open): Likewise.
(ctf_arc_open): Likewise.
If you need to store a large number of CTF containers somewhere, this
provides a dedicated facility for doing so: an mmappable archive format
like a very simple tar or ar without all the system-dependent format
horrors or need for heavy file copying, with built-in compression of
files above a particular size threshold.
libctf automatically mmap()s uncompressed elements of these archives, or
uncompresses them, as needed. (If the platform does not support mmap(),
copying into dynamically-allocated buffers is used.)
Archive iteration operations are partitioned into raw and non-raw
forms. Raw operations pass thhe raw archive contents to the callback:
non-raw forms open each member with ctf_bufopen() and pass the resulting
ctf_file_t to the iterator instead. This lets you manipulate the raw
data in the archive, or the contents interpreted as a CTF file, as
needed.
It is not yet known whether we will store CTF archives in a linked ELF
object in one of these (akin to debugdata) or whether they'll get one
section per TU plus one parent container for types shared between them.
(In the case of ELF objects with very large numbers of TUs, an archive
of all of them would seem preferable, so we might just use an archive,
and add lzma support so you can assume that .gnu_debugdata and .ctf are
compressed using the same algorithm if both are present.)
To make usage easier, the ctf_archive_t is not the on-disk
representation but an abstraction over both ctf_file_t's and archives of
many ctf_file_t's: users see both CTF archives and raw CTF files as
ctf_archive_t's upon opening, the only difference being that a raw CTF
file has only a single "archive member", named ".ctf" (the default if a
null pointer is passed in as the name). The next commit will make use
of this facility, in addition to providing the public interface to
actually open archives. (In the future, it should be possible to have
all CTF sections in an ELF file appear as an "archive" in the same
fashion.)
This machinery is also used to allow library-internal creators of
ctf_archive_t's (such as the next commit) to stash away an ELF string
and symbol table, so that all opens of members in a given archive will
use them. This lets CTF archives exploit the ELF string and symbol
table just like raw CTF files can.
(All this leads to somewhat confusing type naming. The ctf_archive_t is
a typedef for the opaque internal type, struct ctf_archive_internal: the
non-internal "struct ctf_archive" is the on-disk structure meant for
other libraries manipulating CTF files. It is probably clearest to use
the struct name for struct ctf_archive_internal inside the program, and
the typedef names outside.)
libctf/
* ctf-archive.c: New.
* ctf-impl.h (ctf_archive_internal): New type.
(ctf_arc_open_internal): New declaration.
(ctf_arc_bufopen): Likewise.
(ctf_arc_close_internal): Likewise.
include/
* ctf.h (CTFA_MAGIC): New.
(struct ctf_archive): New.
(struct ctf_archive_modent): Likewise.
* ctf-api.h (ctf_archive_member_f): New.
(ctf_archive_raw_member_f): Likewise.
(ctf_arc_write): Likewise.
(ctf_arc_close): Likewise.
(ctf_arc_open_by_name): Likewise.
(ctf_archive_iter): Likewise.
(ctf_archive_raw_iter): Likewise.
(ctf_get_arc): Likewise.
This fills in the other half of the opening/creation puzzle: opening of
already-existing CTF files. Such files are always read-only: if you
want to add to a CTF file opened with one of the opening functions in
this file, use ctf_add_type(), in a later commit, to copy appropriate
types into a newly ctf_create()d, writable container.
The lowest-level opening functions are in here: ctf_bufopen(), which
takes ctf_sect_t structures akin to ELF section headers, and
ctf_simple_open(), which can be used if you don't have an entire ELF
section header to work from. Both will malloc() new space for the
buffers only if necessary, will mmap() directly from the file if
requested, and will mprotect() it afterwards to prevent accidental
corruption of the types. These functions are also used by ctf_update()
when converting types in a writable container into read-only types that
can be looked up using the lookup functions (in later commits).
The files are always of the native endianness of the system that created
them: at read time, the endianness of the header magic number is used to
determine whether or not the file needs byte-swapping, and the entire
thing is aggressively byte-swapped.
The agggressive nature of this swapping avoids complicating the rest of
the code with endianness conversions, while the native endianness
introduces no byte-swapping overhead in the common case. (The
endianness-independence code is also much newer than everything else in
this file, and deserves closer scrutiny.)
The accessors at the top of the file are there to transparently support
older versions of the CTF file format, allowing translation from older
formats that have different sizes for the structures in ctf.h:
currently, these older formats are intermingled with the newer ones in
ctf.h: they will probably migrate to a compatibility header in time, to
ease readability. The ctf_set_base() function is split out for the same
reason: when conversion code to a newer format is written, it would need
to malloc() new storage for the entire ctf_file_t if a file format
change causes it to grow, and for that we need ctf_set_base() to be a
separate function.
One pair of linked data structures supported by this file has no
creation code in libctf yet: the data and function object sections read
by init_symtab(). These will probably arrive soon, when the linker comes
to need them. (init_symtab() has hardly been changed since 2009, but if
any code in libctf has rotted over time, this will.)
A few simple accessors are also present that can even be called on
read-only containers because they don't actually modify them, since the
relevant things are not stored in the container but merely change its
operation: ctf_setmodel(), which lets you specify whether a container is
LP64 or not (used to statically determine the sizes of a few types),
ctf_import(), which is the only way to associate a parent container with
a child container, and ctf_setspecific(), which lets the caller
associate an arbitrary pointer with the CTF container for any use. If
the user doesn't call these functions correctly, libctf will misbehave:
this is particularly important for ctf_import(), since a container built
against a given parent container will not be able to resolve types that
depend on types in the parent unless it is ctf_import()ed with a parent
container with the same set of types at the same IDs, or a superset.
Possible future extensions (also noted in the ctf-hash.c file) include
storing a count of things so that we don't need to do one pass over the
CTF file counting everything, and computing a perfect hash at CTF
creation time in some compact form, storing it in the CTF file, and
using it to hash things so we don't need to do a second pass over the
entire CTF file to set up the hashes used to go from names to type IDs.
(There are multiple such hashes, one for each C type namespace: types,
enums, structs, and unions.)
libctf/
* ctf-open.c: New file.
* swap.h: Likewise.
include/
* ctf-api.h (ctf_file_close): New declaration.
(ctf_getdatasect): Likewise.
(ctf_parent_file): Likewise.
(ctf_parent_name): Likewise.
(ctf_parent_name_set): Likewise.
(ctf_import): Likewise.
(ctf_setmodel): Likewise.
(ctf_getmodel): Likewise.
(ctf_setspecific): Likewise.
(ctf_getspecific): Likewise.
The CTF creation process looks roughly like (error handling elided):
int err;
ctf_file_t *foo = ctf_create (&err);
ctf_id_t type = ctf_add_THING (foo, ...);
ctf_update (foo);
ctf_*write (...);
Some ctf_add_THING functions accept other type IDs as arguments,
depending on the type: cv-quals, pointers, and structure and union
members all take other types as arguments. So do 'slices', which
let you take an existing integral type and recast it as a type
with a different bitness or offset within a byte, for bitfields.
One class of THING is not a type: "variables", which are mappings
of names (in the internal string table) to types. These are mostly
useful when encoding variables that do not appear in a symbol table
but which some external user has some other way to figure out the
address of at runtime (dynamic symbol lookup or querying a VM
interpreter or something).
You can snapshot the creation process at any point: rolling back to a
snapshot deletes all types and variables added since that point.
You can make arbitrary type queries on the CTF container during the
creation process, but you must call ctf_update() first, which
translates the growing dynamic container into a static one (this uses
the CTF opening machinery, added in a later commit), which is quite
expensive. This function must also be called after adding types
and before writing the container out.
Because addition of types involves looking up existing types, we add a
little of the type lookup machinery here, as well: only enough to
look up types in dynamic containers under construction.
libctf/
* ctf-create.c: New file.
* ctf-lookup.c: New file.
include/
* ctf-api.h (zlib.h): New include.
(ctf_sect_t): New.
(ctf_sect_names_t): Likewise.
(ctf_encoding_t): Likewise.
(ctf_membinfo_t): Likewise.
(ctf_arinfo_t): Likewise.
(ctf_funcinfo_t): Likewise.
(ctf_lblinfo_t): Likewise.
(ctf_snapshot_id_t): Likewise.
(CTF_FUNC_VARARG): Likewise.
(ctf_simple_open): Likewise.
(ctf_bufopen): Likewise.
(ctf_create): Likewise.
(ctf_add_array): Likewise.
(ctf_add_const): Likewise.
(ctf_add_enum_encoded): Likewise.
(ctf_add_enum): Likewise.
(ctf_add_float): Likewise.
(ctf_add_forward): Likewise.
(ctf_add_function): Likewise.
(ctf_add_integer): Likewise.
(ctf_add_slice): Likewise.
(ctf_add_pointer): Likewise.
(ctf_add_type): Likewise.
(ctf_add_typedef): Likewise.
(ctf_add_restrict): Likewise.
(ctf_add_struct): Likewise.
(ctf_add_union): Likewise.
(ctf_add_struct_sized): Likewise.
(ctf_add_union_sized): Likewise.
(ctf_add_volatile): Likewise.
(ctf_add_enumerator): Likewise.
(ctf_add_member): Likewise.
(ctf_add_member_offset): Likewise.
(ctf_add_member_encoded): Likewise.
(ctf_add_variable): Likewise.
(ctf_set_array): Likewise.
(ctf_update): Likewise.
(ctf_snapshot): Likewise.
(ctf_rollback): Likewise.
(ctf_discard): Likewise.
(ctf_write): Likewise.
(ctf_gzwrite): Likewise.
(ctf_compress_write): Likewise.
We now enter a series of commits that are sufficiently tangled that
avoiding forward definitions is almost impossible: no attempt is made to
make individual commits compilable (which is why the build system does
not reference any of them yet): the only important thing is that they
should form something like conceptual groups.
But first, some definitions, including the core ctf_file_t itself. Uses
of these definitions will be introduced in later commits.
libctf/
* ctf-impl.h: New definitions and declarations for type creation
and lookup.
libctf maintains two distinct hash ADTs, one (ctf_dynhash) for wrapping
dynamically-generated unknown-sized hashes during CTF file construction,
one (ctf_hash) for wrapping unchanging hashes whose size is known at
creation time for reading CTF files that were previously created.
In the binutils implementation, these are both fairly thin wrappers
around libiberty hashtab.
Unusually, this code is not kept synchronized with libdtrace-ctf,
due to its dependence on libiberty hashtab.
libctf/
* ctf-hash.c: New file.
* ctf-impl.h: New declarations.
CTF functions return zero on success or an extended errno value which
can be translated into a string via the functions in this commit.
The errno numbers start at -CTF_BASE.
libctf/
* ctf-error.c: New file.
include/
* ctf-api.h (ctf_errno): New declaration.
(ctf_errmsg): Likewise.
These utilities are a bit of a ragbag of small things needed by more
than one TU: list manipulation, ELF32->64 translators, routines to look
up strings in string tables, dynamically-allocated string appenders, and
routines to set the specialized errno values previously committed in
<ctf-api.h>.
We do still need to dig around in raw ELF symbol tables in places,
because libctf allows the caller to pass in the contents of string and
symbol sections without telling it where they come from, so we cannot
use BFD to get the symbols (BFD reasonably demands the entire file). So
extract minimal ELF definitions from glibc into a private header named
libctf/elf.h: later, we use those to get symbols. (The start-of-
copyright range on elf.h reflects this glibc heritage.)
libctf/
* ctf-util.c: New file.
* elf.h: Likewise.
* ctf-impl.h: Include it, and add declarations.
The memory-allocation wrappers are simple things to allow malloc
interposition: they are only used inconsistently at present, usually
where malloc debugging was required in the past.
These provide a default implementation that is environment-variable
triggered (initialized on the first call to the libctf creation and
file-opening functions, the first functions people will use), and
a ctf_setdebug()/ctf_getdebug() pair that allows the caller to
explicitly turn debugging off and on. If ctf_setdebug() is called,
the automatic setting from an environment variable is skipped.
libctf/
* ctf-impl.h: New file.
* ctf-subr.c: New file.
include/
* ctf-api.h (ctf_setdebug): New.
(ctf_getdebug): Likewise.
