mirror/nasm
2
0
Fork 0
mirror of https://github.com/netwide-assembler/nasm.git synced 2024-12-21 09:19:31 +08:00
Code Issues Packages Projects Releases Wiki Activity
54dd03bfe5
nasm/pptok.pl

271 lines
7.5 KiB
Perl
Raw Normal View History

Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
#!/usr/bin/perl
Add copyright verbiage to Perl scripts; update LICENSE This adds copyright verbiage to the Perl scripts. Scripts that are known to be clean w.r.t. the 2-clause BSD license are given that license; unclear ones are given the "LGPL for now". Signed-off-by: H. Peter Anvin <hpa@zytor.com>
2009-06-29 07:52:56 +08:00
## --------------------------------------------------------------------------
##
## Copyright 1996-2009 The NASM Authors - All Rights Reserved
## See the file AUTHORS included with the NASM distribution for
## the specific copyright holders.
##
## Redistribution and use in source and binary forms, with or without
## modification, are permitted provided that the following
## conditions are met:
##
## * Redistributions of source code must retain the above copyright
## notice, this list of conditions and the following disclaimer.
## * Redistributions in binary form must reproduce the above
## copyright notice, this list of conditions and the following
## disclaimer in the documentation and/or other materials provided
## with the distribution.
##
## THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND
## CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES,
## INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF
## MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
## DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR
## CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
## SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
## NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES;
## LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION)
## HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN
## CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR
## OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE,
## EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
##
## --------------------------------------------------------------------------
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
#
macros.c: compress by tokenizing macro directives Compress macros.c by representing macro directives with a single byte. We can do this because we only use the ASCII character range inside the standard macro files. Note: we could save significant additional space by not having a pointer array, and instead relying on the fact that we sweep sequentially through the output array.
2008-06-20 07:00:04 +08:00
# Produce pptok.c, pptok.h and pptok.ph from pptok.dat
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
#
require 'phash.ph';
my($what, $in, $out) = @ARGV;
#
# Read pptok.dat
#
open(IN, "< $in") or die "$0: cannot open: $in\n";
while (defined($line = <IN>)) {
chomp $line;
$line =~ s/^\s+//; # Remove leading whitespace
$line =~ s/\s*\#.*$//; # Remove comments and trailing whitespace
next if ($line eq '');
Formatting: kill off "stealth whitespace" "Stealth whitespace" makes it harder to read diffs, and just generally cause unwanted weirdness. Do a source-wide pass to get rid of it.
2007-10-20 05:42:29 +08:00
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
if ($line =~ /^\%(.*)\*$/) {
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
push(@cctok, $1);
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
} elsif ($line =~ /^\%(.*)$/) {
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
push(@pptok, $1);
} elsif ($line =~ /^\*(.*)$/) {
push(@cond, $1);
}
}
close(IN);
Generate automatically correct tests for %if and %elif Automatically generate macros to test for %if and %elif variants, which are guaranteed to be correct across future changes.
2007-09-12 10:12:07 +08:00
@cctok = sort @cctok;
@cond = sort @cond;
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
@pptok = sort @pptok;
# Generate the expanded list including conditionals. The conditionals
# are at the beginning, padded to a power of 2, with the inverses
# interspersed; this allows a simple mask to pick out the condition.
Generate automatically correct tests for %if and %elif Automatically generate macros to test for %if and %elif variants, which are guaranteed to be correct across future changes.
2007-09-12 10:12:07 +08:00
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
while ((scalar @cond) & (scalar @cond)-1) {
push(@cond, undef);
}
@cptok = ();
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
foreach $ct (@cctok) {
foreach $cc (@cond) {
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
if (defined($cc)) {
push(@cptok, $ct.$cc);
push(@cptok, $ct.'n'.$cc);
} else {
push(@cptok, undef, undef);
}
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
}
}
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
$first_uncond = $pptok[0];
@pptok = (@cptok, @pptok);
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
open(OUT, "> $out") or die "$0: cannot open: $out\n";
#
# Output pptok.h
#
if ($what eq 'h') {
macros.c: compress by tokenizing macro directives Compress macros.c by representing macro directives with a single byte. We can do this because we only use the ASCII character range inside the standard macro files. Note: we could save significant additional space by not having a pointer array, and instead relying on the fact that we sweep sequentially through the output array.
2008-06-20 07:00:04 +08:00
print OUT "/* Automatically generated from $in by $0 */\n";
print OUT "/* Do not edit */\n";
print OUT "\n";
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
print OUT "enum preproc_token {\n";
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
$n = 0;
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
foreach $pt (@pptok) {
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
if (defined($pt)) {
printf OUT " %-16s = %3d,\n", "PP_\U$pt\E", $n;
}
$n++;
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
}
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
printf OUT " %-16s = %3d\n", 'PP_INVALID', -1;
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
print OUT "};\n";
Generate automatically correct tests for %if and %elif Automatically generate macros to test for %if and %elif variants, which are guaranteed to be correct across future changes.
2007-09-12 10:12:07 +08:00
print OUT "\n";
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
print OUT "enum pp_conditional {\n";
$n = 0;
foreach $cc (@cond) {
if (defined($cc)) {
printf OUT " %-16s = %3d,\n", "PPC_IF\U$cc\E", $n;
}
$n += 2;
}
print OUT "};\n\n";
Formatting: kill off "stealth whitespace" "Stealth whitespace" makes it harder to read diffs, and just generally cause unwanted weirdness. Do a source-wide pass to get rid of it.
2007-10-20 05:42:29 +08:00
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
printf OUT "#define PP_COND(x) ((enum pp_conditional)((x) & 0x%x))\n",
Formatting: kill off "stealth whitespace" "Stealth whitespace" makes it harder to read diffs, and just generally cause unwanted weirdness. Do a source-wide pass to get rid of it.
2007-10-20 05:42:29 +08:00
(scalar(@cond)-1) << 1;
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
print OUT "#define PP_IS_COND(x) ((unsigned int)(x) < PP_\U$first_uncond\E)\n";
print OUT "#define PP_NEGATIVE(x) ((x) & 1)\n";
print OUT "\n";
Generate automatically correct tests for %if and %elif Automatically generate macros to test for %if and %elif variants, which are guaranteed to be correct across future changes.
2007-09-12 10:12:07 +08:00
foreach $ct (@cctok) {
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
print OUT "#define CASE_PP_\U$ct\E";
$pref = " \\\n";
foreach $cc (@cond) {
if (defined($cc)) {
print OUT "$pref\tcase PP_\U${ct}${cc}\E: \\\n";
print OUT "\tcase PP_\U${ct}N${cc}\E";
$pref = ":\\\n";
}
}
print OUT "\n"; # No colon or newline on the last one
Generate automatically correct tests for %if and %elif Automatically generate macros to test for %if and %elif variants, which are guaranteed to be correct across future changes.
2007-09-12 10:12:07 +08:00
}
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
}
#
# Output pptok.c
#
if ($what eq 'c') {
macros.c: compress by tokenizing macro directives Compress macros.c by representing macro directives with a single byte. We can do this because we only use the ASCII character range inside the standard macro files. Note: we could save significant additional space by not having a pointer array, and instead relying on the fact that we sweep sequentially through the output array.
2008-06-20 07:00:04 +08:00
print OUT "/* Automatically generated from $in by $0 */\n";
print OUT "/* Do not edit */\n";
print OUT "\n";
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
my %tokens = ();
my @tokendata = ();
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
my $n = 0;
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
foreach $pt (@pptok) {
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
if (defined($pt)) {
$tokens{'%'.$pt} = $n;
pptok.c: quick-and-dirty downcasing during prehashing Speed up pptok.c by just doing |= 0x20 instead of calling tolower() for every character during prehashing. This is good enough for our needs, since we don't have any tokens containing the characters @ [ \ ] _ nor any high-bit characters (in which case we'd have to worry about multibyte anyway.)
2007-09-13 01:02:55 +08:00
if ($pt =~ /[\@\[\]\\_]/) {
# Fail on characters which look like upper-case letters
# to the quick-and-dirty downcasing in the prehash
# (see below)
die "$in: invalid character in token: $pt";
}
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
}
$n++;
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
}
my @hashinfo = gen_perfect_hash(\%tokens);
if (!defined(@hashinfo)) {
die "$0: no hash found\n";
}
# Paranoia...
verify_hash_table(\%tokens, \@hashinfo);
Formatting: kill off "stealth whitespace" "Stealth whitespace" makes it harder to read diffs, and just generally cause unwanted weirdness. Do a source-wide pass to get rid of it.
2007-10-20 05:42:29 +08:00
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
($n, $sv, $g) = @hashinfo;
$sv2 = $sv+2;
Formatting: kill off "stealth whitespace" "Stealth whitespace" makes it harder to read diffs, and just generally cause unwanted weirdness. Do a source-wide pass to get rid of it.
2007-10-20 05:42:29 +08:00
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
die if ($n & ($n-1));
Formatting: kill off "stealth whitespace" "Stealth whitespace" makes it harder to read diffs, and just generally cause unwanted weirdness. Do a source-wide pass to get rid of it.
2007-10-20 05:42:29 +08:00
Portability fixes Concentrate compiler dependencies to compiler.h; make sure compiler.h is included first in every .c file (since some prototypes may depend on the presence of feature request macros.) Actually use the conditional inclusion of various functions (totally broken in previous releases.)
2007-10-03 12:53:51 +08:00
print OUT "#include \"compiler.h\"\n";
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
print OUT "#include <inttypes.h>\n";
print OUT "#include <ctype.h>\n";
print OUT "#include \"nasmlib.h\"\n";
Use the crc64 we already use as the perfect hash function prehash Use the same crc64 that we already use for the symbol table hash as the perfect hash function prehash. We appear to get radically faster convergence this way, and the crc64 is probably *faster*, since the table likely to be resident in memory.
2007-10-03 08:40:00 +08:00
print OUT "#include \"hashtbl.h\"\n";
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
print OUT "#include \"preproc.h\"\n";
print OUT "\n";
# Note that this is global.
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
printf OUT "const char * const pp_directives[%d] = {\n", scalar(@pptok);
foreach $d (@pptok) {
if (defined($d)) {
print OUT " \"%$d\",\n";
} else {
print OUT " NULL,\n";
}
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
}
print OUT "};\n";
Formatting: kill off "stealth whitespace" "Stealth whitespace" makes it harder to read diffs, and just generally cause unwanted weirdness. Do a source-wide pass to get rid of it.
2007-10-20 05:42:29 +08:00
pp_directives_len can be uint8_t Save a few hundred bytes...
2008-06-22 09:18:41 +08:00
printf OUT "const uint8_t pp_directives_len[%d] = {\n", scalar(@pptok);
macros.c: compress by tokenizing macro directives Compress macros.c by representing macro directives with a single byte. We can do this because we only use the ASCII character range inside the standard macro files. Note: we could save significant additional space by not having a pointer array, and instead relying on the fact that we sweep sequentially through the output array.
2008-06-20 07:00:04 +08:00
foreach $d (@pptok) {
printf OUT " %d,\n", defined($d) ? length($d)+1 : 0;
}
print OUT "};\n";
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
print OUT "enum preproc_token pp_token_hash(const char *token)\n";
print OUT "{\n";
# Put a large value in unused slots. This makes it extremely unlikely
# that any combination that involves unused slot will pass the range test.
# This speeds up rejection of unrecognized tokens, i.e. identifiers.
print OUT "#define UNUSED 16383\n";
print OUT " static const int16_t hash1[$n] = {\n";
for ($i = 0; $i < $n; $i++) {
my $h = ${$g}[$i*2+0];
print OUT " ", defined($h) ? $h : 'UNUSED', ",\n";
}
print OUT " };\n";
Formatting: kill off "stealth whitespace" "Stealth whitespace" makes it harder to read diffs, and just generally cause unwanted weirdness. Do a source-wide pass to get rid of it.
2007-10-20 05:42:29 +08:00
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
print OUT " static const int16_t hash2[$n] = {\n";
for ($i = 0; $i < $n; $i++) {
my $h = ${$g}[$i*2+1];
print OUT " ", defined($h) ? $h : 'UNUSED', ",\n";
}
print OUT " };\n";
Formatting: kill off "stealth whitespace" "Stealth whitespace" makes it harder to read diffs, and just generally cause unwanted weirdness. Do a source-wide pass to get rid of it.
2007-10-20 05:42:29 +08:00
Use the crc64 we already use as the perfect hash function prehash Use the same crc64 that we already use for the symbol table hash as the perfect hash function prehash. We appear to get radically faster convergence this way, and the crc64 is probably *faster*, since the table likely to be resident in memory.
2007-10-03 08:40:00 +08:00
print OUT " uint32_t k1, k2;\n";
print OUT " uint64_t crc;\n";
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
# For correct overflow behavior, "ix" should be unsigned of the same
# width as the hash arrays.
print OUT " uint16_t ix;\n";
print OUT "\n";
Use the crc64 we already use as the perfect hash function prehash Use the same crc64 that we already use for the symbol table hash as the perfect hash function prehash. We appear to get radically faster convergence this way, and the crc64 is probably *faster*, since the table likely to be resident in memory.
2007-10-03 08:40:00 +08:00
printf OUT " crc = crc64i(UINT64_C(0x%08x%08x), token);\n",
Formatting: kill off "stealth whitespace" "Stealth whitespace" makes it harder to read diffs, and just generally cause unwanted weirdness. Do a source-wide pass to get rid of it.
2007-10-20 05:42:29 +08:00
$$sv[0], $$sv[1];
Use the crc64 we already use as the perfect hash function prehash Use the same crc64 that we already use for the symbol table hash as the perfect hash function prehash. We appear to get radically faster convergence this way, and the crc64 is probably *faster*, since the table likely to be resident in memory.
2007-10-03 08:40:00 +08:00
print OUT " k1 = (uint32_t)crc;\n";
print OUT " k2 = (uint32_t)(crc >> 32);\n";
print OUT "\n";
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
printf OUT " ix = hash1[k1 & 0x%x] + hash2[k2 & 0x%x];\n", $n-1, $n-1;
More automation in the preprocessor conditionals handling Further automate the production of preprocessor conditionals. Now the code automatically folds if/elif and the negatives.
2007-09-12 12:18:37 +08:00
printf OUT " if (ix >= %d)\n", scalar(@pptok);
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
print OUT " return PP_INVALID;\n";
print OUT "\n";
pptok.c: handle holes in the pp_directives array We need to fail if we land in a hole in the pp_directives array, since we have a noncontiguous enum preproc_token now.
2007-09-12 13:18:20 +08:00
print OUT " if (!pp_directives[ix] || nasm_stricmp(pp_directives[ix], token))\n";
Use a perfect hash to look up preprocessor directives Use a perfect hash to look up preprocessor directives, and generate the preprocessor directive list automatically.
2007-09-12 09:29:43 +08:00
print OUT " return PP_INVALID;\n";
print OUT "\n";
print OUT " return ix;\n";
print OUT "}\n";
}
macros.c: compress by tokenizing macro directives Compress macros.c by representing macro directives with a single byte. We can do this because we only use the ASCII character range inside the standard macro files. Note: we could save significant additional space by not having a pointer array, and instead relying on the fact that we sweep sequentially through the output array.
2008-06-20 07:00:04 +08:00
#
# Output pptok.ph
#
if ($what eq 'ph') {
print OUT "# Automatically generated from $in by $0\n";
print OUT "# Do not edit\n";
print OUT "\n";
print OUT "%pptok_hash = (\n";
$n = 0;
foreach $tok (@pptok) {
if (defined($tok)) {
printf OUT " '%%%s' => %d,\n", $tok, $n;
}
$n++;
}
print OUT ");\n";
print OUT "1;\n";
}
Reference in New Issue Copy Permalink