Implements https://github.com/godotengine/godot-proposals/issues/3371.
New `target` presets
====================
The `tools` option is removed and `target` changes to use three new presets,
which match the builds users are familiar with. These targets control the
default optimization level and enable editor-specific and debugging code:
- `editor`: Replaces `tools=yes target=release_debug`.
* Defines: `TOOLS_ENABLED`, `DEBUG_ENABLED`, `-O2`/`/O2`
- `template_debug`: Replaces `tools=no target=release_debug`.
* Defines: `DEBUG_ENABLED`, `-O2`/`/O2`
- `template_release`: Replaces `tools=no target=release`.
* Defines: `-O3`/`/O2`
New `dev_build` option
======================
The previous `target=debug` is now replaced by a separate `dev_build=yes`
option, which can be used in combination with either of the three targets,
and changes the following:
- `dev_build`: Defines `DEV_ENABLED`, disables optimization (`-O0`/`/0d`),
enables generating debug symbols, does not define `NDEBUG` so `assert()`
works in thirdparty libraries, adds a `.dev` suffix to the binary name.
Note: Unlike previously, `dev_build` defaults to off so that users who
compile Godot from source get an optimized and small build by default.
Engine contributors should now set `dev_build=yes` in their build scripts or
IDE configuration manually.
Changed binary names
====================
The name of generated binaries and object files are changed too, to follow
this format:
`godot.<platform>.<target>[.dev][.double].<arch>[.<extra_suffix>][.<ext>]`
For example:
- `godot.linuxbsd.editor.dev.arm64`
- `godot.windows.template_release.double.x86_64.mono.exe`
Be sure to update your links/scripts/IDE config accordingly.
More flexible `optimize` and `debug_symbols` options
====================================================
The optimization level and whether to generate debug symbols can be further
specified with the `optimize` and `debug_symbols` options. So the default
values listed above for the various `target` and `dev_build` combinations
are indicative and can be replaced when compiling, e.g.:
`scons p=linuxbsd target=template_debug dev_build=yes optimize=debug`
will make a "debug" export template with dev-only code enabled, `-Og`
optimization level for GCC/Clang, and debug symbols. Perfect for debugging
complex crashes at runtime in an exported project.
This base implementation is still very barebones but it defines the path
for how exporting will work (at least when embedding the .NET runtime).
Many manual steps are still needed, which should be automatized in the
future. For example, in addition to the API assemblies, now you also
need to copy the GodotPlugins assembly to each game project.
We will be progressively moving most code to C#.
The plan is to only use Mono's embedding APIs to set things at launch.
This will make it much easier to later support CoreCLR too which
doesn't have rich embedding APIs.
Additionally the code in C# is more maintainable and makes it easier
to implement new features, e.g.: runtime codegen which we could use to
avoid using reflection for marshaling everytime a field, property or
method is accessed.
SOME NOTES ON INTEROP
We make the same assumptions as GDNative about the size of the Godot
structures we use. We take it a bit further by also assuming the layout
of fields in some cases, which is riskier but let's us squeeze out some
performance by avoiding unnecessary managed to native calls.
Code that deals with native structs is less safe than before as there's
no RAII and copy constructors in C#. It's like using the GDNative C API
directly. One has to take special care to free values they own.
Perhaps we could use roslyn analyzers to check this, but I don't know
any that uses attributes to determine what's owned or borrowed.
As to why we maily use pointers for native structs instead of ref/out:
- AFAIK (and confirmed with a benchmark) ref/out are pinned
during P/Invoke calls and that has a cost.
- Native struct fields can't be ref/out in the first place.
- A `using` local can't be passed as ref/out, only `in`. Calling a
method or property on an `in` value makes a silent copy, so we want
to avoid `in`.
REGARDING THE BUILD SYSTEM
There's no longer a `mono_glue=yes/no` SCons options. We no longer
need to build with `mono_glue=no`, generate the glue and then build
again with `mono_glue=yes`. We build only once and generate the glue
(which is in C# now).
However, SCons no longer builds the C# projects for us. Instead one
must run `build_assemblies.py`, e.g.:
```sh
%godot_src_root%/modules/mono/build_scripts/build_assemblies.py \
--godot-output-dir=%godot_src_root%/bin \
--godot-target=release_debug`
```
We could turn this into a custom build target, but I don't know how
to do that with SCons (it's possible with Meson).
OTHER NOTES
Most of the moved code doesn't follow the C# naming convention and
still has the word Mono in the names despite no longer dealing with
Mono's embedding APIs. This is just temporary while transitioning,
to make it easier to understand what was moved where.
Main benefits:
- Projects can be built offline. Previously you needed internet
access the first time building to download the packages.
- Changes to packages like Godot.NET.Sdk can be easily tested
before publishing. This was already possible but required
too many manual steps.
- First time builds are a bit faster, as the Sdk package doesn't
need to be downloaded. In practice, the package is very small
so it makes little difference.
Bumped Godot.NET.Sdk to 4.0.0-dev3 in order to enable the
recent changes regarding '.mono/' -> '.godot/mono/'.
Right now, games only work on devices when exported with FullAOT+Interpreter.
There are some issues left that need to addressed for FullAOT alone. Right now,
it's giving issues with the Godot.NativeCalls static constructor.
Configured for a max line length of 120 characters.
psf/black is very opinionated and purposely doesn't leave much room for
configuration. The output is mostly OK so that should be fine for us,
but some things worth noting:
- Manually wrapped strings will be reflowed, so by using a line length
of 120 for the sake of preserving readability for our long command
calls, it also means that some manually wrapped strings are back on
the same line and should be manually merged again.
- Code generators using string concatenation extensively look awful,
since black puts each operand on a single line. We need to refactor
these generators to use more pythonic string formatting, for which
many options are available (`%`, `format` or f-strings).
- CI checks and a pre-commit hook will be added to ensure that future
buildsystem changes are well-formatted.
Implementation for new Variant types Callable, Signal, StringName.
Added support for PackedInt64Array and PackedFloat64Array.
Add generation of signal members as events, as well as support for
user created signals as events.
NOTE: As of now, raising such events will not emit the signal. As such,
one must use `EmitSignal` instead of raising the event directly.
Removed old ThreadLocal fallback class. It's safe to use thread_local now since
it's supported on all minimum versions of compilers we support.
Previously we had a placeholder solution called 'Managed' to benefit from
tooling while editing the a part of the C# API.
Later the bindings generator would create the final 'GodotSharp' solution
including these C# files as well as the auto-generated C# API.
Now we replaced the 'Managed' solution with the final 'GodotSharp' solution
which is no longer auto-generated, and the bindings generator only takes
care of the auto-generated C# API.
This has the following benefits:
- It's less confusing as there will no longer be two versions of the same file
(the original and a generated copy of it). Now there's only one.
- We no longer need placeholder for auto-generated API classes, like Node or
Resource. We used them for benefiting from tooling. Now we can just use the
auto-generated API itself.
- Simplifies the build system and bindings generator. Removed lot of code
that is not needed anymore.
Also added a post-build target to the GodotTools project to copy the output to
the data dir. This makes it easy to iterate when doing changes to GodotTools,
as SCons doesn't have to be executed anymore just to copy these new files.
Make the build system automatically build the C# Api assemblies to be shipped with the editor.
Make the editor, editor player and debug export templates use Api assemblies built with debug symbols.
Always run MSBuild to build the editor tools and Api assemblies when building Godot.
Several bugs fixed related to assembly hot reloading and restoring state.
Fix StringExtensions internal calls not being registered correctly, resulting in MissingMethodException.
- `modules\mono\csharp_script.cpp(576): warning C4099: 'CSharpScriptDepSort': type name first seen using 'class' now seen using 'struct'`
- `modules\mono\signal_awaiter_utils.cpp(144): warning C4003: not enough actual parameters for macro 'ERR_FAIL_V'`
- `modules\mono\editor\net_solution.cpp(101): warning C4129: '%': unrecognized character escape sequence`
- (several) `modules\mono\glue\cs_compressed.gen.h(222): warning C4129: 'E': unrecognized character escape sequence`
- Bundle editor dependencies:
- 'GodotSharp': Root data directory for the editor
- 'Tools': Editor dependencies. Only GodotSharp.dll for now.
- 'Api': Prebuilt GodotSharp and GodotSharpEditor API assemblies.
- 'Mono': Mono files to bundle with the editor.
- 'bin': (Optional, not used for now) Mono bin directory.
- 'etc': Mono configuration files.
- 'lib': Mono dependency shared libraries.
- 'lib/mono/4.5': Framework assemblies.
- Added build option to copy the required files from the mono installation to 'GodotSharp/Mono'. Enable with 'copy_mono_root=yes'. Disabled by default.
- Export template dependencies:
- 'data_AppName'/'data_Godot':
- 'Mono': Mono files to bundle with the game.
- 'etc': Mono configuration files.
- 'lib': Mono dependency shared libraries.
- The data directory is generated when compiling and must be bundled with the export templates. In the case of OSX, the data directory must be placed inside the 'osx.zip' export template.
- In OSX, alternative location for directories (needed for app bundles) are:
- 'data_AppName/Mono/etc' --> '../Resources/GodotSharp/Mono/etc'
- 'data_AppName/Mono/lib' --> '../Frameworks/GodotSharp/Mono/lib'
- The editor can bundle prebuilt API assemblies.
- Generate them with a tools build by running: `--generate-cs-core-api <GodotSharp_OutputDir> --generate-cs-editor-api <GodotSharpEditor_OutputDir> <GodotSharp_OutputDir>/bin/Release/GodotSharp.dll` (This command will be simplified in the future and both projects will be in the same solution)
- Build the solutions and copy the output files to '#bin/GodotSharp/Api'.
- Fixed API assembly being added twice during the export process.
This allows more consistency in the manner we include core headers,
where previously there would be a mix of absolute, relative and
include path-dependent includes.
- We no longer generate RID and NodePath C# classes. Both will be maintained manually.
- We no longer generate C# declarations and runtime registration of internal calls for the following classes: RID, NodePath, String, GD, SignalAwaiter and Godot.Object (partial base).
- We no longer auto-generate the base members of Godot.Object. They will be maintained manually as a partial class.
This makes it easier to maintain these C# classes and their internal calls, as well as the bindings generator which no longer generates C# classes that don't derive from Godot Object, and it no longer generates the Godot.Object base members (which where unreadable in the bindings generator code).
- Added missing 'RID(Object from)' constructor to the RID C# class.
- Replaced MONO_GLUE_DISABLED constant macro with MONO_GLUE_ENABLED.
- Add sources in module/mono/glue even if glue is disabled, but surround glue files with ifdef MONO_GLUE_ENABLED.
- Refactored all builder (make_*) functions into separate Python modules along to the build tree
- Introduced utility function to wrap all invocations on Windows, but does not change it elsewhere
- Introduced stub to use the builders module as a stand alone script and invoke a selected function
There is a problem with file handles related to writing generated content (*.gen.h and *.gen.cpp)
on Windows, which randomly causes a SHARING VIOLATION error to the compiler resulting in flaky
builds. Running all such content generators in a new subprocess instead of directly inside the
build script works around the issue.
Yes, I tried the multiprocessing module. It did not work due to conflict with SCons on cPickle.
Suggested workaround did not fully work either.
Using the run_in_subprocess wrapper on osx and x11 platforms as well for consistency. In case of
running a cross-compilation on Windows they would still be used, but likely it will not happen
in practice. What counts is that the build itself is running on which platform, not the target
platform.
Some generated files are written directly in an SConstruct or SCsub file, before the parallel build starts. They don't need to be written in a subprocess, apparently, so I left them as is.
On macOS, it is common to install packages like Mono through the third-party
package-manager Homebrew. This commit simply adds an additional path to
where Homebrew installs the Mono framework.
- Add (Csc/Vbc/Fsc)ToolExe environment variables when running Mono's MSBuild.
- Fix directory for the 'mono_assemblies_output_dir' argument being created with the '#' top level directory token as part of its name.
- Allow to build with 'mono_static=yes' on Unix without specifying a mono prefix. The build script will try to find the mono prefix using the output from pkg-config.
- Bundle with mscorlib.dll to avoid compatibilities issues
- Add build option 'mono_assemblies_output_dir' to specify the output directory where the assemblies will be copied to. '#bin' by default.
- Make sure to search the mono installation directory for the right architecture in the windows registry.
- Do not build GodotSharpTools directly to #bin dir. Instead build to the default output path and copy it. This way we avoid MSBuild adding files we don't want to #bin.
- Add hint path for MSBuild in OSX.
- Copy shared library on Unix if not statically linking.
- Use vswhere to search MSBuild and search for 14.0 tools version in the registry instead of 4.0.
- SCons will only fallback xbuild when msbuild is not found if 'xbuild_fallback=yes' is passed to the command.
- Use mono's assembly path as FrameworkPathOverride if using with system's MSBuild (not mono's fork).
- Cleanup.
