Updated April 28th: Added illustrations.
A while ago I published a post referencing Hollywoods take on programming anno 2001. It covered how to load additional code into a Unity build at runtime, which can be quite useful for patching, modding and similar post-publishing changes to your build, involving logic changes.
The post was pretty brief, just covering the raw basics of the technique and not dealing with any of the challenges in integrating it with a more complete patching solution.
Luckily I have recently had a chance to return to working with it and implementing it fully on a project. This post will cover the challenges met and my solutions to them.
Unlike the original post, this one is not exactly brief, so, uh, hang on to something?
The basic method of integrating new code in an existing Unity runtime is as follows:
- Build the patch code to a .net assembly.
- Load the byte stream of that assembly to an array at runtime.
- Create a System.Reflection.Assembly instance from the bytes.
- Instantiate classes, attach components and call methods via reflection.
This works really well for simple logic expansions. However if you want to couple the logic update with content such as prefabs and scenes using new components defined in this new assembly, you are out of luck.
New content can be loaded into Unity runtimes via asset bundles (these require Unity pro at time of writing). However in order to build these asset bundles as well as loading them afterwards, Unity needs to know all the components used in the contained scenes and prefabs.
Unfortunately this is not possible as that would require the presence of our new MonoBehaviours in the shipped build, which we are obviously not very interested in, given that we are working on a post-shipping patch.
The challenge is therefore to somehow construct asset bundles which can be built and loaded without the Unity editor or runtime knowing about all the MonoBehaviours used in the bundles.
Right, getting this one sorted will take a few lines to explain, but let’s just jump straight into it. The solution I went with goes a little something like this:
- Compile bundle scripts to an assembly, from which components are attached in bundle scenes and on bundle prefabs.
- Before bundle build, pack each of these components into a container component, holding the data and type of the original, replacing it as attached on the same Game Object.
- When loading the bundle, load the new assembly and then run an unpack step for each container component on each prefab, restoring the original data and references to a newly instantiated component of the correct type, on the same GameObject.
- When scenes from the bundle are loaded, the same unpacking process runs for each container component in there.
While this does sound very straight forward, there are of-course a few gotchas in there. So let us jump in a bit deeper.
The goal here is to make sure that the programmers, level designers and artists working on new bundles get to maintain as much of their familiar Unity workflow as possible.
So whatever building and packing solution we set up, the following must still be supported at the very least:
- Assets are imported and used in prefabs and scenes as per usual.
- Scripts are auto-compiled when saved and can be drag-dropped onto Game Objects.
- Variables and references can easily be tweaked on new bundle components via the inspector.
At first, these workflow requirements do seem to conflict with our technical requirements, given that all code unique to the bundle must be built into a separate assembly loaded at runtime, rather than be part of the main script compile.
However where there’s a will there’s a way and a tall piles of hacks make for interesting realities.
In the Unity Hacks talk I demonstrate how to use an asset post processor to compile java files into a jar plugin automatically. The first step to keeping our workflow without breaking the technical requirements is to copy this, but for C# (and/or JS) files compiled into a .net assembly.
Rather than spending time going into detail on how you would do that, I point you to the beforementioned talk (and its available example code) as well as the gmcs compiler shipped with the Unity editor. You should be able to construct the flow just fine from that.
One important note is that the folder from which your asset post processor picks up your scripts should be somewhere in Assets/WebplayerTemplates, as Unity by default will not compile scripts located there into the main project assembly (this is set up to avoid website js getting mixed up for templates).
I chose a location looking something like this: Assets/WebplayerTemplates/Bundles/MyFirstBundle.
The asset post processor building the assembly should then drop it in a folder like Assets/Bundles/MyFirstBundle and make a duplicate of it, with the .txt extension.
The original will be from where you can drag components and the copy will be the one built into the bundle, since it is now imported as a TextAsset and can therefore be included, unlike the original assembly.
At this point you can start work on the contents of the bundle, saving scenes, prefabs and other assets unique to the bundle into the same folder as the assembly and text file – subfolders under that would work fine as well.
It is important to note that attaching behaviours from the assembly, as well as behaviours defined in the core project to scene objects and prefabs in the bundle is just fine. However prefabs, scenes and scripts outside of the bundle should never reference behaviours or assets from the bundle directly.
It is worth spending a little bit of time writing an automated build step of the core project, which deletes all the bundle folders before running the actual build. That way you can, after testing, be pretty sure that nothing in the core project inadvertently referenced the contents of a bundle directly.
Should you feel like wanting a bit earlier warning on something like that, I would recommend spending a bit of time writing an extension checking for such references as scenes are saved and prefabs imported.
The folder created earlier in under Assets/WebPlayerTemplates is only used for storing scripts and building them into assemblies. So at the point of building the assembly, all that matters is the contents of the folder with the resulting assembly, its .txt duplicate and any other assets placed in there.
Before running the actual bundle build process, we first (as outlined in the Approach) need to pack and remove all instances of MonoBehaviours defined in the bundle assembly.
First, run through each prefab and scene in the bundle folder and consider each instance of these behaviours.
For each, attach to the same Game Object an instance of your container component type, defined in the core project – I am calling it BundledBehaviour in this post.
Store the full type name of the original in the BundledBehavior (we will need it to re-instantiate the original when unpacking) and add the two as a pair to a list.
Next, all serialised fields on the original must be stored in the BundledBehaviour. Iterating over that is a breeze with the SerialisedObject API.
Value types can just be straight up serialised to a byte array and chucked onto the BundledBehaviour. For my implementation I chose to use protobuf-net for that, but you can really use whatever meets your fancy.
Once the value types are dealt with, we should move on to the next behaviour in the list, before handling reference types in a second pass. The reason for this will become apparent when we start dealing with inter-component referencing.
Bam. Done. Wasn’t that quick? On to the second pass…
Reference types get a bit trickier than value types. For each serialised reference, we need to add to a list on the BundledBehaviour an instance of a class pairing the field name and the reference. In the context of this post I will refer to that container class as ReferenceField.
At this point we unfortunately need to start distinguishing between asset type references, like textures and audio clips, and component type references. Where storing component references in a field of type Component works just fine, unfortunately asset references need to be stored on a field of their exact type.
I forget the exact reason for the Unity serialisation system working like this, but for the purpose of this post, how about we just file that under “the law of the land” and move on?
The result is that the ReferenceField type will need to have a “field name” string field, a “component reference” Component field and then an “X reference” X field for each asset type X.
Annoying to deal with, but remember that these are just references, so after having written that list once, the slight overhead and dealing with it in general – through that wonderful interface you decided to put on it, will carry a minimum of pain.
When storing component references, do remember to first check if the reference is contained in that list of MonoBehaviour → BundledBehaviour pairs built earlier. If so, store the a reference to the BundledBehaviour in stead.
Now that all components instances from the bundle assembly have had a BundledBehaviour instance created, containing the type name, value type field data and reference type field references, we can safely go ahead and destroy all the original behaviours.
Great! Finally before building the actual bundle, a neat trick is to create a little asset bundle manifest – basically a custom asset type which will be set as the main asset of the bundle, referencing assets we need handy.
Again, since there is plenty of documentation on how to do that in the Unity Hacks talk, I would direct you there rather than explain it here.
The bundle manifest should reference the TextAsset of the bundle assembly (since we need to load that before doing anything else) as well as all the prefabs being built into the assembly.
Once that is set up, all there is left to do is to initiate the asset bundle build process, specifying the bundle manifest as the main asset, rejoicing that this crazy cabal of hacks somehow worked.
So now we have an asset bundle which we need to somehow magically turn into a whole new self-contained game runtime or whatever it was we put in there.
First up, via bundle manifest – accessible as the main asset of the bundle, the bytes of the assembly need to be passed to a System.Reflection.Assembly constructor.
With that done, all the behaviours defined in the assembly are available for attaching.
Next, all prefabs should be unpacked first. For each BundledBehaviour attached, run an Unpack function.
By passing the type name stored on the BundledBehaviour to a GetType call, we can get a reference to the original type, which we can pass to AddComponent, creating a new instance of the original. Store this pairing between BundledBehaviour and new instance in a list – just like during the packing process.
Once available deserialise the simple fields, as stored in byte form on the BundledBehaviour, onto the new instance.
For each asset reference in the list of ReferenceFields, copy over the reference to the correct field on the new instance.
As with the packing process, dealing with reference type fields should be pushed to a second pass – once all instances of the original types are available.
We can now pass over all BundledBehaviour previously treated and copy all references stored in the list of ReferenceFields to the members with the proper names – as specified in the same.
When copying over a reference, check first if it is one of our BundledBehaviour instances and if so, copy over the new instance it is paired with in stead.
With all original data and references restored, all instances of BundledBehaviour on the prefabs can now be destroyed.
Scenes can now be loaded normally from the bundle, since all prefabs have now been properly unpacked. BundledBehaviour should in its Awake handler start the same unpacking process used on the prefabs, so that loading a scene automatically unpacks it.
Violá! New inter-depended content and logic is now live in your runtime!
Recommendations and Limitations
Seeing as the BundledBehaviours are using Awake, that basically rules out behaviours depending on one-another during Awake. Not to worry though, you still have OnEnable and Start for initialisation. Just keep it in mind.
Depending on your use case for this, you might end up with an interesting web of different versions of bundles loaded on any given client machine. I would therefore recommend not spending time and resources on building a dependency tree between bundles and in stead strictly forbid one bundle using components defined in another. Additionally it would be a good idea to designate one namespace per bundle.
As mentioned in the original post, this whole trick depends on the mono JIT runtime being available. That means no fun for you if you are targeting iOS or the consoles – at time of writing, those are the platforms on which Unity employs AOT rather than JIT.
Before we run out and celebrate our great fortune, a word on security: Since a likely application for this technique is to load self-contained bundles over the internet, remember that a third party could potentially insert themselves mid-stream and modify the contents of your transmission.
If we were just transmitting textures and meshes, that should be just fine, but in this instance we are transmitting executable code which we intend to run in our application.
So it would probably be a good idea to somehow guard against someone inserting malicious code in there. At the very least you should employ some public key signing of the bundles – optimally doing the whole transfer on a secured connection (whatever that means these days).
And that is about it. A fair bit of text, but I hope you found some use in this postmortem-ish post. I had fun writing it at least :)