2024-04-15 - Update, update, rollback
I'm finally back at work on the project, after a three weeks long break due to other work commitments and to the setup of a new PC.
It always take a little time to go back to a project after a break, but unfortunately I can't avoid some interruptions. Let's start by copy/pasting the TODO I left myself at the end of week 17:
I still need to add the tuning of the capture frequency, which shouldn't take long if I use the same UI control I used for the speed.
I want to be able to visualize a reference skeleton when recording clips, to help in capturing looping clips
I need to be able to playback a clip on a skinned mesh and not only using the debug gizmos (which only work in editor!)
after doing that, I need to test everything with an Android build running standalone on Quest (I've been using Airlink for now)
I'd say that the capture frequency setup is not a priority. I want to capture on device, so I need to have everything running there, including the reference skeleton to capture looping clips.
All this implies that it makes sense to start from the third item of the TODO list: I need to be able to playback clips using as target a suitable skinned mesh, and not only the editor-only skeleton visualization.
I had already cloned the repository of the project and installed Unity during my PC setup last week, but I only had tested that it was running in editor, with no headset connected. Now, instead, using Oculus Link, something is broken: I have a never seen before exception thrown from `XRPass.cs
`, a file part of the `com.unity.render-pipelines.core
` package.
The fun begins (not). Do I have a problem with the new PC setup, did I forget to commit something to the repository, or what? Let's dig deeper...
On my old PC, I was working on the project using Unity 2022.3.19f1.
While doing my new PC setup, in a moment of optimism, I installed the suggested release, which was just a few minor versions ahead: 2022.3.23f1.
It is reasonable to install the "old" version to see if the problem is related to the new release (or maybe to some of the packages I'm using that needs to be updated too).
And, surprise: since the day I installed Unity on the new PC, there's a new minor release available, the 2022.3.24f1. With some luck, the new minor release could fix the problem. Worth a shot...
I installed Unity 2022.3.24f1, but unfortunately the problem was still there. I also updated some packages from the package manager. Meanwhile, I googled the exception, and found a thread about the same problem I was having.
Well, at least is not just me, I guess. There's no fixes/workaround suggested, and the post is from today (!). I could try to find a workaround, but I prefer to let Unity employees do their job, and focus on my game. Time to surrender and install exactly the same version of Unity I was using on the old pc, which is probably what I should have done in first place. Silly optimism!
Guess what, with version 2022.3.19f1, the exception disappeared. Yay.
Still, the "particular" player avatar doesn't show up. This looks more like an error of mine.
I went and compared using _WinMerge_ the two copies of the project, on old and new pc.
This is exactly the kind of thing that makes me keep the old pc (and data) around for now, until I'm sure everything is in order on the new system.
I did find the problem, a couple of modified files in the Samples of the `com.meta.movement
` package which I left out of the repository (around 800 Mb!). My bad for modifying those files directly, instead of making copies of them (placed in my packages) and modify the copies.
After copying the modified samples, the project works well (and runs significantly faster than on the old PC, which will be great to improve my iteration time).
When I thought it was over, the Oculus App prompted me for an app and drivers update. And... I have the v64 firmware update available on the headset too, which I'm actually quite curious about, since I heard it improves the passthrough quality.
I'm almost too scared to try if everything keeps working after the updates... should I leave it for tomorrow?
No, I need to know.
Pheew, everything worked. Oculus app updated, firmware v64 running on the headset, and the Unity project looks fine. I lost an afternoon, but maybe tomorrow I will actually be able to do some progress.
2024-04-16 - Removing the sample assets
I have very little time today, but let's see if I manage to get at least one thing in.
There's quite a lot of SDK code related to the animation of a skinned through body tracking data.
I'm talking about code in the namespace `Oculus.Movement
` (from the `com.meta.movement
` package).
If needed, I will analyse it in depth, like I did for the `Oculus.Interaction.Body.Input
` namespace (from the `com.meta.xr.sdk.interaction
` package) in the `Deep dive into body tracking` special post.
But first, I'm going to try to cheat and use (almost) "blindly" the player character setup which I already have in the Unity scene as a starting point. I'm going to try to clone it and just change the data source, so that it uses data I pass in (from an animation clip) instead of the body tracking data coming from the headset.
Before cloning the character, I want to remove the dependency from the Samples assets I discussed yesterday, because otherwise I would make it worse: I would duplicate something depending on assets which shouldn't be in the project in the first place.
This is more time consuming that it should be, because some dependencies can be not obvious.
Something particular annoying are the prefab hierarchies, like this one:
So, in the scene I have `ArmatureSkinningUpdateRetargetSkeletonProcessor
`, but if I duplicate it in my package and rename it to `PRAvatar_SkinningUpdateRetargetSkeletonProcessor
`, that duplicate will still have as variant parent the `ArmatureSkinningUpdateRetarget
` prefab.
And that makes sense. But if want to proceed with my cloning, and duplicate `ArmatureSkinningUpdateRetarget
` to `PRAvatar_SkinningUpdateRetarget
`, there's no way (AFAIK) in Unity to "reparent" `PRAvatar_SkinningUpdateRetargetSkeletonProcessor
` to `PRAvatar_SkinningUpdateRetarget
`.
This is a case where I resort to violence, which mean editing the YAML of the prefabs to do the reparenting manually. The process is not obvious, so I'm going to describe it.
When you duplicate an asset, the duplicate gets a new `guid
`. These global identifiers are used by Unity to handle references between assets. Thanks to this approach, one can freely rename assets without breaking stuff.
So, to do the reparenting, we must change, in the child prefab, the references to the parent prefab `guid
`, replacing them with references to the "cloned father-to-be". You can find the `guid
` of an asset opening with a text editor its `.meta
` file.
Let's see, for example, the contents of `ArmatureSkinningUpdateRetarget.prefab.meta
`:
fileFormatVersion: 2
guid: eb6bc10f92728a546a3d8c144d0ef787
PrefabImporter:
externalObjects: {}
userData:
assetBundleName:
assetBundleVariant:
We also need to take note of the "cloned father-to-be" prefab, checking `PRAvatar_SkinningUpdateRetarget.prefab.meta`.
Done that, we know that
`
ArmatureSkinningUpdateRetarget
` has `guid
` `eb6bc10f92728a546a3d8c144d0ef787
``
PRAvatar_SkinningUpdateRetarget
` has `guid
` `521e1b23a8494cd438c7a90ea68f6d82
`
At this point, we can open with a text editor the child prefab, `PRAvatar_SkinningUpdateRetargetSkeletonProcessor.prefab
`, and do a "replace all" that changes every occurrence of `eb6bc10f92728a546a3d8c144d0ef787
` to `521e1b23a8494cd438c7a90ea68f6d82
`.
After doing that, when I go back to Unity, I can verify in the inspector that the re-parenting has happened, so now we have:
I repeated the process to reparent `PRAvatar_SkinningUpdateRetarget
` so that it points to `PRAvatar_SkinningUpdateTPose
` and not `ArmatureSkinningUpdateTPose
`.
As soon as I had my "complete" replica of the prefab hierarchy, I cloned the character and duplicated other referenced assets, so that the references point to the copy and not to the version coming from the Samples.
I put in the scene two character hierarchies: the original, and the new copy.
At this point, I can enable/disable the two `Character
` and `PR_Character
` nodes and they behave the same.
After a bit of supplementary trial and error for assets that I overlooked, I got my body tracked avatar working properly, and with no dependency from the Samples assets, exactly as I wanted.
The ultimate test was deleting the Samples folder of the movement package and test that everything was still working.
Here's my version of the assets. I'm not sure of the role of everything, but at lest now I know exactly which elements are in play.
Tomorrow I should hopefully manage to have multiple instances of my character rig running, with two different data sources activating the mesh animation (live tracking data from the headset, and recorded tracking data from a motion clip).
2024-04-17 - Playback on SkinnedMeshRenderer
Before wiring the different input data, I have to do a couple of minor things to prepare the character rig to exist in multiple instances.
First, I need to refactor the hack that I did a few weeks ago to scale down the legs to `0.01f
`: that is currently done by `PlayerAvatarBhv
`, which also handles other things like the mana/health bars.
I refactored the legs scaling moving it to a new `UpperBodyCharacterBhv
` script.
Every instance of the character rig will have this behaviour, independently from where the animation data comes from.
Now, I want to be able to easily differentiate between the player avatar and other instances, and the obvious thing that comes to my mind is changing color to the particles that make up the body.
I added to the VFX of the body a parameter `ParticlesColorGradient
` to be able to set the color by script.
The VFX node takes in a gradient, but being able to change the start color is sufficient.
This is the color change method I put into `UpperBodyCharacterBhv
`:
public void setParticlesColor(Color c) {
var rColorKeys = new GradientColorKey[] {
new GradientColorKey(c, 0f),
new GradientColorKey(Color.white, 1f),
};
var rAlphaKeys = new GradientAlphaKey[] {
new GradientAlphaKey(1f, 0f),
new GradientAlphaKey(1f, 1f)
};
Gradient rGradient = new Gradient();
rGradient.SetKeys(rColorKeys, rAlphaKeys);
m_rVFX.SetGradient("ParticlesColorGradient", rGradient);
}
Finally, it's time to start feeding to a second instance of the character some animation clip data.
The `PRAvatar_SkinningUpdateRetargetSkeletonProcessor
` prefab has a pretty complex setup, with a bunch of behaviours storing quite a lot of settings.
I don't see direct references to `OVRBody
`, so hopefully the other behaviours only access the tracking data through the interfaces it implements.
So, what I'm going to do, hoping that the Meta folks did things in a sensible way, is substituting the `OVRBody
` behaviour, that I know fetches the tracking data from the headset, with another behaviour (`UpperBodyCharacterDataProviderBhv
`) implementing the same interfaces.
The screenshot shows what my modified setup looks like, after collapsing all the attached scripts. Notice that I have removed `OVRBody
` and added `UpperBodyCharacterDataProviderBhv
`, which has empty stubs for the methods implementing the interfaces
`
OVRSkeleton.IOVRSkeletonDataProvider
``
OVRSkeletonRenderer.IOVRSkeletonRendererDataProvider
`
I'm now going to implement these interfaces and then add to `UpperBodyClipPlayerBhv
` a reference to this new behaviour, so that it can update its data, like it already does with the `PlayerAvatarDebugGizmos
` scripts that shows the debug skeleton we've been using to visualize recorded clip until today.
I implemented the interfaces and wired up the nodes appropriately, and it was finally time for testing. What happened?
Everybody loves videos featuring these kind of bugs, so here you are - I have no shame.
It looks like I overlooked something - it's not that strange, as the data expected by the two visualization differs. I have a pretty good idea of where to look to fix it, but it's late - I'll get it working properly first thing tomorrow.
2024-04-18 - Playback debugging
As it often happens, the problem shown in yesterday's video clip is harder to fix than expected.
I did some progress, but I also found a serious issue shat is shown by in the following clips, where I have attached two "runtime gizmos" to the hips bone of both character rigs, the blueish one for the playing character and the pinkish one for the clip playback.
The runtime gizmo I mentioned is a debug prefab I did a long time ago and always have around: you can make yours too in 5 minutes with zero 3D modeling skills, putting together three cones and three cylinders, and applying on them three unlit red/green/blue materials.
As you can see, the hips bone of the clip playback doesn't turn, while the arrow sticking out from my belly reasonably rotates according to my body position.
Everything looks working fine with the "skeleton" visualization because, I'm assuming, it doesn't really use the bones rotations, but just the joints positions, drawing lines between them.
Trying to use the same information to animate a skinned mesh, instead, fails miserably.
Additionally, the hands don't move at all, so there must be some other issue too.
I might hack around this, saving the data from the skinned mesh animation after the processing, but I don't really like that idea. It looks like I will need to dig deeper.
2024-04-19 - On-device serialization troubles
I doubt I could solve the issues I found out about yesterday in one afternoon, so I decided to end the week doing some other stuff.
This way, I'm going to tick off something from my list, and will postpone the more challenging task to next week.
Now that I have something, even if not correct, that should show up when running on Quest and not only in the Unity editor, I can make sure that the saving and loading of clips work on the headset too.
That's not to take for granted, as I'm using custom paths and file system access on Android has its own rules.
I tried running a build, and indeed, even if it doesn't crash and it plays back the latest recorded clip (that gets automatically set as "current clip"), the UI with the list of saved clips doesn't update.
Something is off, let's see if I can figure it out.
I don't remember if I already discussed the current save/load code, which is super basic.
Anyway, let's see the relevant parts.
private const string sANIMCLIPS_PATH = "_animclips/";
private static readonly JsonSerializerSettings JSON_SER_SETTINGS =
new JsonSerializerSettings {
TypeNameHandling = TypeNameHandling.Auto,
};
public static void storeAnimClip<H, T>(
string sClipId,
AnimClip<H, T> rClip
)
where H : ICopyFrom<H>
where T : ICopyFrom<T>, IInterpolable<T> {
string sFilePath = sANIMCLIPS_PATH + sClipId + ".ac.json";
store(sFilePath, rClip);
}
public static void store<T>(string sFilePath, T rData) {
if (Application.platform == RuntimePlatform.Android) {
sFilePath = Application.persistentDataPath + "/" + sFilePath;
}
string sDirPath = new FileInfo(sFilePath).Directory.FullName;
if (!Directory.Exists(sDirPath)) {
Directory.CreateDirectory(sDirPath);
}
Debug.Log("store to: " + sFilePath);
string sJsonText = serialize<T>(rData);
File.WriteAllText(sFilePath, sJsonText);
}
private static string serialize<T>(T rData) {
string sRet = JsonConvert.SerializeObject(
rData,
Formatting.Indented,
JSON_SER_SETTINGS
);
return sRet;
}
Good: while on PC I just save close to the project files, on Android I remembered that I must save into `Application.persistentDataPath
`, and I'm also logging the file path in `store
`.
I tried saving a clip and then checked the game data path on device, which is:
/sdcard/Android/data/com.BinaryCharm.ParticularRealityPrototypeURP/files/
To access the Quest filesystem I use SideQuest.
I'm not sure why the Meta Quest Developer Hub only lets me access some areas of the device filesystem (in its File Manager window) and if there's any fixes for that, but well, SideQuest works, I'm going to use that.
Anyway, in SideQuest the data path contains an `_animclips
` subdirectory, which is good news, but it's empty. Let's dig deeper...
This is one of those cases where attaching the debugger to the application running natively on an attached Android device (in this case, the Quest headset) is invaluable.
It kills iteration time, but if something works on PC and not on device, one needs to see what's going on, and that's the best way.
When I step into `JsonConvert.SerializeObject
`, called by my `serialize
` method, the control flow jumps out at some point. I know what that means: an exception got thrown.
Unfortunately, in these cases when you are running on device, you don't get the exception conveniently shown in the Unity console, and you need to search into the device logs, conveniently accessible through Meta Quest Developer Hub.
Here's the problem:
JsonSerializationException: Self referencing loop detected for property 'normalized' with type 'UnityEngine.Vector3'. Path 'rFrames[0].rJointPoses[0].vPos.normalized.normalized'.
I remember running into this in the past, but not exactly when, nor what I did to fix it.
After a quick google search, I remembered: this is one of the issues in the serialization of Unity types when using `Newtonsoft.Json
` library.
But I added to my project the additional package that fixes these problems since the day I added any kind of serialization, exactly because I had already ran into the issue in the past. And it's working on PC. So what's going on?
I'm not even upset: is one of those frequent remainders that "write once, deploy everywhere" is and has always been not more than a fairy tale: in practice, there's all kinds of subtle platform differences that make things go sideways.
Summarizing, the problem should be that, on Android, the JSON library is not using the custom converters provided by `Newtonsoft.Json-for-Unity.Converters
`.
The documentation says:
This package automatically adds all its converters to the `JsonConvert.DefaultSettings` if that value has been left untouched.
How does that work? I glimpsed at the code in `UnityConverterInitializer.cs
` and there's a part using reflection to find the suitable converters that makes my spider-sense tingle. It's happened to me in the past that code stripping has thrown away needed classes that weren't explicitly referenced. Maybe that's what's happening?
But I won't dig deeper if I don't need to, and the documentation also explains how to explicitly initialize the serialization settings providing additional converters, and that sounds worth a try. By referring to them, I might also avoid the classes to be stripped from the build.
new JsonSerializerSettings {
Converters = new JsonConverter[] {
new StringEnumConverter(),
new VersionConverter(),
new Vector2Converter(),
new Vector3Converter(),
new Vector4Converter(),
new Vector2IntConverter(),
new Vector3IntConverter(),
new QuaternionConverter(),
new ColorConverter(),
new Color32Converter(),
new Matrix4x4Converter(),
new LayerMaskConverter(),
new RangeIntConverter(),
new Hash128Converter(),
new BoundsConverter(),
new BoundsIntConverter(),
new PlaneConverter(),
new RectConverter(),
new RectIntConverter(),
new RectOffsetConverter()
},
ContractResolver = new UnityTypeContractResolver(),
TypeNameHandling = TypeNameHandling.Auto,
};
It's not pretty, because I need to enumerate the classes, but it's also simple and explicit - the kind of thing which is perfect to see if something works.
And guess what? It worked.
Here's the first clip of the clip management running on Quest and not on PC.
Good, another small step forward has been done.
You can see how saving and loading is super-slow, locking the game for seconds, but that's fine for a development-only file format. In the final build of the game there won't be any JSON handling: at some point I will switch to tightly packed binary files and asynchronous file system operations.