2024-02-12 - Revisiting the Meta samples
As first step in the process of having a proper player avatar, I'm going to revisit the Meta documentation. I remember trying a pretty good full body skeleton sample when, during the first phase of prototyping, I was first exploring the SDK.
I checked my notes and tried again the sample I found interesting, which is `DebugBodyJoints
` from the XR interaction SDK Samples.
It works pretty well, doing a good job deriving the arms and shoulders transforms on the basis of the (incomplete) tracking data. This is what I plan to use, in the spirit of Particular Reality where the game gets designed around the hardware limitations.
While in a realistic setting providing the avatar with legs could be important (I mean: I think that showing only the hands like in Half Life: Alyx is fine, but a floating torso can be weird), in the abstract game world I'm creating there's no reason why the entity the player is controlling should have any legs. Also, I've yet to see an inverse kinematic system able to "guess the legs" in a convincing manner without additional trackers.
While looking for information about the sample in the online documentation, I discovered there's now an extra package (`Unity-Movement
`) providing high level features related to body tracking, eye tracking and face tracking.
I'm only interested in the body tracking stuff, but it's worth checking out this new package and see if it provides something valuable for the project (it might offer better IK, or a more convenient API, etc).
Reading about `Unity-Movement
`, I found out that there's been a problem with some Unity versions, including the one I'm using:
Unity 2022 and 2023 introduces a problem with the retargeting scenes. Transform positions cannot be retargeted to the Humanoid character correctly. This problem is resolved in Unity 2022.3.11 and Unity 2023.1.17f1.
So it's "forced Unity update" time, from 2022.3.6f1 to the currently available 2022.3.19f1.
And there's a Quest firmware update. And the controllers batteries are out. And the hand tracking doesn't work in Link anymore. Reboot. Oh, now only one hand works. Let's try on the notebook...
...looks like it's one of those days!
After a bit of struggling, I managed to get my system working as expected and tested the samples provided by the `Unity-Movement
` package.
They're interesting, even if I'm not sure I have properly configured the samples, as the "legs guessing" is even worse than I expected.
I don't notice differences in the quality of the upper body tracking, but it's hard to say from a quick test. Tomorrow I'm going to dig deeper and check the sources of both samples.
Here's a video of me testing one of the new samples.
Sorry for the glitch on the right! It was maybe related to the fact that I captured while using Oculus Link, while I usually capture from a build running on device. Or maybe it's because I updated Unity (sigh).
2024-02-13 - Integrating body tracking
I went and enabled body tracking in my game scene, following the directions provided by the documentation.
Basically, it was about setting a few options in `OVRManager
`.
A notable thing is the addition of three layers to the project setup:
3: HiddenMesh
10: Character
11: MirroredCharacter
The names should provide a strong hint of their usage in the scene rendering.
And of course, there's a little details bringing a bit of uncertainty, with the documentation that contradicts itself.
In the configuration steps, it says:
Under General, make sure “Body Tracking” is supported. Click General if that view isn’t showing.
- If you want to use IOBT, please select “High” for “Body Tracking Fidelity” in the “Movement Tracking” section.
- If you want to use Full Body, please select “Full Body” for the “Body Tracking Joint Set” in the “Movement Tracking” section.
I want IOBT (Inside Out Body Tracking), upper body only and no "Generative Legs", so that's what I set.
But then, the documentation tells to use the Project Setup Tool to fix any outstanding issues, and this is what I get:
I will ignore it for now, leaving the joint set set to `Upper Body`.
After a while, I decided to try something a little hacky and, instead of modifying my setup taking the sample scene as reference, node by node, I tried to basically merge the two scenes, taking just what I needed from the sample.
After quite a bit of rewiring nodes and testing (the kind of thing that remembers me why I hate Unity), I got what I wanted: the game working as usual, but with the body tracking and the character model borrowed from the Meta samples.
I'm still far from done, of course. Let's make a quick TODO list:
fix/remove the shadows
disable the legs
have some debug visualization (mirrored avatar, bones highlights)
work on the avatar visuals
restore the health/mana indicators
2024-02-14 - Fixing some avatar details
As anything related to skinned meshes animation, body tracking details are a bit complicated.
On one side, I want to have an idea of what is going on behind the scenes of the simple integration I did.
On the other, I want to proceed with prototyping without spending too much time on technical details that might become irrelevant at some point (e.g. if I decide to ditch Unity).
So, I'm going to do some debatable, low quality hacks, and I'm going to add in my "long term" TODO list some notes about doing better at some point.
First, I'm going to hide the legs.
The body is a single skinned mesh, but with three materials rendering, respectively, body, arms and legs.
So, as a quick and dirty fix, I can change the legs material so that it uses a shader that that doesn't draw anything:
Shader "BinaryCharm/Null" {
SubShader {
Tags {"Queue" = "Geometry" }
Lighting Off
Pass {
ZWrite Off
ColorMask 0
}
}
}
That's not great because the legs bone transforms will still be updated, even if not really needed, but it took 5 minutes.
Disabling the shadow was also trivial.
What else from the TODO list?
Restoring the health/mana indicators, that should be attached to the back of the hands.
This was a bit more complicated: I had to change the way I fetched the wrists poses, which didn't work with the new scene setup.
To access the nodes without having to deal with the quite sophisticated retargeting that manages the skinned mesh, at least for now, I opted for the lower level access shown by the `DebugBodyJoints
` demo, using `OVRBody
`, which worked pretty easily.
I also took from that demo the "debug skeleton" visualization, which will definitely come useful.
What else? I want something more... particular for the avatar.
I tried doing another scene mash-up, applying to the player skinned mesh the vfx I tried on day 14 of PoC prototyping.
Unfortunately, I failed miserably.
Sometimes you can't just hack things around and get results, without understanding all the behind the scenes details. That's one of those times!
Just to avoid misunderstanding: I'm just talking about prototyping, when a quick result is more valuable that a clean and totally "under control" implementation... I would never ship stuff blindly taken from sample scenes.
I'll try to approach the problem differently tomorrow.
2024-02-15 - Busy!
Unexpectedly, today something came up, and I can't work on the project. I'll try to make up for it on Saturday.
2024-02-16 - Skinned mesh sampling
While tuning the energy bars offset, I realized they don't stick perfectly to the hands as with the previous method. It looks like there's no escape from studying the details of the body input data processing.
But first, I want to do more testing about the skinned mesh sampling I'd like to use to spawn particles along the player avatar.
I found another example on the Unity forums, a bit more basic but that should get me started with the kind of look I want.
By working with it, I realized that a key problem was that the character mesh didn't have the read/write option checked in its import settings, which was required for sampling the mesh.
I played a bit with some settings and got the first basic results.
Then, I disabled the default, semi-transparent hands that I've had in the scene since the beginning. Unless debug mode is active, where they might still come useful.
I also used the same "null shader" to the rest of the robotic avatar (as I already did with the legs).
Here's a little video update with the current state:
The "hidden legs" trick didn't last for long: they might not be rendered, but they are there and get sampled, so there's particles spawned on the legs too.
I need to fix that before proceeding.
2024-02-17 - Hiding the legs (again)
I went back and forth between the SDK documentation and the scripts attached to the character rig, trying to gain a better understanding of the complex processing that, from the input data, animates the skinned mesh of the character.
My idea was altering the process to scale the legs bones to a very small value (zero usually best avoided), which should work as a quick fix to hide the legs.
After a couple of hours, I was starting to get a big frustrated, because I know that this is all temporary: eventually, I will have a custom character mesh, made by a 3D artist, and properly rigged to just use the upper body input.
So, I tried yet another dirty hack: I added this in my `PlayerAvatarBhv
` script:
[SerializeField] private Transform m_rLeftLegTr;
[SerializeField] private Transform m_rRightLegTr;
private void LateUpdate() {
m_rLeftLegTr.localScale = 0.01f * Vector3.one;
m_rRightLegTr.localScale = 0.01f * Vector3.one;
}
...and linked the transform to the root bones of the legs, in the hierarchy that is used by the skinned mesh renderer:
It worked! That's what I should have been doing since the beginning.
I need to periodically remind myself that I'm just prototyping and, especially for things related to the temporary visualization layer and Unity, I can be a little dirty.
You can see me pointing at the "micro-legs" at the end of the video.
Time to wrap up this little Saturday extra - am I still in time for `#screenshotSaturday
`?