2024-02-26 - Body pose data
Let's start with a brief recap.
During week #14, I introduced a basic player avatar with the core characteristics I want for the game:
upper body tracking
particle based VFX
I mostly took and adapted some elements from sample projects, writing very little code.
Then, I stopped for a moment and dived into the Meta SDK to gain a better understanding of how what the body tracking data is, and how it flows.
I should be now ready to make better use of the body tracking and to properly integrate it into my systems.
As first step, I'm going to store the body tracking data into the game state.
That would not be needed for normal gameplay, but I want to
have the body pose in the savegame, so that you can see a preview of the saved game state, including the avatar
have a layer where I can process/alter the tracking data, before using it to drive the avatar skinned mesh
be able to rewind, animating an "out of body" character by saved tracking data
get ready to record animation clips of body tracking data (they will come useful to prototype enemy attacks)
From my analysis, I know that I essentially have to save the poses of the upper body joints.
So, I'm going to insert into my `PlayerAvatar
` struct (part of the game state), an array of `PosRot
`, my (more serialization friendly) alternative to `Pose
` that I already discussed in the past.
I must be careful to not save poses in world space, because what I actually want are poses relative to the platform the player is standing on.
Let's see what I manage to do.
Saving the data was easy, but now I need some way to visualize it.
The SDK script I already used for the debug display of the body skeleton, `BodyDebugGizmos
`, should be a good start to implement a test visualization.
`BodyDebugGizmos
` conveniently inherited from `SkeletonDebugGizmos
`, which took care of the actual debug drawing. So I just had to do something similar, with the difference that my child class, `PlayerAvatarDebugGizmos
`, does not draw data coming from a `Body
` (like `BodyDebugGizmos
` does) but from a `PlayerAvatar.Data
` (containing "my version" of the body tracking data).
using UnityEngine;
using Oculus.Interaction.Body;
using Oculus.Interaction.Body.Input;
using BinaryCharm.ParticularReality.StateManagement;
namespace BinaryCharm.ParticularReality.DebugManagement {
public class PlayerAvatarDebugGizmos : SkeletonDebugGizmos {
public enum CoordSpace {
World,
Local,
}
[Tooltip("The coordinate space in which to draw the skeleton. " +
"World space draws the skeleton at the world Body location. " +
"Local draws the skeleton relative to this transform.")]
[SerializeField]
private CoordSpace _space = CoordSpace.World;
public CoordSpace Space {
get => _space;
set => _space = value;
}
private VisibilityFlags GetModifiedDrawFlags() {
VisibilityFlags modifiedFlags = Visibility;
if (HasNegativeScale && Space == CoordSpace.Local) {
modifiedFlags &= ~VisibilityFlags.Axes;
}
return modifiedFlags;
}
protected override bool TryGetParentJointId(
BodyJointId jointId,
out BodyJointId parent) {
if (m_rSkeletonMapping == null) {
parent = BodyJointId.Invalid;
return false;
}
bool ret =
m_rSkeletonMapping.TryGetParentJointId(jointId, out parent);
return ret;
}
protected override bool TryGetWorldJointPose(
BodyJointId jointId,
out Pose pose) {
if (m_rAvatarData == null) {
pose = new Pose();
return false;
}
bool result;
switch (_space) {
default:
case CoordSpace.World:
result = m_rAvatarData.getJointPose(jointId, out pose);
break;
case CoordSpace.Local:
result =
m_rAvatarData.getJointPoseFromRoot(jointId, out pose);
pose.position = transform.TransformPoint(pose.position);
pose.rotation = transform.rotation * pose.rotation;
break;
}
return result;
}
private PlayerAvatar.Data m_rAvatarData;
private ISkeletonMapping m_rSkeletonMapping;
public void setData(
PlayerAvatar.Data rAvatarData,
ISkeletonMapping skeletonMapping) {
m_rAvatarData = rAvatarData;
m_rSkeletonMapping = skeletonMapping;
}
public void clearData() {
m_rAvatarData = null;
m_rSkeletonMapping = null;
}
private void Update() {
if (m_rAvatarData == null || m_rSkeletonMapping == null) return;
foreach (BodyJointId joint in m_rSkeletonMapping.Joints) {
Draw(joint, GetModifiedDrawFlags());
}
}
}
}
I made it so I can manually set a `PlayerAvatar.Data
` and a `ISkeletonMapping
`, and the script keeps drawing that configuration until either it gets new data or is "cleared".
Then, I added another small test script which simply sets the body/skeleton data when I press the `B
` key (you can spot me using the keyboard with my left hand in the test video below).
That's it for today!
2024-02-27 - App, Game, Sim
I wrote that I want to be able to save body animation clips. But I'd also like to have a fully working state rewind.
I'm going to start implementing a data structure that keeps in memory stores a certain number of the latest game states.
I did a bit of refactoring and added a new layer between `App.Data
` and `Game.Data
`.
Basically, what was previously the `Game.Data
` is now `Sim.Data
` (as in "simulation"), and the `Game.Data` now contains a bunch of `Sim.Data
`.
Let's say we want 10 seconds of state available for rewind: at 120 Hz, which would be ideal, that would be 1200 `Sim.Data
` entries. If needed, at some point, I'm going to lower the state saving frequency and interpolate.
Of course, we will use a ring buffer, so we're going to constantly overwrite the same entries, with no allocations.
It's mandatory to do this kind of thing that we are able to do a deep copy of the state: at every simulation step, we are going to copy the previous state to the next slot in the circular buffer, before updating it by running the simulation.
Thanks to how I'm organizing things, this should be pretty easy to enforce: I will request that every data type which is part of the game state implements a `ICopyFrom<T>
` interface, similar to what was done in the SDK code I analysed last week.
Ok, I added a bunch of simple `CopyFrom
` methods that fill in the values copying from another instance, taking care of actually copying values and not references when needed.
I'm out of time, but tomorrow (actually: the day after, because tomorrow I'm busy) I should be able to have this simple "state history" running.
2024-02-29 - Visualizing Body History
Today I'd like to build a test visualization of the "body state history", meaning: a debug visualization for the previous simulation states held in the `Game.Data
`.
First, I need to complete the refactoring I started on Tuesday: I did some ground work, but the history itself doesn't get updated yet.
Making the changes took a while, and the new execution logic highlighted a problem in my current FSM implementation, about when the input signals are processed.
I added a workaround, but at some point a proper fix is going to be needed.
Anyway, this is the core part of the execution: the previous execution state gets copied to the next slot in the buffer, and the simulation is carried forward. The pointer to the current execution state gets updated.
public override void update(float fDT) {
base.update(fDT);
int iCurr = getFSMData().m_iCurrentSimIdx;
int iNext = (iCurr + 1) % m_rSimStateManager.Length;
rSimStates[iNext].CopyFrom(rSimStates[iCurr]);
m_rSimStateManager[iNext].update(fDT);
getFSMData().m_iCurrentSimIdx = iNext;
}
Now I can add a bit of test code to visualize (some of) the body tracking data from previous states.
I instantiated a bunch of `PlayerAvatarDebugGizmos
` and pointed them at data from a certain offset in the past.
Well, something went wrong at some point, but I'm almost there! I'll fix it tomorrow - meanwhile, let's have a laugh:
2024-03-01 - Visualization fix, planning
Let's start by fixing the buggy history visualization from yesterday.
Ok, I changed the sampling and update logic, and some parameters of the visualization, and now it's something closer to what I was imagining:
The code is short and simple enough to show:
//[...]
private PlayerAvatarDebugGizmos[] m_rPrevBodyFrames;
private const int iSTEP = 10;
private const int iTRAIL_SIZE = Game.iSIM_HISTORY_SIZE / iSTEP;
private void Start() {
m_rPrevBodyFrames = new PlayerAvatarDebugGizmos[iTRAIL_SIZE];
for (int i = 0; i < iTRAIL_SIZE; ++i) {
GameObject rNew = new GameObject("sim_" + i.ToString("000"));
rNew.setParentAndResetLocalTransform(gameObject.transform);
m_rPrevBodyFrames[i] =
rNew.AddComponent<PlayerAvatarDebugGizmos>();
m_rPrevBodyFrames[i].Visibility =
SkeletonDebugGizmos.VisibilityFlags.Bones;
}
}
private void Update() {
GameStateManager rGSM = AppStateManagerBhv.getGameStateManager();
if ((rGSM.getCurrSimIdx() % iSTEP) == 0) {
for (int i = 0; i < iTRAIL_SIZE; ++i) {
Sim.Data rSimData =
rGSM.getPrevSimStateManager(i * iSTEP);
PlayerAvatar.Data rPAD =
rSimData.m_rPlayerAvatarFSMEx.getData();
m_rPrevBodyFrames[i].setData(
rPAD,
BodyInputManagerBhv.i().getSkeletonMapping()
);
float fDarkness = 1f - (i / (float)iTRAIL_SIZE);
m_rPrevBodyFrames[i].BoneColor =
new Color(fDarkness, fDarkness, fDarkness, 1f);
}
}
}
//[...]
At this point, I have many somewhat correlated tasks to do.
I must drive the character skinned mesh by my version of the body tracking data, in the game state (it's still using the "source" data, coming straight from the Meta samples)
I need to establish a data format for an animation clip of body tracking data, which I could extract from the "sim state history" or save separately with some start/stop recording commands. It must be easy to adjust the clip (cutting some frames, for example)
I must have a system to playback a saved clip on a skinned mesh
It might be worth considering binary serialization and some packing, now that I'm saving and loading a significant amount of data
I need to implement rewind, and I need to have save, load, pause and rewind interact in a sensible and robust way.
Next week I'm unfortunately busy, but right after that, I'm going to attack these tasks one by one.