Before proceeding with implementing new features, I wanted to gain a deeper understanding of the part of the Meta SDK that takes care of the body tracking data and its processing.
Otherwise, I risked reimplementing things that are already available, and potentially doing a worse job at it.
Additionally, I had a little less time to work on the project this week, so I decided to not do a "development week" entry. Instead, here's a new "extra" post collecting and organizing the notes I took while analysing the SDK code.
I'm going to present elements in a bottom up fashion, so that (if possible) when talking about something, its dependencies have already been discussed earlier, incrementally building deeper and deeper knowledge.
We are going to discuss part of the Meta XR Interaction SDK (v60), and specifically:
the whole `
Oculus.Interaction.Body.Input
` namespacea few elements in `
Oculus.Interaction.Input
``
ICopyFrom<in TSelfType>
` (defined in `HandPrimitives.cs
`)`
IDataSource
`, `IDataSource<TData>
` (defined in `DataSource.cs
`)`
DataSource<TData>
``
DataModifier<TData>
`
There were other things that I wanted to cover, but I ran out of time.
Which things?
the `
Oculus.Interaction.Body.PoseDetection
` namespacethe part of the Movement SDK about body tracking
Maybe in the next "extra" post!
Enough chit-chat, let's start the analysis from `Oculus.Interaction.Input
`, which contains some lower level building blocks used by `Oculus.Interaction.Body.Input
`.
`Oculus.Interaction.Input
`: interfaces
These three simple interfaces are our starting point.
`
interface ICopyFrom<in TSelfType>
`consists of a `
CopyFrom<TData>(TData source)
` method which fills the object by doing a deep copy from `source
`
`
interface IDataSource
`looks like a simple two ways signalling mechanism with a method to mark "update required" and an event to signal data input data is available (additionally, there's a data version value, a counter incremented at every update)
`
interface IDataSource<TData> : IDataSource
`generic interface extending `
IDataSource
` with the addition of a `TData GetData()
` method which allows to fetch a data item
`Oculus.Interaction.Input
`: `DataSource
` and `DataModifier
`
Two abstract classes are relevant to our analysis:
`
DataSource<TData>
``
DataModifier<TData>
`
Let's see some details of both.
`DataSource`
abstract class DataSource<TData> : MonoBehaviour, IDataSource<TData>
where TData : class, ICopyFrom<TData>, new()
`DataSource
` is an abstract class inheriting from `MonoBehaviour
` (so, that can be attached to a scene node) and that implements the `IDataSource<TData>
` interface, for a generic `TData
`, which must be a class with a public parameterless constructor and a `CopyFrom<TData>
` method.
The two abstract elements, left to be implemented in concrete child classes are
`
protected abstract void UpdateData()
``
protected abstract TData DataAsset { get; }
`Returns the current DataAsset, without performing any updates.
The implementation of `GetData()
` is worth looking into:
public TData GetData()
{
if (RequiresUpdate())
{
UpdateData();
_requiresUpdate = false;
}
return DataAsset;
}
So, it's when `GetData()
` gets called that (if needed) the data item gets updated (through the call to `UpdateData()
`). `RequiresUpdate()
` simply returns the value of the `_requiresUpdate
` boolean flag, which gets reset after a data update. But when does it get set to `true
`? In two cases
in `
MarkInputDataRequiresUpdate()
`, which also increments `_currentDataVersion
` and raises the `InputDataAvailable
` event , which is documented as "Notifies that new data is available for query via GetData() method."in turn, `
MarkInputDataRequiresUpdate()
` gets called depending on the configured `UpdateModeFlags
`: `Manual
`, `UnityUpdate
`, `UnityFixedUpdate
`, `UnityLateUpdate
`, `AfterPreviousStep
`
in `
ResetUpdateAfter(IDataSource updateAfter, UpdateModeFlags updateMode)
`, which I only see called in `DataModifier.ResetSources()
`
Summarizing: `DataSource
` looks like an interesting building block which takes care of fetching and processing some input data to be provided to callers, but in a smart way: the fetching and processing only happens when needed (because there's new data available, and because the data has been requested by a caller).
`DataModifier
`
abstract class DataModifier<TData> : DataSource<TData>
where TData : class, ICopyFrom<TData>, new() { ... }
`DataModifier
` is another abstract class, that builds upon `DataSource
`: in this case, there's the possibility to do some optional data processing on the data fetched from the wrapped `IDataSource
`
the original data asset is stored in `
_thisDataAsset
`, while the modified asset, returned through the `DataAsset
` property, is stored in `_currentDataAsset
``
UpdateData
` is overridden to call the modification method, but only if the `_applyModifier
` flag is set - otherwise, the original data asset is passed throughThe abstract method doing the modification is defined like this:
`
protected abstract void Apply(TData data);
`
Having described these foundational elements, we can move to the `Oculus.Interaction.Body.Input` namespace and see the role of everything we have discussed so far.
`Oculus.Interaction.Body.Input
`
The `Oculus.Interaction.Body.Input
` namespace contains:
`
BodyJointId
`, `ReadOnlyBodyJointPoses
` (in `BodyPrimitives.cs
`)`
ISkeletonMapping
``
IBody
``
BodyDataAsset
``
BodyJointsCache
``
Body
`
Let's discuss them in the more sensible order I could find.
Body Primitives
The `BodyPrimitives.cs
` file contains three simple definitions:
`
BodyJointId
` enumeration`
Constants
` static class`
ReadOnlyBodyJointPoses
` class
The `BodyJointId
` enumeration identifies the human upper body skeleton joints handled by this part of the SDK:
public enum BodyJointId
{
Invalid = -1,
[InspectorName("Body Start")]
Body_Start = 0,
[InspectorName("Root")]
Body_Root = Body_Start + 0,
[InspectorName("Hips")]
Body_Hips = Body_Start + 1,
//...
[InspectorName("Right Hand/Right Hand Little Distal")]
Body_RightHandLittleDistal = Body_Start + 68,
[InspectorName("Right Hand/Right Hand Little Tip")]
Body_RightHandLittleTip = Body_Start + 69,
[InspectorName("Body End")]
Body_End = Body_Start + 70,
}
Some notes about the enumeration:
`
Invalid
` is obviously used to indicate an invalid/empty value for the enumeration`
Body_Start
` and `Body_End
` are going to be used to iterate on the identifiers, and do not "really" identify body joints like all the others (except `Invalid
`)The first joint, `
Body_Root
`, is mapped to the same `0
` value also assigned to `Body_Start
`. I get why they did it, but it makes me slightly nervous and I think it's the first time I see it in "real code".
The `Constants` static class contains only a single entry indicating the number of joints, inferred by the `BodyJointId.Body_End`value:
public static class Constants
{
public const int NUM_BODY_JOINTS = (int)BodyJointId.Body_End;
}
Finally, `ReadOnlyBodyJointPoses
` is basically an enumerable wrapper over a read-only `Pose
` array:
class ReadOnlyBodyJointPoses : IReadOnlyList<Pose>
`ISkeletonMapping
` interface
This interface is about providing two essential pieces of information:
`
IEnumerableHashSet<BodyJointId> Joints { get; }
`fetches the set of joint identifiers supported by a skeleton
`
bool TryGetParentJointId(BodyJointId jointId, out BodyJointId parent)
`fetches the parent joint identifier for `
jointId
`, provided there is one
`BodyDataAsset` class
The `BodyDataAsset
` class is just a few lines, but is very important, because it basically stores a packet of "body tracking data", in a pretty raw form, and implements the `CopyFrom
` "deep copy" method (as required by `ICopyFrom
`).
class BodyDataAsset : ICopyFrom<BodyDataAsset>
Its properties are:
public ISkeletonMapping SkeletonMapping { get; set; }
public Pose Root { get; set; } = Pose.identity;
public float RootScale { get; set; } = 1f;
public bool IsDataValid { get; set; } = false;
public bool IsDataHighConfidence { get; set; } = false;
public Pose[] JointPoses { get; set; } = new Pose[Constants.NUM_BODY_JOINTS];
public int SkeletonChangedCount { get; set; } = 0;
Notice the `JointPoses
` array, to store a full set of joints poses.
`IBody
` interface
This `IBody
` interface is super interesting and well documented in the `IBody.cs
` source file.
I'm going to provide a shorter comment on its members
`
ISkeletonMapping SkeletonMapping
`the skeleton information associated with the body
`
bool IsConnected
`is the body connected and tracked?
`
bool IsHighConfidence
`is the tracking data marked as high confidence? Implies `
IsConnected == true
`
`
bool IsTrackedDataValid
`are tracking poses available for body root and joints?
I'm not sure about in which cases `
IsConnected
` might be `true
` but `IsTrackedDataValid
` would be `false
`, as the comment on `IsConnected
` explicitly says "connected and tracked"still, this looks like the value to check before trying to get joint poses
`
float Scale
`scale of the body skeleton applied to world joint poses
`
bool GetRootPose(out Pose pose)
`if available, root pose of the body in world space
`
bool GetJointPose(BodyJointId bodyJointId, out Pose pose)
`if available, pose of the requested body joint in world space, scale applied
`
bool GetJointPoseLocal(BodyJointId bodyJointId, out Pose pose)
`if available, pose of the requested body joint in local space, relative to its parent joint, scale not applied
`
bool GetJointPoseFromRoot(BodyJointId bodyJointId, out Pose pose)
`if available, pose of the requested body joint relative to the root joint, scale not applied
`
int CurrentDataVersion
`incremented every time data changes
`
event Action WhenBodyUpdated
`raised each time the body is updated with new data
Summarizing, the `IBody
` interface allows us to...
fetch the set of valid joints, and their structure (in terms of parent-child relationships)
through `
SkeletonMapping
`
check if there's body data available in a certain instant
through `
IsConnected
`, `IsHighConfidence
` and `IsTrackedDataValid
`
check if there's "new" data available
through `
CurrentDataVersion
` (in case of polling) and `WhenBodyUpdated
` (in case of event handling)
fetch the poses of the skeleton joints, with a certain degree of flexibility
through the `
Scale
` property, `GetRootPose()
` and all the `GetJointPose*
` methods
`BodyJointsCache
` class
This class has a pretty descriptive name: a cache for some body joints poses.
By glancing at the implementation, I can guess that the objective of this class is minimizing computation of derived data so that it can be accessed efficiently in different contexts.
Let's see more in detail what does the code do, starting from the the member data:
private Pose[] _originalPoses = new Pose[Constants.NUM_BODY_JOINTS];
private Pose[] _posesFromRoot = new Pose[Constants.NUM_BODY_JOINTS];
private Pose[] _localPoses = new Pose[Constants.NUM_BODY_JOINTS];
private Pose[] _worldPoses = new Pose[Constants.NUM_BODY_JOINTS];
private ReadOnlyBodyJointPoses _posesFromRootCollection;
private ReadOnlyBodyJointPoses _worldPosesCollection;
private ReadOnlyBodyJointPoses _localPosesCollection;
private ulong[] _dirtyJointsFromRoot;
private ulong[] _dirtyLocalJoints;
private ulong[] _dirtyWorldJoints;
private Matrix4x4 _scale;
private Pose _rootPose;
private Pose _worldRoot;
private ISkeletonMapping _mapping;
Basically, it looks like we have three series of joint poses and data related to them:
from root
world
local
...and some other "global" data:
scale
root pose
world root pose
So how does the caching work? Let's start from the constructor, and notice some peculiar allocations:
_dirtyJointsFromRoot = new ulong[DIRTY_ARRAY_SIZE];
_dirtyLocalJoints = new ulong[DIRTY_ARRAY_SIZE];
_dirtyWorldJoints = new ulong[DIRTY_ARRAY_SIZE];
`DIRTY_ARRAY_SIZE` is defined like this
private const int DIRTY_ARRAY_SIZE = 1 + (Constants.NUM_BODY_JOINTS / 64);
If you're familiar with this kind of trick, you will have understood already that these `ulong
` arrays are used as "bitfields": an `ulong
` is 64 bits, so this little calculation takes care of finding out how many `ulong
` are needed to have a bit flag for each joint.
Right now, with `Constants.NUM_BODY_JOINTS
` amounting to 70, we need 2 `ulong
` (which would be enough for up to 128 joints).
I guess that, to need more than that, it would mean that we have reached bare-foot VR with toes tracking... some future proofing for this class, definitely (some would say a bit overkill)!
Additionally, the constructor initialized the `ReadOnlyBodyJointPoses
` with references to the respective arrays:
_localPosesCollection = new ReadOnlyBodyJointPoses(_localPoses);
_worldPosesCollection = new ReadOnlyBodyJointPoses(_worldPoses);
_posesFromRootCollection = new ReadOnlyBodyJointPoses(_posesFromRoot);
So, that was the constructor, but what about the methods?
All the following `Get
` methods call a private "update" method for the requested data and then provide it to the caller:
`
bool GetAllLocalPoses(out ReadOnlyBodyJointPoses localJointPoses)
``
void UpdateAllLocalPoses()
`
`
bool GetAllPosesFromRoot(out ReadOnlyBodyJointPoses posesFromRoot)
``
void UpdateAllPosesFromRoot()
`
`
bool GetAllWorldPoses(out ReadOnlyBodyJointPoses worldJointPoses)
``
void UpdateAllWorldPoses()
`
`
Pose GetLocalJointPose(BodyJointId jointId)
``
void UpdateLocalJointPose(BodyJointId jointId)
`
`
Pose GetJointPoseFromRoot(BodyJointId jointId)
``
void UpdateJointPoseFromRoot(BodyJointId jointId)
`
`
Pose GetWorldJointPose(BodyJointId jointId)
``
void UpdateWorldJointPose(BodyJointId jointId)
`
The "world root pose" doesn't have an update method, so its method just returns `_worldRoot
`
`
Pose GetWorldRootPose()
`
There are two private methods dealing with the bitfields arrays, to check if a specific bit is set, or to reset it, hiding the bit mangling behind a simple interface:
`
bool CheckJointDirty(BodyJointId jointId, ulong[] dirtyFlags)
``
void SetJointClean(BodyJointId jointId, ulong[] dirtyFlags)
`
And finally... the meaty `Update
` method, in the public API:
`
void Update(BodyDataAsset data, int dataVersion, Transform trackingSpace = null)
`
What does `Update
` do?
it sets all the bits of the flags array to 1 (dirty: there's new data and the cache needs to be updated)
if `
data
` contains valid body data, it executes this code:
_scale = Matrix4x4.Scale(Vector3.one * data.RootScale);
_rootPose = data.Root;
_worldRoot = _rootPose;
if (trackingSpace != null)
{
_scale *= Matrix4x4.Scale(trackingSpace.lossyScale);
_worldRoot.position = trackingSpace.TransformPoint(_rootPose.position);
_worldRoot.rotation = trackingSpace.rotation * _rootPose.rotation;
}
for (int i = 0; i < Constants.NUM_BODY_JOINTS; ++i)
{
_originalPoses[i] = data.JointPoses[i];
}
This consists basically of three steps:
updates the `
_scale
`, `_rootPose
` and `_worldRoot
`members copying from `data
`if a `
trackingSpace
` transform is set, it adjustes `_scale
` and recalculates the `_worldRoot
` pose by considering such transformcopies the joint poses contained in `
data
` to the local `_originalPoses
` array
The last thing to discuss to complete the analysis of `BodyJointsCache
` are the private `Update*
` methods that, as we anticipated, are called by the `Get*
` methods before returning their results.
`UpdateAllWorldPoses()
`, `UpdateAllLocalPoses()
` and `UpdateAllPosesFromRoot()
` are not very interesting: they just call on every joint, respectively, `UpdateWorldJointPose(joint)
`, `UpdateLocalJointPose(joint)
` and `UpdateJointPoseFromRoot(joint)
`.
So, these three methods are where the meaningful calculations must be. Let's check them one by one, starting from the simpler.
private void UpdateJointPoseFromRoot(BodyJointId jointId)
{
if (!CheckJointDirty(jointId, _dirtyJointsFromRoot))
{
return;
}
Pose originalPose = _originalPoses[(int)jointId];
Vector3 posFromRoot = Quaternion.Inverse(_rootPose.rotation) *
(originalPose.position - _rootPose.position);
Quaternion rotFromRoot = Quaternion.Inverse(_rootPose.rotation) *
originalPose.rotation;
_posesFromRoot[(int)jointId] = new Pose(posFromRoot, rotFromRoot);
SetJointClean(jointId, _dirtyJointsFromRoot);
}
First, the dirty flag for the joint id is checked. If it is not set, there's no work to do: the `_posesFromRoot[(int)jointId]
` value is already up to date.
Otherwise, that pose is recalculated, starting from `_originalPoses[(int)jointId]
`.
As we have already discussed, `_originalPoses
` gets updated with the `BodyDataAsset
` data given as input to the main `Update
` method. So, that's our starting point, together with the `_rootPose
` pose (also set in the same `Update
` method).
Pose originalPose = _originalPoses[(int)jointId];
Vector3 posFromRoot = Quaternion.Inverse(_rootPose.rotation) *
(originalPose.position - _rootPose.position);
Quaternion rotFromRoot = Quaternion.Inverse(_rootPose.rotation) *
originalPose.rotation;
_posesFromRoot[(int)jointId] = new Pose(posFromRoot, rotFromRoot);
In case you are not familiar with these kind of formulas, it's worth explaining more in detail:
`originalPose.position - _rootPose.position
` is an offset vector starting from `_rootPose.position
` and terminating in `originalPose.position
` (imagine a segment connecting in 3D space the root point of the skeleton and the joint we're updating )`
Quaternion.Inverse(_rootPose.rotation)
` is the inverse rotation of `_rootPose
`, so basically the rotation that "cancels out" the rotation of the root pose in world space
The "poseFromRoot" we are calculating is basically an offset (both in position and rotation) that can be applied to the pose of the root element to reconstruct the joint pose.
The rotation offset is:
Quaternion rotFromRoot = rootPoseInverseRotation * originalPose.rotation;
Multiplying two rotation quaternions is like composing two rotations one after the other.
So, multiplying for an inverse rotation quaternion is like "removing" that rotation instead of adding it.
In this specific case, the base rotation of the root is "canceled" from the "total" world space rotation of the joint, leaving in `rotFromRoot
` only the rotation offset from root to joint.
The position offset calculation follows a similar approach:
Vector3 posFromRoot = rootPoseInverseRotation * offsetFromRootToJoint;
When you multiply a vector by a rotation quaternion, you rotate the vector. So multiplying by the inverse rotation of the root "removes" the base rotation of the root itself, leaving us with a clean relative position offset from root to joint.
Now, let's see `UpdateWorldJointPose()
`:
private void UpdateWorldJointPose(BodyJointId jointId)
{
if (!CheckJointDirty(jointId, _dirtyWorldJoints))
{
return;
}
Pose fromRoot = GetJointPoseFromRoot(jointId);
fromRoot.position = _scale * fromRoot.position;
fromRoot.Postmultiply(GetWorldRootPose());
_worldPoses[(int)jointId] = fromRoot;
SetJointClean(jointId, _dirtyWorldJoints);
}
This method builds upon the already discussed `GetJointPoseFromRoot()
`, used to calculate the offset from the root joint. Then, such "offset pose" is composed with the root pose, obtaining the world space pose.
If the calculations look a bit unfamiliar, is because they make use of a couple of utility methods hosted in `PoseUtils.cs
`:
public static void Postmultiply(this ref Pose a, in Pose b)
{
Multiply(b, a, ref a);
}
public static void Multiply(in Pose a, in Pose b, ref Pose result)
{
result.position = a.position + a.rotation * b.position;
result.rotation = a.rotation * b.rotation;
}
Finally, let's see the slightly more complicated `UpdateLocalJointPose()
` method.
private void UpdateLocalJointPose(BodyJointId jointId)
{
if (!CheckJointDirty(jointId, _dirtyLocalJoints))
{
return;
}
if (_mapping.TryGetParentJointId(jointId, out BodyJointId parentId) &&
parentId >= BodyJointId.Body_Root)
{
Pose originalPose = _originalPoses[(int)jointId];
Pose parentPose = _originalPoses[(int)parentId];
Vector3 localPos = Quaternion.Inverse(parentPose.rotation) *
(originalPose.position - parentPose.position);
Quaternion localRot = Quaternion.Inverse(parentPose.rotation) *
originalPose.rotation;
_localPoses[(int)jointId] = new Pose(localPos, localRot);
}
else
{
_localPoses[(int)jointId] = Pose.identity;
}
SetJointClean(jointId, _dirtyLocalJoints);
}
In this case, the case where a joint might not have a parent is handled differently, returning `Pose.identity
`. The parent is looked up thanks to the `_mapping
` member, set in `Update()
`, describing the skeleton structure.
Considering that the skeleton structure doesn't normally change, and that almost every joint has a parent, I feel like there could be a lot of saved dictionary lookups, but they probably decided to keep it simple and generic.
Anyway, when there's a parent, the local pose is calculated, with exactly the same approach used in `UpdateJointPoseFromRoot()
`, but using the parent pose instead of the root pose (naturally: we're calculating an offset from the parent joint and not from the root joint).
This concludes our tour of `BodyJointsCache
`, which was a bit more intimidating at a glance, but not doing anything particularly fancy after closer inspection: the cache calculates once, starting from "original" poses passed in as input, some derived data for each joint: its pose in world space, its offset from the parent joint and its offset from the root joint. A "dirty flag" mechanism is used to only do computation if the original poses have changed since the last computation. Moreover, computations are done in a "lazy" manner - only if somebody calls a `Get
` method requiring that data (directly or indirectly).
`Body
` class
With `Body`, the pieces of the puzzle finally come together:
public class Body : DataModifier<BodyDataAsset>, IBody
So, we know it's `DataModifier
` (a `DataSource
` which also does some data processing, as previously discussed), and that the type of data item it processes is `BodyDataAsset
`. All this, implementing the `IBody
` interface.
The SDK authors did a good job of handling the complexity of the system through the architecture we described, and the class implementation is pretty simple. Let's discuss it briefly.
public bool IsConnected => GetData().IsDataValid;
public bool IsHighConfidence => GetData().IsDataHighConfidence;
public float Scale => GetData().RootScale;
public ISkeletonMapping SkeletonMapping => GetData().SkeletonMapping;
public bool IsTrackedDataValid => GetData().IsDataValid;
This part of the class implements part of the `IBody` interface through the `BodyDataAsset` implementation.
public event Action WhenBodyUpdated = delegate { };
An empty delegate initialized the `WhenBodyUpdated` event, so that it can be invoked.
private BodyJointsCache _jointPosesCache;
An instance of the `BodyJointsCache
` we have extensively discussed (which isolates another piece of complexity).
[SerializeField, Optional]
private Transform _trackingSpace;
This `Transform` field is to optionally be assigned through the inspector. The tooltip says:
If assigned, joint pose translations into world space will be performed via this transform. If unassigned, world joint poses will be returned in tracking space.
What about the methods?
In the private part of the API:
`
InitializeJointPosesCache()
`guess what, initializes `
_jointPosesCache
`
`
CheckJointPosesCacheUpdate()
`when there's new data (based on the "version" counter), calls the `
Update
` method of `BodyJointsCache
`, passing in the new `BodyDataAsset
` input data, its version, and the `_trackingSpace
` transform
In the public part, instead:
`
MarkInputDataRequiresUpdate()
` overrides the definition in `DataSource
` bymaking sure `
_jointPosesCache
` is initializedraising the `
WhenBodyUpdate
` event of the `IBody
` interface
`
bool GetRootPose(out Pose pose)
`if available, fetches the "world root" pose
`
bool GetJointPose(BodyJointId bodyJointId, out Pose pose)
`if available, fetches the joint pose in world space
`
bool GetJointPoseLocal(BodyJointId bodyJointId, out Pose pose)
`if available, fetches the joint pose in local space, relative to their parent joint pose
`
bool GetJointPoseFromRoot(BodyJointId bodyJointId, out Pose pose)
`if available, fetches the joint pose relative to the root pose
It's worth noting that `Apply
`, the abstract method inherited from `DataModifier
`, has an empty implementation:
protected override void Apply(BodyDataAsset data)
{
// Default implementation does nothing,
// to allow instantiation of this modifier directly
}
This might feel a little strange (if `Apply
` does nothing, why inherit from `DataModifier
` and not from `DataSource
`?), but it makes sense: it establishes the presence of an `Apply
` method in any subclass of `Body
`.
Summarizing, `Body
` is a data source of `BodyPoseData
` elements, with optional processing, and a pretty sophisticated caching that also offers some derived data. The mix of these elements is used to implement the `IBody
` interface.
The full picture (well, a diagram)
A class diagram should be an excellent way to recap our understanding.
This diagram is not perfect: I ran into a mermaid limitation about mixing inheritance and generic types. So, `IDataSource
` and `IDataSource<TData>
` are collapsed, and I had to do some extra annotation on some extends. Of course, the `...` indicate that I skipped something, which is crucial in my opinion to keep this kind of diagram useful: I only left in the most important information. And I always label explicitly the links, because I never remember the meaning behind each kind of arrow/line and I feel like I am not the only one.
Anyway, this concludes our analysis of the `Oculus.Interaction.Body.Input
` namespace.
We only skipped the `BodySkeletonMapping
` class, which does does not look directly referenced by other things.
Just one more thing…
We discussed how to access body tracking data through a `Body
`, which is a `DataSource
` of `BodyDataAsset
`. But from where does `Body
` itself fetch the data?
If we go back to `DataSource.cs
` for closer inspection, there's this part:
[SerializeField, Interface(typeof(IDataSource))]
[Optional(OptionalAttribute.Flag.DontHide)]
private UnityEngine.Object _updateAfter;
private IDataSource UpdateAfter;
private int _currentDataVersion;
In `DataModifier.cs
` there's a similar snippet:
[Header("Data Modifier")]
[SerializeField, Interface(nameof(_modifyDataFromSource))]
protected UnityEngine.Object _iModifyDataFromSourceMono;
private IDataSource<TData> _modifyDataFromSource;
So, `Body
` is a data source, but is itself taking data from another data source linked via inspector.
We can see that in the rig in the Unity scene `Body
` refers to the `FromOVRBodyDataSource
` behaviour on the `OVRBodyDataSource
` node.
`FromOVRBodyDataSource
` is a `DataSource<BodyDataAsset>
` too, but finally, if we look at its code, and specifically the implementation of the `UpdateData()`
method, we can see this is where the `BodyDataAsset
` originates, before starting its trip down the chain.
protected override void UpdateData()
{
var data = DataProvider.GetSkeletonPoseData();
if (!data.IsDataValid)
{
return;
}
//... here _bodyDataAsset is initialized on the basis of data
}
So the data comes from `DataProvider
`, in a different format (`OVRSkeleton.SkeletonPoseData
`) that gets converted to `BodyDataAsset
`.
And what's `DataProvider
`?
It is set through the inspector too, and it's the `OVRBody
` script attached to the same node, shown in the bottom right part of the image above.
`OVRBody
` is, indeed, defined as implementing the `OVRSkeletonRenderer.IOVRSkeletonDataProvider
` interface (defined in `OVRSkeleton.cs
`).
public interface IOVRSkeletonDataProvider
{
SkeletonType GetSkeletonType();
SkeletonPoseData GetSkeletonPoseData();
bool enabled { get; }
}
In the `GetSkeletonPoseData()
` implementation of `OVRBody
`, we can find yet another passage/transformation of data, with the creation of a `OVRSkeleton.SkeletonPoseData
` that gets filled in with the data coming from `_bodyState
` structure.
So what is this `_bodyState
` now, and where does it come from? Well:
private void GetBodyState(OVRPlugin.Step step)
{
if (OVRPlugin.GetBodyState4(step, _providedSkeletonType, ref _bodyState))
{
_hasData = true;
_dataChangedSinceLastQuery = true;
}
else
{
_hasData = false;
}
}
Finally, we reached the end of the line: a call into the `OVRPlugin
` class, in the `com.meta.xr.sdk.core
` package.
If you go and look for `GetBodyState4
` in `OVRPlugin.cs
`, you will find some other juicy details, like the handling of a different number of joints for upper body or full body tracking.
And of course, the call into the actual native library:
var result = OVRP_1_92_0.ovrp_GetBodyState4(stepId, -1, out var bodyState4Internal);
...and it's declaration:
[DllImport(pluginName, CallingConvention = CallingConvention.Cdecl)]
public static extern Result ovrp_GetBodyState4(
Step stepId, int frameIndex, out BodyState4Internal bodyState
);
If you want to go deeper than this, I'm afraid you're going to need to reverse engineer `OVRPlugin.dll
`. I'm happy to stop here.
So, final recap of the data flow:
Yikes!
Next week I'll be back at work on the game and I'll take care of recording and playing back body tracking data. Thanks to the analysis I did, I have a pretty good idea of where to start.